Knowledge base construction method and system for multi-source heterogeneous files

Abstract

The invention relates to a knowledge base construction method and system for multi-source heterogeneous files, and the method comprises the steps: receiving Word, PDF, Excel and other multi-source heterogeneous files, and converting the multi-source heterogeneous files into standardized representation through a file analysis function; an event boundary recognition technology fusing rule matching, entity recognition, semantic similarity and large language model verification is adopted, and a cross-paragraph complete business event is accurately extracted; generating structured information through an information extraction function; the information is converted into vectors, full-text indexes and a graph database to be stored by means of a multi-dimensional storage conversion function; and verifying and correcting an extraction result through a quality auditing function, and finally outputting a knowledge base containing an event set, an entity relationship and vector representation. According to the method, the problem of knowledge fragmentation in traditional document processing is effectively solved, and the consistency, the searchability and the reasonability of knowledge are remarkably improved.

Description

Technical Field

[0001] This invention relates to the technical fields of natural language processing, information extraction and knowledge engineering, specifically to a method and system for constructing a knowledge base of multi-source heterogeneous documents, and more particularly to a technology for constructing a knowledge base of multi-source heterogeneous documents based on a complete business event modeling method. Background Technology

[0002] In the actual operation of enterprises, a large number of heterogeneous business documents from multiple sources are generated, such as technical reports, meeting minutes, fault records, quality inspection documents, and operation logs. These documents carry valuable experience, knowledge, and event information for the enterprise. However, most enterprises still use traditional document management systems or keyword-based full-text search technology, which can only achieve document storage and simple retrieval. It is difficult to extract and organize knowledge from document content in a deep, structured, and semantic way, thus limiting the effectiveness of knowledge reuse and decision support.

[0003] Patent document CN111177306A discloses a data processing method and apparatus. This method involves acquiring unstructured documents, performing knowledge organization on the unstructured documents to obtain corresponding knowledge entries, storing the unstructured documents and knowledge entries in a knowledge base, obtaining search keywords, and performing a search based on the search keywords and the unstructured documents and knowledge entries stored in the knowledge base to obtain the target documents corresponding to the search keywords. Specifically, the existing technology has the following shortcomings: Document parsing technologies have limitations: Traditional document parsing tools (such as python-docx and Apache POI) are primarily designed for surface text extraction and struggle to accurately handle documents with complex layouts, such as nested tables, mixed text boxes, and technical charts. Especially when processing engineering documents, existing methods often lose key structural and semantic information, leading to content fragmentation. For PDF documents, although deep learning-based layout analysis techniques exist, in practical applications, OCR accuracy significantly decreases when faced with poor scan quality, skewed images, watermark interference, or complex table images, severely impacting the reliability of subsequent information extraction.

[0004] Insufficient ability to maintain the integrity of business events: Existing document processing systems typically employ fixed-length sliding windows or mechanical segmentation methods based on paragraphs and pages, lacking the ability to identify and maintain the semantic integrity of business events. In actual business documents, the description of a complete event (such as fault handling or quality improvement) is often scattered across multiple paragraphs, tables, or even different pages, including both phenomenological descriptions and causal analysis and solutions. Traditional segmentation methods easily lead to fragmented event information, disrupting the coherence of the event context, and consequently affecting the accurate extraction and retrieval of event-level knowledge.

[0005] Integrating multi-source heterogeneous data is challenging: Enterprise documents come in diverse formats, including Word, Excel, PDF, CSV, and system logs. Current processing methods often involve designing independent parsing processes for different formats, lacking a unified framework and semantic consistency management. More importantly, different departments or business systems use different names for the same entity or concept (e.g., "fault" vs. "abnormality," "high-speed train" vs. "train"), lacking effective entity alignment and semantic unification mechanisms. This leads to fragmented knowledge and makes it difficult to form a unified and interconnected knowledge system.

[0006] Lack of knowledge quality assurance mechanisms: Existing information extraction systems generally rely on manual review and verification of extraction results, resulting in low automation and processing efficiency that cannot meet the needs of large-scale document processing in enterprises. Especially when models face domain migration or changes in document types, the extraction results are unstable, and the lack of automated error detection and correction mechanisms severely restricts the efficiency and credibility of knowledge base construction.

[0007] Therefore, there is an urgent need to develop an automated knowledge construction technology that can deeply integrate document parsing, event modeling, semantic extraction, and quality assurance, so as to effectively identify, extract in a structured manner, and systematically store business events in multi-source heterogeneous documents, and provide support for enterprise knowledge management, intelligent retrieval, and decision analysis. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for constructing a knowledge base from multiple heterogeneous files.

[0009] According to the present invention, a method for constructing a knowledge base from multiple heterogeneous files includes the following steps: Step S1: Receive multi-source heterogeneous document input, including Word documents, PDF files and Excel spreadsheets; Step S2: Convert the original document into a standardized representation using a document parsing function; Step S3: Identify event boundaries in the document using the event boundary recognition function; Step S4: Extract structured information from the identified events using an information extraction function; Step S5: Convert the extracted information into vector representation, full-text index, and graph representation using a multidimensional storage transformation function; Step S6: Verify and correct the extraction results using a quality audit function; Step S7: Output a knowledge base containing a set of events, a set of entity relationships, and a set of vector representations.

[0010] Preferably, step S2 includes the following steps: Step S2.1: Parse the underlying XML structure of the Word document, recursively process nested tables and text boxes, extract paragraph, table and image content and convert it to Markdown format; Step S2.2: Perform image rendering, tilt correction, noise reduction and watermark removal on the PDF document. Use a deep learning layout analysis model to identify text, tables and image regions, and extract text content through OCR technology. Step S2.3: Standardize cross-file fields in the Excel spreadsheet by calculating semantic similarity.

[0011] Preferably, step S3 includes the following steps: Step S3.1: Identify the event start marker based on rule features; Step S3.2: Calculate the overlap of entities in adjacent paragraphs using named entity recognition; Step S3.3: Calculate semantic similarity using a pre-trained language model; Step S3.4: Extract the time expression to determine time continuity; Step S3.5: Identify event boundaries by combining the above features, and perform boundary verification using a large language model.

[0012] Preferably, step S4 includes the following steps: Step S4.1: Perform word segmentation, part-of-speech tagging, named entity recognition, and dependency parsing on the event text; Step S4.2: Extract structured fields based on rule templates; Step S4.3: Extract the relationships between entities; Step S4.4: Extract complex semantic information using a large language model; Step S4.5: Perform confidence fusion and inference completion on the multi-source extraction results.

[0013] Preferably, step S5 includes the following steps: Step S5.1: Generate multi-granularity vector representations of events, including full-text vectors, abstract vectors, title vectors, and question-answer pair vectors; Step S5.2: Build a full-text index that supports keyword and structured queries; Step S5.3: Construct a knowledge graph, with events and entities as nodes and relationships as edges, forming a networked storage structure.

[0014] Preferably, step S6 includes the following steps: Step S6.1: Differentiate between key fields and ordinary fields, and set review priorities; Step S6.2: Construct verification prompts based on the large language model, and re-extract and compare the original values; Step S6.3: Evaluate the extraction results from three dimensions: exact matching, fuzzy matching, and semantic matching; Step S6.4: Based on the matching results, mark the results as passed, format standardized, or manually reviewed; Step S6.5: Generate an audit report and provide suggestions for optimizing the extraction algorithm.

[0015] Preferably, the knowledge base in step S7 includes: a vector database layer, a full-text index layer, and a graph database layer; The vector database layer is used to store semantic vector representations and support similarity retrieval; the full-text index layer is used to store structured event content and support keyword retrieval; the graph database layer is used to store entity relationship networks and support association analysis and reasoning.

[0016] This invention also provides a knowledge base construction system for multi-source heterogeneous files, the system comprising the following modules: Module M1: Receives multi-source heterogeneous document input, including Word documents, PDF files, and Excel spreadsheets; Module M2: Converts the raw document into a standardized representation using document parsing functions; Module M3: Identifies event boundaries in a document using event boundary recognition functions; Module M4: Extracts structured information from identified events using information extraction functions; Module M5: Converts extracted information into vector representations, full-text indexes, and graph representations using multidimensional storage transformation functions; Module M6: Verifies and corrects the extraction results through a quality audit function; Module M7: Outputs a knowledge base containing sets of events, entity relationships, and vector representations.

[0017] Preferably, module M2 includes the following modules: Module M2.1: Parses the underlying XML structure of Word documents, recursively processes nested tables and text boxes, extracts paragraph, table and image content and converts it to Markdown format; Module M2.2: Performs image rendering, tilt correction, noise reduction, and watermark removal on PDF documents. It uses a deep learning layout analysis model to identify text, tables, and image regions, and extracts text content through OCR technology. Module M2.3: Standardizes cross-file fields in Excel spreadsheets through semantic similarity calculation; Module M3 includes the following modules: Module M3.1: Identifies event start markers based on rule-based features; Module M3.2: Calculates the overlap of entities in adjacent paragraphs using named entity recognition; Module M3.3: Calculates semantic similarity using a pre-trained language model; Module M3.4: Extract time expressions to determine time continuity; Module M3.5: Integrates the above features to identify event boundaries and uses a large language model for boundary verification; Module M4 includes the following modules: Module M4.1: Performs word segmentation, part-of-speech tagging, named entity recognition, and dependency parsing on event text; Module M4.2: Extracting structured fields based on rule templates; Module M4.3: Extracting relationships between entities; Module M4.4: Extracts complex semantic information through a large language model; Module M4.5: Performs confidence fusion and inference completion on results extracted from multiple sources.

[0018] Preferably, module M5 includes the following modules: Module M5.1: Generates multi-granularity vector representations of events, including full-text vectors, abstract vectors, title vectors, and question-answer pair vectors; Module M5.2: Builds full-text indexes, supporting keyword and structured queries; Module M5.3: Constructs a knowledge graph, with events and entities as nodes and relationships as edges, forming a networked storage structure; Module M6 includes the following modules: Module M6.1: Differentiate between key fields and ordinary fields, and set approval priorities; Module M6.2: Constructs verification prompts based on a large language model, re-extracts and compares the original values; Module M6.3: Evaluates extraction results from three dimensions: exact matching, fuzzy matching, and semantic matching; Module M6.4: Marks the results as passed, format standardized, or manually reviewed based on the matching results; Module M6.5: Generates audit reports and provides suggestions for optimizing the extraction algorithm; The knowledge base in module M7 includes: a vector database layer, a full-text index layer, and a graph database layer; The vector database layer is used to store semantic vector representations and support similarity retrieval; the full-text index layer is used to store structured event content and support keyword retrieval; the graph database layer is used to store entity relationship networks and support association analysis and reasoning.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention utilizes a multi-dimensional event boundary recognition method that integrates rule matching, entity recognition, semantic similarity calculation, and large language model verification. The system can accurately identify complete business events that span paragraphs and structures, effectively solving the knowledge fragmentation problem caused by traditional mechanical segmentation; it significantly improves the semantic consistency of event descriptions and the accuracy of subsequent retrieval and reasoning. 2. This invention targets various document formats such as Word, PDF, and Excel, employing an adaptive parsing strategy to accurately restore nested tables and complex layouts by recursively parsing the underlying XML structure of docx documents; it combines deep learning layout analysis and OCR post-processing optimization to improve the extraction quality of text and tables in scanned PDFs; and it introduces a semantic-based field standardization mechanism to effectively solve the mapping problem of synonyms with different names, providing a high-quality structured data foundation for knowledge integration. 3. This invention utilizes the powerful semantic understanding and few-shot learning capabilities of large language models, enabling the system to quickly adapt to new document types and extraction requirements, reducing reliance on large amounts of labeled data and fixed templates. It innovatively introduces an automatic review mechanism based on large language models, which achieves automatic error correction and quality assessment of extraction results through multi-dimensional comparison and verification, significantly reducing manual review costs and improving the efficiency and reliability of knowledge base construction. Attached Figure Description

[0020] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the principle of the present invention. Detailed Implementation

[0021] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0022] Example 1: Reference Figure 1 According to the present invention, a method for constructing a knowledge base from multiple heterogeneous files includes the following steps: Step S1: Receive multi-source heterogeneous document input, including Word documents, PDF files and Excel spreadsheets; Step S2: Convert the original document into a standardized representation using a document parsing function; Step S2.1: Parse the underlying XML structure of the Word document, recursively process nested tables and text boxes, extract paragraph, table, and image content, and convert it into Markdown format; Step S2.2: Perform image rendering, skew correction, noise reduction, and watermark removal on the PDF document, use a deep learning layout analysis model to identify text, table, and image regions, and extract text content using OCR technology; Step S2.3: Standardize cross-file fields in the Excel table through semantic similarity calculation.

[0023] Step S3: Identify event boundaries in the document using the event boundary recognition function; Step S3.1: Identify event start markers based on rule features; Step S3.2: Calculate the entity overlap between adjacent paragraphs using named entity recognition; Step S3.3: Calculate semantic similarity using a pre-trained language model; Step S3.4: Extract time expressions to determine temporal continuity; Step S3.5: Identify event boundaries by combining the above features and perform boundary verification using a large language model.

[0024] Step S4: Extract structured information from the identified events using an information extraction function; Step S4.1: Perform word segmentation, part-of-speech tagging, named entity recognition, and dependency parsing on the event text; Step S4.2: Extract structured fields based on rule templates; Step S4.3: Extract relationships between entities; Step S4.4: Extract complex semantic information using a large language model; Step S4.5: Perform confidence fusion and inference completion on the extraction results from multiple sources.

[0025] Step S5: Convert the extracted information into vector representation, full-text index, and graph representation using a multi-dimensional storage transformation function; Step S5.1: Generate multi-granularity vector representations of events, including full-text vectors, summary vectors, title vectors, and question-answer pair vectors; Step S5.2: Construct a full-text index that supports keyword and structured queries; Step S5.3: Construct a knowledge graph with events and entities as nodes and relationships as edges to form a networked storage structure.

[0026] Step S6: Verify and correct the extraction results using a quality audit function; Step S6.1: Differentiate between key fields and ordinary fields, and set audit priorities; Step S6.2: Construct verification prompts based on a large language model, re-extract, and compare with the original values; Step S6.3: Evaluate the extraction results from three dimensions: exact matching, fuzzy matching, and semantic matching; Step S6.4: Mark the results as passed, format standardized, or manually audited based on the matching results; Step S6.5: Generate an audit report and provide suggestions for optimizing the extraction algorithm.

[0027] Step S7: Output a knowledge base containing event sets, entity relationship sets, and vector representation sets. The knowledge base includes: a vector database layer, a full-text index layer, and a graph database layer; the vector database layer is used to store semantic vector representations and support similarity retrieval; the full-text index layer is used to store structured event content and support keyword retrieval; the graph database layer is used to store entity relationship networks and support association analysis and reasoning.

[0028] This embodiment's method targets various document formats such as Word, PDF, and Excel. It employs an adaptive parsing strategy, recursively parsing the underlying XML structure of the docx file to accurately reconstruct nested tables and complex layouts. It extracts page elements and their corresponding information positions through decomposition and reconstruction. The algorithm used is a document adaptive parsing and reconstruction algorithm. Input: Source document file path DocPath, image upload API address ApiUrl; Output: Structured JSON data (StructuredData) and rebuilt document (RebuiltDoc); Phase 1: Element Analysis and Extraction Document Parser (Document Path): First, the parser is initialized, the document object is loaded, and a unique output directory is created. Simultaneously, a deduplication set and a URL mapping table are established for subsequent processing. During the image resource extraction stage, the media directory in the document ZIP structure is traversed, image data is extracted one by one and saved to the output directory, and each image is assigned an incrementing number and its path, filename, and content type are recorded.

[0029] A recursive strategy is used for table parsing. A function is defined to process the table (table object, nested or not). When traversing each row and cell of the table, it first checks if the cell contains a nested table. If so, it recursively calls itself to retrieve the nested data. Simultaneously, it extracts the cell's text content and image references, ensuring no duplicate content is added to the row data structure via hash comparison, and finally returns the complete structured table data. During paragraph processing, all paragraphs in the document are traversed and their text content is extracted. A function is also called to extract image references (paragraph object) to obtain the associated image information within the paragraph. When the text is not empty or contains image references, the paragraph data is added to the result set after deduplication verification.

[0030] To establish a mapping between images and URLs, all images are uploaded in batches to a specified API interface. Upon receiving the upload results, a triple mapping is created for each image, using the original path, original filename, and new filename as keys, ensuring that the corresponding URL can be quickly retrieved through various methods later. Finally, the parsed results are exported as structured JSON, containing document identifiers, filenames, table arrays (including nested hierarchical relationships), paragraph arrays (including image reference positions), and image metadata arrays (including the uploaded URL address).

[0031] Phase Two: Document Restructuring and Formatting Document Reconstructor (Structured Data): After loading parsed structured data from a JSON file, it begins reconstructing the document element sequence. An array of content items is created, and all non-nested tables and paragraphs are iterated through, added to the array along with their original indices. These are then sorted in ascending order by index to restore the original document's element order. Formatting rendering is implemented recursively. A function is defined to format tables (table objects), processing each row and cell individually. It first outputs the cell text content; if the cell contains an image, it outputs the image URL; if nested tables exist, it recursively calls itself to continue processing the inner structure. Similarly, a function is defined to format paragraphs (paragraph objects), outputting the paragraph text and its associated image URLs sequentially.

[0032] During the chapter identification phase, the current chapter variable is initialized to the default chapter. The sorted content items are traversed; when a paragraph matching the title characteristics is encountered (e.g., short text starting with "Chapter X" or ending with a colon), the accumulated content of the current chapter is saved and the system switches to the new chapter; otherwise, the formatted content of the current item is appended to the current chapter. The final document is generated according to the target output format. If the format is plain text, all formatted content is output in sequence; if it is HTML, corresponding tags are wrapped and image elements are embedded; if output by chapter, the content of each chapter is written to a separate file. The reconstructed document object is returned for later use.

[0033] This embodiment innovatively introduces an automatic review mechanism based on a large language model. Through multi-dimensional comparison and verification, it achieves automatic error correction and quality assessment of the extracted results, significantly reducing the cost of manual review and improving the efficiency and reliability of knowledge base construction. The algorithm used is an automatic review and error correction algorithm based on a large language model. Input: DocSet, a collection of documents to be reviewed; RefField, a reference field identifier; and AuditFields, a configuration field for the target field. Output: Audit Report, Update Suggestions; Phase 1: Intelligent Field Extraction and Validation Field Auditor (Document Collection, Audit Configuration): The system first initializes the large language model connection and database client. For each document to be audited, it extracts the complete text from its reference fields as a semantic baseline. During the field extraction phase, a structured prompt word template is built, explicitly requiring the model to return the extraction results according to a specific XML tag format, i.e., wrapping the target field value in...<FIELD_VALUE> The tag allows for three standardized return states: if the field content is explicit, return the specific value; if the field is not found in the document, return a "not found" flag; and if the field exists but the content is ambiguous, return a "not specified" flag.

[0034] When calling the large language model API, an exponential backoff retry strategy is employed, and a low temperature parameter is set to ensure extraction stability. After receiving the model response, a multi-level regular expression parsing engine is launched, first attempting to match the criteria.<FIELD_VALUE> The tags extract their internal text. If the main pattern fails to match, alternative patterns such as "field value:" and "extraction result:" are tried in sequence to ensure that key information can still be accurately captured even if the model output format is slightly off. All extracted results are cleaned and standardized, and then compared with the original field values ​​of the document at the semantic level for consistency.

[0035] To address the review needs of large-scale document collections, the system offers both synchronous and asynchronous processing modes. In asynchronous mode, an HTTP connection pool is created, and a semaphore mechanism is used to control the number of concurrent requests. Multiple target fields are extracted in parallel during the review of a single document, significantly improving overall throughput. After each batch of documents is reviewed, detailed information on inconsistent fields is recorded in real time, including the original value, extracted value, and field semantic description, providing sufficient evidence for subsequent error correction.

[0036] Phase Two: Multi-dimensional Consistency Determination Algorithm Consistency Assessment (Original Value, Extracted Value): When performing content consistency checks, the original field values ​​and model extracted values ​​are first standardized preprocessed, including removing leading and trailing whitespace and converting them to lowercase to eliminate format differences. Intelligent recognition is then performed for special states of the extracted values. When the extraction result is "Not Found," "Undefined," or null, it is determined whether the original field is also null or undefined, achieving correct matching in null value scenarios.

[0037] For non-empty fields, a strict string equivalence comparison is performed. If the two are completely identical, the verification is marked as passed; otherwise, an inconsistency is recorded. The system generates a structured audit record for each document, including the document identifier, audit timestamp, detailed comparison results for each field, and an overall inconsistency flag, supporting subsequent statistical analysis and batch processing decisions.

[0038] Phase 3: Automatic Error Correction and Quality Assessment Application of Audit Results (Audit Report, Execution Mode): After loading the complete audit report, the system iterates through the audit records of all documents and filters out entries with inconsistencies. For each document to be updated, it extracts fields that failed validation, filters out cases marked "not found" by the model to avoid mistakenly deleting valid data, and uses the extracted values ​​of the remaining inconsistent fields as correction suggestions. In dry run mode, the system only counts the number of potential update operations and the range of fields involved, and outputs a detailed change preview log for manual review and decision-making.

[0039] In formal execution mode, a partial update request is constructed for each document, modifying only inconsistent fields while keeping other fields unchanged. Changes are submitted via the database atomic update interface, and anomalies are captured. Throughout the process, key metrics such as the number of successfully updated documents, the number of corrected fields, the number of skipped documents, and the number of failed operations are continuously tracked. Upon completion of the review, a comprehensive quality report is generated, calculating the total number of documents, the number of inconsistent documents, and the overall consistency rate to assess the knowledge base data quality level. This report not only includes macro-level statistical information but also retains fine-grained review results for each document, supporting in-depth analysis by field type, error patterns, and other dimensions, providing data support for continuous optimization of extraction rules and model hints.

[0040] The present invention also provides a knowledge base construction system for multi-source heterogeneous files. The knowledge base construction system for multi-source heterogeneous files can be implemented by executing the process steps of the knowledge base construction method for multi-source heterogeneous files. That is, those skilled in the art can understand the knowledge base construction method for multi-source heterogeneous files as a preferred embodiment of the knowledge base construction system for multi-source heterogeneous files.

[0041] Example 2: This invention also provides a knowledge base construction system for multi-source heterogeneous files, the system comprising the following modules: Module M1: Receives multi-source heterogeneous document input, including Word documents, PDF files, and Excel spreadsheets; Module M2: Converts raw documents into a standardized representation using document parsing functions; Module M2.1: Parses the underlying XML structure of Word documents, recursively processes nested tables and text boxes, extracts paragraph, table, and image content, and converts it to Markdown format; Module M2.2: Performs image rendering, skew correction, noise reduction, and watermark removal on PDF documents, uses a deep learning layout analysis model to identify text, table, and image regions, and extracts text content using OCR technology; Module M2.3: Standardizes cross-file fields in Excel spreadsheets through semantic similarity calculation; Module M3: Identifies event boundaries in a document using an event boundary recognition function; Module M3.1: Identifies event start markers based on rule features; Module M3.2: Calculates entity overlap between adjacent paragraphs using named entity recognition; Module M3.3: Calculates semantic similarity using a pre-trained language model; Module M3.4: Extracts temporal expressions to determine temporal continuity; Module M3.5: Identifies event boundaries by combining the above features and performs boundary verification using a large language model. Module M4: Extracts structured information from identified events using information extraction functions; Module M4.1: Performs word segmentation, part-of-speech tagging, named entity recognition, and dependency parsing on event text; Module M4.2: Extracts structured fields based on rule templates; Module M4.3: Extracts relationships between entities; Module M4.4: Extracts complex semantic information using a large language model; Module M4.5: Performs confidence fusion and inference completion on extraction results from multiple sources.

[0042] Module M5: Converts extracted information into vector representations, full-text indexes, and graph representations through multi-dimensional storage transformation functions; Module M5.1: Generates multi-granularity vector representations of events, including full-text vectors, summary vectors, title vectors, and question-answer pair vectors; Module M5.2: Constructs a full-text index, supporting keyword and structured queries; Module M5.3: Constructs a knowledge graph, with events and entities as nodes and relationships as edges, forming a networked storage structure. Module M6: Verifies and corrects the extraction results through a quality audit function; Module M6.1: Differentiates between key fields and ordinary fields and sets audit priorities; Module M6.2: Constructs verification prompts based on a large language model, re-extracts and compares with the original values; Module M6.3: Evaluates the extraction results from three dimensions: exact matching, fuzzy matching, and semantic matching; Module M6.4: Marks the results as passed, formatted, or manually audited based on the matching results; Module M6.5: Generates an audit report and provides suggestions for optimizing the extraction algorithm. Module M7: Outputs a knowledge base containing event sets, entity relationship sets, and vector representation sets. The knowledge base includes: a vector database layer, a full-text index layer, and a graph database layer; the vector database layer stores semantic vector representations and supports similarity retrieval; the full-text index layer stores structured event content and supports keyword retrieval; the graph database layer stores entity relationship networks and supports association analysis and reasoning.

[0043] Example 3: The main objective of this invention is to construct a multi-source heterogeneous document knowledge base using a business event modeling approach, thereby achieving intelligent processing of enterprise business documents and structured knowledge management. Specifically, the goal is to provide a method for identifying and maintaining business events based on semantic integrity. Currently, enterprise document processing systems typically employ mechanical segmentation strategies with fixed lengths or paragraph boundaries, lacking consideration for the integrity of business events. To address this issue, this invention proposes an intelligent event boundary identification method that uses a deep learning model to understand the semantic structure of documents and accurately identify the start and end positions of business events. Specifically, this method combines multi-dimensional features such as rule matching, entity recognition, and semantic similarity calculation to dynamically determine the correlation between text paragraphs. When processing complex business events spanning multiple paragraphs, such as fault analysis reports and quality improvement records, this method can maintain the integrity of the event description, avoid the fragmentation of key information, and provide high-quality text units for subsequent knowledge extraction and retrieval.

[0044] This invention achieves standardized processing of multi-source heterogeneous documents through deep document parsing technology. Existing document parsing tools have many limitations when handling complex document formats, failing to accurately extract nested structures and maintain hierarchical relationships. To address this, the invention develops an adaptive document parsing framework with specialized parsing strategies for different document formats. For docx documents, it directly parses the underlying XML structure, recursively processes nested elements, and fully restores multi-layered tables and complex layouts. For PDF documents, it employs a deep learning-based layout analysis model combined with OCR technology to accurately extract text, tables, and images. For tabular data such as Excel, it standardizes fields across documents through semantic similarity calculation. In this way, the invention can convert multi-source heterogeneous business documents into a unified structured representation, laying a data foundation for knowledge base construction.

[0045] This invention establishes an intelligent knowledge extraction and quality assurance mechanism based on a large language model. Traditional information extraction methods rely on large amounts of labeled data and predefined templates, resulting in poor adaptability to new document types and reliance on manual review for extraction quality. Therefore, this invention utilizes the powerful semantic understanding capabilities of a pre-trained language model to design an intelligent field extraction algorithm. This algorithm employs few-shot learning techniques, requiring only a small number of examples to adapt to new document types and extraction needs. Simultaneously, this invention innovatively introduces an automatic review mechanism based on a large language model, detecting extraction errors through comparative verification and achieving automated assurance of knowledge quality. This dual mechanism not only improves the accuracy of knowledge extraction but also significantly reduces the workload of manual review, making the construction of large-scale knowledge bases possible.

[0046] This invention establishes a multi-dimensional collaborative knowledge storage and retrieval architecture. Enterprises have diverse knowledge query needs, requiring not only semantic similarity retrieval but also precise matching and relational reasoning. A single storage method is insufficient to meet these complex requirements. Therefore, this invention designs a three-layer collaborative storage architecture: a vector database layer stores semantic representations of documents and events, supporting similarity retrieval; a full-text index layer stores structured event content, enabling precise queries based on keywords and fields; and a graph database layer constructs an entity relationship network, supporting complex association analysis and reasoning. These three layers are tightly linked through a unified identifier system, coupled with an intelligent query routing mechanism, which can automatically select the optimal retrieval strategy based on the query intent, providing users with accurate and comprehensive knowledge services.

[0047] This invention proposes a technology for constructing a multi-source heterogeneous document knowledge base based on business event modeling. Its core lies in transforming unstructured document content into structured business event representations and establishing a multi-dimensional knowledge storage and retrieval system. The specific technical steps are as follows: A technology for constructing a multi-source heterogeneous document knowledge base based on a complete business event modeling method; the input document set is set as follows: ,in Indicates the first One document, document type The system output is a knowledge base. Where $E$ represents the set of events extracted. Represents a set of entity relations. Vectors represent sets.

[0048] The system processing flow is defined as a six-tuple: ,in: Document parsing function Convert the original document into a standardized representation. Event boundary identification function Identify event boundaries in a document Information extraction function Extracting structured information Multidimensional storage conversion function Generate vectors, indices, and graph representations. Quality audit function Verify and correct the extraction results System overall architecture and data flow design: This system accepts document input in various formats, including Word documents, PDF files, Excel spreadsheets, and CSV data files. The core task of the system is to understand the content of these documents and convert it into a structured knowledge representation, ultimately outputting a knowledge base that can be efficiently retrieved and utilized.

[0049] The entire processing flow can be summarized into five main stages. First is the document parsing stage, which converts raw documents of different formats into a unified, standardized representation, laying the foundation for subsequent processing. Next is the event boundary identification stage, where the system intelligently identifies which content in the document belongs to the same business event and which are the dividing points between different events. The third stage is information extraction, which extracts structured field information from the identified event text. The fourth stage is multi-dimensional storage transformation, converting the extracted information into vector, index, and graph formats for storage. Finally, the quality review stage verifies and corrects the extraction results.

[0050] These five stages are not a simple linear process, but involve feedback and iteration. For example, problems discovered in the quality audit stage will be fed back to the information extraction stage for improvement, and the results of event boundary identification will also affect the choice of information extraction strategy.

[0051] Multi-source document standardization processing; deep parsing of Word documents; defining parsing functions for docx documents. Parsing Word documents is one of the key entry points for the entire system. A modern docx document is actually a compressed file containing multiple XML files describing the document's content and formatting. The system first needs to decompress the document to obtain its core content files. After decompression, the system constructs a Document Object Model (DOM), which is like creating a hierarchical map of the document content. By performing a depth-first traversal of this model, the system can extract all content in the order of human reading. During the traversal, the system pays particular attention to three types of important elements: paragraphs, tables, and images.

[0052] For paragraph content, the system not only extracts the text itself but also retains the paragraph's style information, such as whether it is a heading and what level of heading style is used. This formatting information is crucial for subsequent event boundary identification, as headings often mark the beginning of new events.

[0053] Processing tables is relatively complex, requiring accurate reconstruction of their row and column structure. This is especially true when the table contains merged cells, as the system needs to correctly identify the actual row and column range occupied by each cell. The system constructs a matrix to represent the entire table, ensuring that merged cells occupy their correct positions within the matrix. This precise table parsing lays the foundation for subsequent extraction of structured data from the table.

[0054] Image content is extracted and saved as a separate file, while the image's reference position is preserved in the document structure. This ensures that image information is not lost while maintaining the association between the image and the text content.

[0055] Finally, all the parsed content will be converted into a unified Markdown format. Markdown was chosen as the intermediate format because it preserves the document's structural information while facilitating subsequent text processing and analysis.

[0056] PDF document layout analysis and content extraction: Processing PDF documents presents challenges completely different from processing Word documents. The PDF format is primarily used to maintain visual consistency, but this also makes content extraction difficult. The system employs a deep learning-based approach to process PDF documents.

[0057] The first step in the process is to render the PDF page as a high-resolution image. Choosing a resolution of 300 DPI strikes a balance between processing speed and recognition accuracy. After rendering, the system performs a series of preprocessing operations on the image.

[0058] Secondly, tilt correction is a necessary preprocessing step because scanned documents often have slight tilts, which can affect the accuracy of subsequent text recognition. The system determines the degree of tilt by detecting the angle of the text lines and then rotates the image to correct it.

[0059] Denoising primarily targets noise generated during the scanning process. While this noise is negligible to the human eye, it can interfere with OCR recognition. The system employs image processing techniques such as median filtering to remove this noise while maintaining the clarity of text edges. Additionally, many formal documents contain watermarks, and these semi-transparent background patterns can severely affect text recognition. The system identifies and removes watermarks by analyzing the color distribution and transparency features of the image.

[0060] After preprocessing, the system uses a deep learning model for layout analysis. This model can identify different regions on the page, including text paragraphs, tables, and images. Each identified region is labeled with its type and precise location on the page.

[0061] Based on the layout analysis, the system employs targeted processing methods for different types of regions. Text regions are recognized using OCR technology, and post-processing is performed to correct common OCR errors. Table regions utilize a specialized table recognition model, which not only recognizes the text within cells but also reconstructs the table structure. Image regions are cropped out and saved separately.

[0062] All extracted content is then assembled in the correct reading order. The system merges adjacent text blocks to avoid unnecessary segmentation of the same paragraph. This process ensures that the extracted results are both complete and conform to human reading habits.

[0063] Standardize tabular data fields; when processing multiple tabular files, it's common to encounter situations where the same concept uses different field names. For example, "Customer Name," "Customer Name," and "Customer Name" all refer to the same concept. The system needs to establish a field mapping mechanism to unify these different expressions.

[0064] The first step in standardization is to collect the field names appearing in all tables. The system iterates through each table file, extracts its header information, and builds a complete dictionary of raw fields. This dictionary presents a complete picture of the data, helping the system understand the range of fields that need to be processed.

[0065] Next, the system needs to create or load a standard field library. If the organization already has predefined standard field names, the system will use them directly. Otherwise, the system will select the most common or standardized expression from the collected fields as the standard.

[0066] The core of field mapping is calculating the similarity between the original field and the standard field. The system employs a multi-dimensional similarity calculation method. Edit distance can capture spelling similarities, such as "customer name" and "customer name" differing by only one character. Semantic similarity is calculated using word vector technology, which can identify synonymous relationships such as "customer" and "customer".

[0067] The system will find the best matching standard field for each original field. If the similarity is high, the match is reliable and a mapping relationship can be established directly. If the similarity is moderate, the system will mark it as requiring manual confirmation. If the similarity is very low, it may be a new concept, and the system will create a new standard field for it.

[0068] After the mapping relationship is established, the system will also perform a consistency check. For example, if multiple different original fields are mapped to the same standard field, the system needs to use context information to determine whether this mapping is reasonable and avoid erroneous merging.

[0069] This standardization mechanism ensures that the same information from different data sources can be processed uniformly, laying the foundation for subsequent data integration and knowledge extraction.

[0070] Semantic understanding-based event boundary identification is a crucial step in dividing continuous document content into independent business events. This process is akin to reorganizing a book without chapters into a structured chapter framework. The system employs a multi-feature comprehensive judgment method to identify event boundaries. The first step is the identification of regular features. Many documents use specific markers when describing new events, such as numbering (I, II, III or 1, 2, 3), distinctive words ("for," "about," "event summary"), or special formatting (such as heading styles). The system scans each paragraph, looking for these obvious event initiation markers.

[0071] Relying solely on rules is insufficient, as not all documents have clear formatting tags. The system also analyzes entity consistency. A complete event typically involves specific entities such as people, places, and organizations. Using named entity recognition technology, the system can extract entities mentioned in each paragraph and then calculate the entity overlap between adjacent paragraphs. If two paragraphs mention completely different entities, they likely belong to different events.

[0072] Semantic similarity is another important criterion. The system uses pre-trained language models such as BERT to convert text into vector representations, and then calculates the semantic similarity between adjacent content. Content belonging to the same event should be semantically coherent, while semantic jumps often occur during event transitions.

[0073] Temporal continuity is also an important consideration. The system extracts time expressions from the text to determine whether adjacent content describes events from the same period. If the time span is large, such as suddenly jumping from "March 2023" to "September 2024," this usually indicates a transition in events.

[0074] The system will make a boundary determination based on all the above features. When the rule features strongly indicate the start of a new event, or the entity overlap is very low, or the semantic similarity decreases significantly, or a time jump occurs, the system will determine that this is an event boundary.

[0075] After initial boundary identification, the system performs secondary verification using a large language model. For two adjacent event segments, the system constructs prompts for the large language model to determine whether they should belong to the same event. This verification can capture semantic relationships that are difficult to discover using rule-based and statistical methods.

[0076] Through this multi-layered boundary recognition mechanism, the system can accurately divide document content into independent business events, providing clear processing units for subsequent structured information extraction.

[0077] Structured information extraction; the goal of information extraction is to extract structured field information from identified event text. This process requires understanding the semantics of the text, identifying key information, and categorizing it into predefined fields.

[0078] The system first preprocesses the event text, including word segmentation, part-of-speech tagging, named entity recognition, and dependency parsing. These basic natural language processing steps provide rich linguistic features for subsequent information extraction.

[0079] Rule-based template extraction is the most direct and effective method. For information with standardized formats, such as time and number, the system uses predefined regular expression patterns for matching. Time information may appear in various formats, such as "March 15, 2024" or "2024-03-15". The system's rules need to cover these common formats and convert them uniformly into the standard format.

[0080] Entity relation extraction helps build the knowledge structure within an event. The system identifies all entity pairs in the text and then analyzes the relationships between them. For example, in the sentence "Zhang San is responsible for handling server failures," the system needs to identify the "responsible for handling" relationship between "Zhang San" (person entity) and "server failures" (event entity). This relation extraction not only provides structured information but also constructs a local knowledge graph.

[0081] For complex semantic information, the system leverages the understanding capabilities of a large language model. Through carefully designed prompts, the system guides the large language model to extract specific fields from the text. The prompts clearly specify the types of fields to be extracted, formatting requirements, and important considerations. The advantage of a large language model lies in its ability to understand context and handle implicit information and complex expressions.

[0082] Different extraction methods may yield different results, necessitating result fusion within the system. For each field, the system collects all candidate values ​​from rule extraction, relation inference, and the large language model, calculating a confidence score for each. During fusion, results with high confidence scores are prioritized, but other candidate values ​​are also recorded for subsequent review.

[0083] Some information, while important, may not be directly mentioned in the text. The system will attempt to infer and complete the information using existing data. For example, if the text describes the scope of the fault's impact and the handling process in detail, the system can infer the severity level of the fault. This reasoning ability makes the extracted results more complete and useful.

[0084] A multi-dimensional knowledge storage architecture is employed; the extracted structured information needs to be stored in a way that suits different retrieval requirements. The system uses three storage methods—vector, full-text index, and knowledge graph—to form a complementary knowledge representation system.

[0085] Vectorized storage primarily serves semantic similarity retrieval. The system generates vector representations at multiple granularities for each event. Full-text vectors capture the overall semantics of the event, suitable for general similarity matching. Summary vectors are generated based on brief summaries of the event, reducing computational overhead while preserving semantic information. Title vectors focus on the event's topic, suitable for rapid topic matching.

[0086] Of particular note is the generation of question-answer pair vectors. The system converts each structured field into a question-answer pair format; for example, "failure time: 2024-03-15" is converted into "Q: When did the failure occur? A: March 15, 2024". This conversion enables the system to find relevant information more accurately when faced with natural language questions.

[0087] Full-text indexing is built to support keyword retrieval and structured queries. Each event is converted into an indexed document, containing text fields such as title, body, and abstract, as well as structured fields such as time and type. The system performs word segmentation on Chinese text, extracts keywords, and builds an inverted index. For nested event structures, the system maintains their hierarchical relationship, supporting complex nested queries.

[0088] The construction of a knowledge graph organizes information into a network structure. Each event and entity is a node in the graph, and the relationships between them form edges. Event nodes contain the basic attributes of the event, while entity nodes record the type and name of the entity. Edges represent various relationships, such as the entities involved in the event, the temporal relationship between events, or the causal relationship.

[0089] These three storage methods work together to support different types of retrieval needs. Vector retrieval excels at handling fuzzy queries such as "find similar events," full-text indexing is suitable for precise keyword matching and conditional filtering, and knowledge graphs support complex relational queries and reasoning.

[0090] To ensure data consistency across storage systems, the system generates a globally unique identifier for each storage item. This identifier contains information such as data type, source, and timestamp, ensuring that the relationships between data are not lost.

[0091] An automatic knowledge quality review mechanism is employed; quality review is a crucial step in ensuring the accuracy of the knowledge base. The system uses an automatic review mechanism based on a large language model to verify and correct the extraction results.

[0092] The review process begins by prioritizing fields. The system categorizes fields into critical and general fields. Event titles, occurrence times, responsible organizations, and processing results are considered critical fields and require priority review. This prioritization mechanism ensures that the most important information receives the most stringent quality control.

[0093] For each field awaiting review, the system will construct a specific verification prompt. The prompt design will be adjusted according to the field type. For example, for a time field, the prompt will require identifying all time expressions and selecting the most appropriate one; for an entity field, the prompt will emphasize extracting only explicitly mentioned information to avoid speculation.

[0094] The system performs multi-dimensional comparisons between the original extracted values ​​and the values ​​re-extracted from the large language model. Exact matching checks if two values ​​are completely identical. Fuzzy matching measures string-level similarity using algorithms such as edit distance, tolerating minor differences in format. Semantic matching compares semantic similarity using word vector techniques, enabling the identification of synonyms.

[0095] Based on the comparison results, the system will make corresponding processing decisions. If the similarity is high, it indicates that the original extraction is accurate, and the system will mark it as verified and appropriately increase the confidence level. If there are format differences but semantic consistency, the system will perform format standardization. If there are significant differences, the system will evaluate which value is more credible and may choose to correct the original value or mark it as requiring manual review.

[0096] Once the review is complete, the system will generate a detailed review report. The report includes statistical information on various review results, such as the verification pass rate and the proportion requiring manual review. The system will also analyze error patterns and identify common extraction error types. Based on this analysis, the system can generate improvement suggestions to guide the optimization of the extraction algorithm.

[0097] This automated review mechanism forms a closed loop for quality assurance. Through continuous review and feedback, the system's extraction accuracy will continuously improve, and the quality of the knowledge base will be reliably guaranteed.

[0098] Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0099] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0100] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for constructing a knowledge base of multi-source heterogeneous files, characterized in that, The method comprises the following steps: Step S1: receiving multi-source heterogeneous document input, the document comprising a Word document, a PDF file and an Excel table; Step S2: converting the original document into a standardized representation through a document parsing function; Step S3: identifying event boundaries in the document through an event boundary identification function; Step S4: extracting structured information from the identified events through an information extraction function; Step S5: converting the extracted information into a vector representation, a full-text index and a graph representation through a multi-dimensional storage conversion function; Step S6: verifying and correcting the extraction results through a quality audit function; Step S7: outputting a knowledge base containing an event set, an entity relationship set and a vector representation set.

2. The method of claim 1, wherein, The step S2 comprises the following steps: Step S2.1: parsing the Word document to extract the underlying XML structure, recursively processing nested tables and text boxes, extracting paragraph, table and image content and converting them into Markdown format; Step S2.2: performing image rendering, skew correction, noise removal and watermark removal on the PDF document, identifying text, table and image regions using a deep learning layout analysis model, and extracting text content through OCR technology; Step S2.3: standardizing cross-file fields through semantic similarity calculation for Excel tables.

3. The method of claim 1, wherein, The step S3 comprises the following steps: Step S3.1: identifying event start markers based on rule features; Step S3.2: calculating the entity overlap degree of adjacent paragraphs through named entity recognition; Step S3.3: calculating semantic similarity through a pre-trained language model; Step S3.4: extracting time expressions to judge time continuity; Step S3.5: identifying event boundaries by integrating the above features, and verifying the boundaries using a large language model.

4. The method of claim 1, wherein, The step S4 comprises the following steps: Step S4.1: performing word segmentation, part-of-speech tagging, named entity recognition and dependency syntax analysis on the event text; Step S4.2: extracting structured fields based on rule templates; Step S4.3: extracting relationships between entities; Step S4.4: extracting complex semantic information through a large language model; Step S4.5: fusing and reasoning the extraction results from multiple sources to complete the extraction.

5. The method of claim 1, wherein, The step S5 comprises the following steps: Step S5.1: generating multi-granularity vector representations of events, including full-text vectors, summary vectors, title vectors and question-answer pair vectors; Step S5.2: constructing a full-text index to support keyword and structured queries; Step S5.3: constructing a knowledge graph with events and entities as nodes and relationships as edges, forming a networked storage structure.

6. The method of claim 1, wherein, The step S6 comprises the following steps: Step S6.1: distinguishing between key fields and ordinary fields and setting audit priorities; Step S6.2: constructing verification prompts based on a large language model, re-extracting and comparing the original values; Step S6.3: evaluating the extraction results from three dimensions: exact match, fuzzy match and semantic match; Step S6.4: verifying the results, standardizing the format or marking them for manual review according to the matching results; Step S6.5: generating an audit report and providing optimization suggestions for the extraction algorithm.

7. The method of claim 1, wherein, The knowledge base in the step S7 comprises: a vector database layer, a full-text index layer and a graph database layer; The vector database layer is used for storing semantic vector representation and supporting similarity retrieval; the full-text index layer is used for storing structured event content and supporting keyword retrieval; and the graph database layer is used for storing entity relationship network and supporting association analysis and reasoning.

8. A multi-source heterogeneous file knowledge base construction system, characterized in that, The system comprises the following modules: Module M1: receiving multi-source heterogeneous document input, the document comprising a Word document, a PDF file and an Excel table; Module M2: converting the original document into a standardized representation through a document parsing function; Module M3: identifying event boundaries in the document through an event boundary identification function; Module M4: extracting structured information from the identified event through an information extraction function; Module M5: converting the extracted information into vector representation, full-text index and graph representation through a multi-dimensional storage conversion function; Module M6: verifying and correcting the extraction result through a quality audit function; Module M7: outputting a knowledge base comprising an event set, an entity relationship set and a vector representation set.

9. The system of claim 8, wherein, The module M2 comprises the following modules: Module M2.1: parsing the Word document to extract paragraph, table and image content and converting them into Markdown format; Module M2.2: performing image rendering, tilt correction, noise removal and watermark removal on the PDF document, identifying text, table and image areas through a deep learning layout analysis model, and extracting text content through OCR technology; Module M2.3: achieving cross-file field standardization through semantic similarity calculation on the Excel table; The module M3 comprises the following modules: Module M3.1: identifying event starting marks based on rule features; Module M3.2: calculating entity overlap degree of adjacent paragraphs through named entity recognition; Module M3.3: calculating semantic similarity through a pre-trained language model; Module M3.4: extracting time expressions to judge time continuity; Module M3.5: identifying event boundaries by comprehensively integrating the above features, and verifying the boundaries through a large language model; The module M4 comprises the following modules: Module M4.1: performing word segmentation, part-of-speech tagging, named entity recognition and dependency syntax analysis on the event text; Module M4.2: extracting structured fields based on rule templates; Module M4.3: extracting relationships between entities; Module M4.4: extracting complex semantic information through a large language model; Module M4.5: fusing and reasoning to complete the extraction results from multiple sources.

10. The multi-source heterogeneous file's knowledge base construction system of claim 8, wherein, The module M5 comprises the following modules: Module M5.1: generating multi-granularity vector representation of events, including full-text vector, summary vector, title vector and question-answer pair vector; Module M5.2: constructing a full-text index to support keyword and structured queries; Module M5.3: constructing a knowledge graph with events and entities as nodes and relationships as edges to form a networked storage structure; The module M6 comprises the following modules: Module M6.1: distinguishing between key fields and ordinary fields and setting audit priorities; Module M6.2: Construct verification prompts based on large language models, re-extract and compare original values; Module M6.3: Evaluate the extraction results from three dimensions of exact match, fuzzy match and semantic match; Module M6.4: According to the matching result, verify pass, format standardization or manual review mark; Module M6.5: Generate audit report and provide extraction algorithm optimization suggestions; The knowledge base in the module M7 includes: vector database layer, full-text index layer and graph database layer; The vector database layer is used to store semantic vector representation and support similarity retrieval; The full-text index layer is used to store structured event content and support keyword search; The graph database layer is used to store entity relationship network and support association analysis and reasoning.

Images