Multi-source heterogeneous data knowledge base system construction method, equipment and medium

Abstract

The invention discloses a knowledge base system construction method and device for multi-source heterogeneous data and a medium, and relates to the technical field of artificial intelligence and natural language processing. The method comprises the following steps: integrating a dynamic graph convolutional network and a hierarchical attention mechanism to construct a multi-modal document analysis engine; performing semantic structure analysis on the original heterogeneous document on the basis of a multi-modal document analysis engine to extract document structure features and content semantic features, and constructing an original document relationship model on the basis of the document structure features and the content semantic features; based on the original document relationship model, performing classification fusion on heterogeneous data in the original heterogeneous document to obtain a to-be-stored heterogeneous data corpus, and processing the to-be-stored heterogeneous data corpus by using a graph neural network to establish a cross-modal semantic association index; and based on the cross-modal semantic association index, performing classified storage on the to-be-stored heterogeneous data corpora by utilizing a preset heterogeneous database so as to complete knowledge base system construction of the multi-source heterogeneous data.

Description

Technical Field

[0001] This application relates to the technical fields of artificial intelligence and natural language processing, and particularly to a method, device, and medium for constructing a knowledge base system for multi-source heterogeneous data. Background Art

[0002] With the rapid development of artificial intelligence and big data technologies, the data accumulated and to be processed in various industries presents the characteristics of multi-source, heterogeneous, massive, and complex associations, including a large amount of structured tabular data, semi-structured data (such as XML, JSON), and unstructured text, images, documents, etc. Efficiently integrating these multi-source heterogeneous data and constructing a unified knowledge system is the basis for realizing advanced applications such as intelligent retrieval, knowledge Q&A, and decision support. At present, although intelligent applications based on technologies such as large language models have made remarkable progress in single-modal data processing, they still face severe challenges in the comprehensive processing and retrieval of complex multi-source heterogeneous data.

[0003] The existing technologies mainly have the following bottlenecks in constructing a multi-source heterogeneous knowledge base system: First, there are significant modal feature differences and semantic fragmentation problems in data from different sources and modalities (such as text, tables, images, document formats), and there is a lack of unified technical means for effective cross-modal semantic alignment and knowledge fusion, resulting in fragmentation between data and making it difficult to form a unified knowledge representation. Second, traditional knowledge base construction methods are usually designed for single data of a specific type or domain, and it is difficult to dynamically adapt to the complex relationship mining and evolution tracking of multi-source heterogeneous data, and the ability to analyze deep-level relationships such as reference networks, version iterations, and topic associations between documents is insufficient. Third, existing methods often use a single type of database in the knowledge storage and retrieval links, and cannot efficiently support the graph structure query reasoning of knowledge graphs, the semantic similarity retrieval of unstructured text, and the complex association analysis of structured data at the same time, resulting in low cross-modal comprehensive query efficiency and limited accuracy.

[0004] These technical bottlenecks greatly restrict the in-depth mining and utilization of the value contained in multi-source heterogeneous data and affect the effectiveness of intelligent retrieval and analysis applications. Therefore, how to construct a knowledge base system that can effectively integrate multi-source heterogeneous data, achieve cross-modal semantic unified representation, and support efficient collaborative query has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] Embodiments of this application provide a method, device, and medium for constructing a knowledge base system for multi-source heterogeneous data to solve the following technical problems: how to construct a knowledge base system that can effectively integrate multi-source heterogeneous data, achieve cross-modal semantic unified representation, and support efficient collaborative query.

[0006] In a first aspect, an embodiment of the present application provides a method for constructing a knowledge base system for multi-source heterogeneous data. The method includes: integrating a dynamic graph convolutional network and a hierarchical attention mechanism to construct a multimodal document parsing engine; based on the multimodal document parsing engine, performing semantic structure parsing on the original heterogeneous documents to extract document structure features and content semantic features, and constructing an original document relationship model based on the document structure features and content semantic features; based on the original document relationship model, classifying and fusing the heterogeneous data in the original heterogeneous documents to obtain a heterogeneous data corpus to be stored, and processing the heterogeneous data corpus to be stored using a graph neural network to establish a cross-modal semantic association index; where the heterogeneous data corpus to be stored includes knowledge graph data, text corpus data, images, and tabular data; based on the cross-modal semantic association index, using a preset heterogeneous database to classify and store the heterogeneous data corpus to be stored to complete the construction of the knowledge base system for multi-source heterogeneous data.

[0007] In an embodiment of the present application, performing semantic structure parsing on the original heterogeneous documents based on the multimodal document parsing engine specifically includes: using the dynamic graph convolutional network and combining the document structure attention mechanism to parse the layout information of the original heterogeneous documents to extract document structure features; using the dynamic graph convolutional network and combining the text semantic attention mechanism to parse the entities, themes, and semantic relationships of the text content in the original heterogeneous documents to extract content semantic features.

[0008] In an embodiment of the present application, the original document relationship model includes: a document citation network, a document evolution path, and a theme association graph; constructing the original document relationship model based on the document structure features and content semantic features specifically includes: using the dynamic graph convolutional network and combining the hierarchical attention mechanism to trace the citation relationships between documents to construct a document citation network; analyzing the version association information in the document citation network to determine the document evolution path; using the hierarchical attention mechanism to fuse the document structure features, content semantic features, and external domain knowledge graph information to identify key theme nodes and construct a cross-document theme association graph.

[0009] In one embodiment of the present application, based on the original document relationship model, heterogeneous data in the original heterogeneous documents is classified and fused to obtain the heterogeneous data corpus to be stored, specifically including: for knowledge graph data, constructing a graph classification model based on subgraph topological features and content semantic features, dynamically aggregating entity and its relationship information, and updating the knowledge graph subgraph features; for text corpus data, using a preset mixed-granularity text segmentation algorithm, combining the logical structure information in the document structure features and the semantic coherence information in the content semantic features, and segmenting the text content into paragraph-level, sentence-level, and phrase-level text blocks; for image and table data, using an improved ViLT model architecture, adding an embedding layer for processing table structure information, and designing a cross-modal alignment loss function to map the content semantics of images and tables to a vector representation space consistent with the text description, and generating chart semantic summaries.

[0010] In one embodiment of the present application, a graph neural network is used to process the heterogeneous data corpus to be stored to establish a cross-modal semantic association index, specifically including: inputting the knowledge graph subgraph features, different-granularity text block information, and chart semantic summary vectors into the graph neural network; using the graph neural network to model the potential semantic associations between the knowledge graph subgraph features, text block information, and chart semantic summary vectors; based on the modeling results, generating a cross-modal semantic association index connecting the knowledge graph data, text corpus data, and table data.

[0011] In one embodiment of the present application, based on the cross-modal semantic association index, a preset heterogeneous database is used to classify and store the heterogeneous data corpus to be stored, specifically including: storing the entities and their relationship networks included in the knowledge graph data into a graph database; using a pre-trained language model to convert the paragraph-level, sentence-level, and phrase-level text blocks into semantic vectors, and storing the semantic vectors into a vector database; normalizing the image and table data and storing them into a relational database.

[0012] In one embodiment of the present application, a dynamic graph convolutional network and a hierarchical attention mechanism are integrated to construct a multi-modal document parsing engine, specifically including: designing a main architecture of the dynamic graph convolutional network including a dynamic graph update strategy; in the dynamic graph convolutional network architecture, integrating a document structure attention module and a text semantic attention module respectively; introducing a hierarchical attention mechanism module, and configuring it to fuse the layout structure features and external domain knowledge information, and capture the cross-reference relationship, version iteration relationship, and theme association relationship between documents.

[0013] In one embodiment of the present application, the method further includes: based on the constructed knowledge base system of multi-source heterogeneous data, in response to a user query request, co-invoke a graph database, a vector database, and a relational database through cross-modal semantic association indexing; use a graph neural network to perform semantic fusion reasoning on the cross-modal semantic association indexing, and perform intelligent retrieval in combination with the query results of each database, and return a multi-source result feedback that integrates knowledge graph entities, semantically similar texts, and associated structured data.

[0014] In a second aspect, an embodiment of the present application further provides a device for constructing a knowledge base system of multi-source heterogeneous data. The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method for constructing a knowledge base system of multi-source heterogeneous data as described in any one of the above.

[0015] In a third aspect, an embodiment of the present application further provides a non-volatile computer storage medium storing computer-executable instructions, which when executed, implement a method for constructing a knowledge base system of multi-source heterogeneous data as described in any one of the above.

[0016] The method, device, and medium for constructing a knowledge base system of multi-source heterogeneous data provided by the embodiments of the present application have the following beneficial effects: by integrating a dynamic graph convolutional network and a hierarchical attention mechanism, in-depth parsing and relationship tracing of multi-modal documents are realized, and the problems of modal fragmentation and semantic fragmentation of multi-source heterogeneous data are solved; at the multi-modal semantic understanding layer, a unified cross-modal semantic association indexing is established by using mixed-granularity text segmentation, an improved cross-modal alignment model (ViLT), and graph dynamic aggregation technology, significantly improving the accuracy of semantic alignment and knowledge fusion ability between different modal data (text, table, image, knowledge graph); at the storage application layer, an innovative collaborative indexing mechanism of a graph database, a vector database, and a relational database is adopted, enabling the system to simultaneously have efficient knowledge graph reasoning, semantic similarity retrieval, and structured data association analysis capabilities, thereby comprehensively optimizing the representation, storage, and intelligent retrieval efficiency of multi-source heterogeneous data, providing a solid dynamic and structured knowledge base for applications such as intelligent question answering and data analysis based on large language models, and strongly supporting the in-depth mining of enterprise data value and the improvement of intelligent decision-making level. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0018] Figure 1Flowchart of a method for constructing a knowledge base system for multi-source heterogeneous data provided by an embodiment of the present application;

[0019] Figure 2 Schematic internal structure diagram of a device for constructing a knowledge base system for multi-source heterogeneous data provided by an embodiment of the present application. Detailed implementation manners

[0020] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present application.

[0021] The embodiments of the present application provide a method, device, and medium for constructing a knowledge base system for multi-source heterogeneous data, which are used to solve the following technical problems: how to construct a knowledge base system that can effectively integrate multi-source heterogeneous data, achieve cross-modal semantic unified representation, and support efficient collaborative query.

[0022] The technical solutions proposed in the embodiments of the present application will be described in detail below with reference to the drawings.

[0023] Figure 1 Flowchart of a method for constructing a knowledge base system for multi-source heterogeneous data provided by an embodiment of the present application. As Figure 1 shown, a method for constructing a knowledge base system for multi-source heterogeneous data provided by an embodiment of the present application specifically includes the following steps:

[0024] Step 101: Integrate a dynamic graph convolutional network and a hierarchical attention mechanism to construct a multi-modal document parsing engine.

[0025] In an embodiment of the present application, integrating a dynamic graph convolutional network and a hierarchical attention mechanism to construct a multi-modal document parsing engine specifically includes: designing a main architecture of a dynamic graph convolutional network including a dynamic graph update strategy; in the dynamic graph convolutional network architecture, integrating a document structure attention module and a text semantic attention module respectively; introducing a hierarchical attention mechanism module, and configuring it to fuse layout structure features and external domain knowledge information, and capture cross-reference relationships, version iteration relationships, and topic association relationships between documents.

[0026] In this embodiment, this step is the starting point for constructing the knowledge base system, aiming to create a powerful tool for parsing and understanding raw documents from multiple sources and formats (i.e., "multi-source heterogeneous"). The "Dynamic Graph CNN" (DGCN) is a specially designed graph neural network architecture, and its core advantage lies in its ability to handle the situation where the nodes (representing documents or their internal elements) and edges (representing relationships) in the graph change dynamically as new information is added or the analysis deepens, which is very suitable for the scenario of the evolution of citation relationships between documents. The "Hierarchical Attention Mechanism" is a technology that simulates the human cognitive process. It can dynamically allocate computing resources (attention weights) at different levels (e.g., the overall document level, chapter level, sentence level) and different types of information sources (such as document layout structure information, text semantic content, external domain knowledge graph), focusing on the parts that are more important for the current task (such as identifying key topics or citation relationships).

[0027] The specific process of constructing the multi-modal document parsing engine includes: First, design the main architecture of DGCN, and crucially incorporate a dynamic graph update strategy. This strategy means that during the parsing process, the structure of the internal representation graph of the network (the connections and weights between nodes) is not fixed, but will be adjusted and optimized in real-time or incrementally according to newly parsed document information (such as newly discovered references, new semantic associations), ensuring that the model can continuously capture the dynamic characteristics of the evolution of the document collection. Second, in the framework of DGCN, a document structure attention module and a text semantic attention module are respectively incorporated. The document structure attention module is specifically used to focus on the layout information of the document, such as the position of paragraphs, the hierarchy of headings, the presence of tables / illustrations and other visual or organizational features ("structural features"). Its role is to guide the model to understand the organizational logic of the document, so as to better extract structural information such as chapter division and document type. The text semantic attention module focuses on the text content itself, and is responsible for identifying key entities (such as names of people, places, technical terms), topic distributions, and semantic associations between sentences / paragraphs ("content semantic features"), and achieving in-depth semantic parsing by understanding the essence of the text. Finally, a hierarchical attention mechanism module is introduced and configured to play a high-level overall coordination role in the entire parsing engine. This module deeply integrates the layout structure features (from the structure attention module) and external domain knowledge graph information (a pre-constructed domain-related knowledge base, providing professional background concepts and relationships). Its core function is to systematically capture complex relationship types between documents: this includes identifying the cross-reference relationship where document A cites document B; tracking the version iteration relationship between different revisions of the same report (for example, the main change path of the document content from V1 to V2); and identifying the topic association relationship formed by different documents due to discussing the same theme or concept (for example, multiple white papers in the same industry all involve the theme of "artificial intelligence ethics").

[0028] For example, in an enterprise's scientific and technological literature repository, the engine can automatically parse technical specification documents in PDF format (identifying their chapter structures and internal references), project reports in Word format (tracking the updated content between multiple versions), product description web pages in HTML format (extracting link relationships and text descriptions), and research data tables in Excel format (parsing their table content and annotations). During the parsing process, the dynamic nature of DGCN allows it to establish citation relationships between a newly added review report and multiple historical patent documents it cites, and continuously update the network as more reports are added. The hierarchical attention mechanism ensures that when the engine understands that one report cites another (citation relationship), it not only focuses on the text context of the citation point (semantic level), but also considers the position of the cited report in the entire topic map (such as whether it belongs to a core basic theory document) and the format position of the citation marker in the document (such as in a reference list or a parenthetical citation in a paragraph), so as to more comprehensively understand the meaning and importance of the citation.

[0029] Step 102: Based on the multi-modal document parsing engine, perform semantic structure parsing on the original heterogeneous documents to extract document structure features and content semantic features, and construct an original document relationship model based on the document structure features and content semantic features.

[0030] In an embodiment of the present application, performing semantic structure parsing on the original heterogeneous documents based on the multi-modal document parsing engine specifically includes: using a dynamic graph convolutional network, combined with a document structure attention mechanism, to parse the layout information of the original heterogeneous documents to extract document structure features; using a dynamic graph convolutional network, combined with a text semantic attention mechanism, to parse the entities, themes, and semantic relationships of the text content in the original heterogeneous documents to extract content semantic features.

[0031] In an embodiment of the present application, the original document relationship model includes: a document citation network, a document evolution path, and a theme association graph; constructing the original document relationship model based on the document structure features and content semantic features specifically includes: using a dynamic graph convolutional network, combined with a hierarchical attention mechanism, to track the citation relationships between documents and construct a document citation network; analyzing the version association information in the document citation network to determine the document evolution path; using the hierarchical attention mechanism to fuse the document structure features, content semantic features, and external domain knowledge graph information to identify key theme nodes and construct a cross-document theme association graph.

[0032] In this embodiment, extracting document structure features includes: using the document structure attention mechanism within the DGCN framework described in step 101 to specifically analyze the layout information of the original heterogeneous documents (such as PDF, Word, HTML, Excel, etc.). This includes identifying and analyzing the visual and organizational elements in the document, such as page layout (column separation), chapter title hierarchy (heading 1, heading 2, etc.), paragraph indents and spacing, list structure, table position, chart annotation position, and formula numbering area, etc. Through the identification and analysis of this information, the macroscopic organizational structure of the document (such as the main part division of the document) and microscopic typesetting features (such as emphasized information marked with special formats) are extracted, collectively referred to as "document structure features". These features are crucial for understanding the logical framework of the document, identifying the importance of content, and the associated positions of non-text elements (such as charts) and text. For example, in a PDF of an annual report, the engine can accurately identify the table of contents structure, the location of financial statements and their headings, and abstract them into structure feature vectors.

[0033] Extracting content semantic features includes: using the text semantic attention mechanism within the DGCN framework described in step 101 to analyze the essential meaning of the text content in the original heterogeneous documents. This involves performing natural language processing, focusing on identifying key entities in the text (such as product models, technical terms, person names, organization names), exploring potential discussion topics (such as "market trends", "technological innovation"), and analyzing lexical and grammatical structures to understand semantic relations (such as the behavioral relationship, attribute relationship, contrast relationship, etc. between entities). Finally, the "content semantic features" that represent the core content information and internal language logic of the document are extracted. For example, when parsing a product technical document, the engine can identify the names of key product components (entities), the main technical points discussed (topics), and semantic relations such as "Component A depends on Component B".

[0034] Based on the above-extracted "document structure features" and "content semantic features", combined with the hierarchical attention mechanism integrated in step 101 (responsible for fusing structural and semantic information and paying attention to external knowledge), a relationship model that comprehensively reflects the internal connections of the original document set is constructed. This model consists of three core parts:

[0035] Document citation network: Using the DGCN (relying on its dynamic graph update characteristics), combined with the hierarchical attention mechanism (paying attention to information such as the position of citation marks, the semantics of citation contexts, and the topics of cited documents), to trace and model the explicit citation or reference relationships between documents (for example, Paper A cites Report B in the reference list or in the text). This process constructs a directed graph network, where nodes represent documents and edges represent citation relationships.

[0036] Document evolution path: By deeply analyzing the version association information (Versioning Information) in the document citation network (such as by identifying the version number in the document title, the revision time in the metadata, or the significant addition, deletion, and modification marks at the content level), determine and display the iterative update trajectory (i.e., the evolution path) of the same document or related document collection over time. For example, identify the process of revising an enterprise strategic planning document from version 2023 to version 2024 to form its historical version chain.

[0037] Topic association graph: Once again, utilize the hierarchical attention mechanism to actively fuse the "document structure features" (such as title and chapter levels) extracted in step 102, the "content semantic features" (especially the identified topics and key entities) extracted in step 102, and optional external domain knowledge graph information (providing the background concept system of the topic). Through this mechanism, identify the key topic nodes that are of core importance or have extensive relevance in the entire document collection (such as "5G technology", "supply chain security", etc.). Based on these key topic nodes and the discussion and mention relationships of the document content around them, mine and construct a topic association network graph (Topic Association Graph) that spans multiple document boundaries. In this graph, the nodes can be topics or documents, and the edges represent the strength of the document containing the topic or the co-occurrence between topics.

[0038] Step 103: Based on the original document relationship model, classify and fuse the heterogeneous data in the original heterogeneous documents to obtain the heterogeneous data corpus to be stored, and use a graph neural network to process the heterogeneous data corpus to be stored to establish a cross-modal semantic association index.

[0039] In the embodiments of this application, the heterogeneous data corpus to be stored includes knowledge graph data, text corpus data, images, and table data.

[0040] In an embodiment of this application, based on the original document relationship model, classify and fuse the heterogeneous data in the original heterogeneous documents to obtain the heterogeneous data corpus to be stored, which specifically includes: for the knowledge graph data, construct a graph classification model based on subgraph topological features and content semantic features, dynamically aggregate entity and its relationship information, and update the knowledge graph subgraph features; for the text corpus data, use a preset mixed-granularity text segmentation algorithm, combine the logical structure information in the document structure features and the semantic coherence information in the content semantic features to segment the text content into paragraph-level, sentence-level, and phrase-level text blocks; for the image and table data, use an improved ViLT model architecture, add an embedding layer for processing table structure information, and design a cross-modal alignment loss function to map the content semantics of the image and table to a vector representation space consistent with the text description to generate a chart semantic summary.

[0041] In one embodiment of the present application, a graph neural network is used to process the heterogeneous data corpus to be stored, so as to establish a cross-modal semantic association index, which specifically includes: inputting the knowledge graph subgraph features, text block information of different granularities, and chart semantic summary vectors into the graph neural network; using the graph neural network to model the potential semantic associations among the knowledge graph subgraph features, text block information, and chart semantic summary vectors; and generating a cross-modal semantic association index that connects the knowledge graph data, text corpus data, and tabular data based on the modeling results.

[0042] In this embodiment, this step takes the relationship model constructed in step 102 as the input and guidance, deeply processes the actual knowledge content (structured, semi-structured, and unstructured data) contained in the document, and fuses them into a unified semantic representation ("corpus"), and establishes an association index for different modal knowledge fragments through a graph neural network.

[0043] It should be noted that during the classification and fusion process, for the knowledge graph data (KG Data): In the original heterogeneous document, there may be embedded small knowledge graph fragments (such as the model-parameter relationship in an enterprise product document) or structured relationships formed through text extraction. Based on this, a graph classification model based on subgraph topological features (Subgraph Topological Feature) and content semantic features (from step 102) is constructed. This model analyzes the entity relationship connection patterns within the subgraph (such as star structure, chain structure), the connections between subgraphs (bridging relationships), and combines the semantic similarity of the content (such as subgraphs in documents discussing the same concept), and dynamically aggregates entity and their relationship information (such as merging semantically similar entities, refining core relationships), so as to update and form knowledge graph subgraph features that can better represent the overall knowledge structure.

[0044] When dealing with text corpus data (Text Data): To improve the retrievability and relevance of text knowledge, a preset Mixed-Granularity Text Segmentation Algorithm is adopted. This algorithm does not simply segment by a fixed number of characters. Instead, it comprehensively considers two aspects of features: logical structure information in document structure features, such as clear section / chapter boundaries, paragraph breaks, and list levels; and semantic coherence information in content semantic features, such as high co-occurrence rates of entities within a paragraph, natural topic shifts, and syntactic dependency relationships. Based on the combined judgment of these features, continuous text content is accurately segmented into three text blocks with different granularities: paragraph-level text blocks composed of larger meaning units (such as subsections); sentence-level text blocks that express complete meanings; and phrase-level text blocks that express precise concepts or keywords (such as proper noun phrases, verb phrases). This multi-level segmentation facilitates subsequent retrieval and analysis at different granularities.

[0045] When dealing with image and tabular data (Image&Tabular Data): Traditional methods are difficult to effectively integrate non-text visual information. To solve this problem, this solution adopts a multi-modal model built on the improved ViLT model architecture (Vision-and-Language Transformer). The key points of the improvement are: adding an embedding layer specifically for processing tabular structure information, which can understand the row and column structures of tables, the semantics of table headers, and the relationships between cell data (such as numerical sequences, classification labels). At the same time, a cross-modal alignment loss function is designed. The core function of this function is to drive the model to learn to map the visual content in images (such as bar charts, process diagrams) and the structured data information in tables to a vector representation space that is exactly the same as the semantic space used in text descriptions during the training process. Finally, the model can output a chart semantic summary (vector or feature representation) that represents the core semantic information of the chart.

[0046] Merge the results of the above processing to form the "heterogeneous data corpus to be stored". It mainly includes: updated knowledge graph sub-graph features (refined structured knowledge fragments), text block information after multi-granularity segmentation (paragraph, sentence, phrase levels), and chart semantic summaries (vector representations) converted from image / table data. These elements together constitute the knowledge "atoms" to be stored in the knowledge base.

[0047] To achieve deep semantic association queries between different modal data fragments (text blocks, knowledge sub-graphs, chart summaries), this step uses a graph neural network (GNN) to establish a cross-modal semantic association index.

[0048] Specifically, input the knowledge graph sub-graph features (features representing the entity and its relationship graph structure), processed text block information at different granularities (semantic feature vectors or position information at the paragraph, sentence, and phrase levels), and chart semantic summary vectors (semantic vectors of images / tables) in the heterogeneous data corpus to be stored into the graph neural network model. The graph neural network is good at processing the association information between nodes. Here, the GNN regards the above various types of input features (i.e., graph sub-graph nodes, text block nodes at different granularities, chart summary nodes) as nodes in a large "association graph" (even isolated text blocks or chart summaries are regarded as single nodes). Through its message passing and neighborhood aggregation mechanisms, the GNN models the potential semantic associations between these nodes (e.g., a paragraph describing experimental results may be highly relevant to the chart summary of the table showing experimental data; a phrase-level text block defining "blockchain" may be closely related to the "blockchain technology" sub-graph entity in the knowledge graph). Based on the calculation results of the GNN, the model can generate an association index that connects all knowledge graph data nodes, text corpus data nodes, and table / image data nodes, which is the so-called "cross-modal semantic association index". This index is not a simple keyword inverted index, but a complex network structure that can reflect the deep connections (such as synonymy, context relevance, part-whole, causality, etc.) between knowledge fragments in a multi-dimensional semantic space. It enables subsequent retrieval to perform association searches across data type boundaries.

[0049] Step 104: Based on the cross-modal semantic association index, use a preset heterogeneous database to classify and store the heterogeneous data corpus to be stored, so as to complete the construction of the knowledge base system for multi-source heterogeneous data.

[0050] In one embodiment of the present application, based on a cross-modal semantic association index, a preset heterogeneous database is used to classify and store the heterogeneous data corpus to be stored, specifically including: storing the entities and their relationship networks contained in the knowledge graph data in the graph database; using a pre-trained language model to convert paragraph-level, sentence-level, and phrase-level granular text blocks into semantic vectors, and storing the semantic vectors in the vector database; and normalizing the image and table data and storing them in the relational database.

[0051] In this embodiment, this step follows the "heterogeneous data corpus to be stored" generated in step 103 and its core link, the "cross-modal semantic association index," and is a key link in the final implementation of the knowledge base system. Its core goal is to persistently store heterogeneous knowledge content that has been deeply analyzed, integrated, and semantically associated in a dedicated database that best suits its characteristics and access patterns, thereby building an efficient, scalable, and multi-level knowledge base that supports complex queries. The specific implementation process is as follows:

[0052] To maximize storage efficiency and query performance, the present invention pre-sets and selects three types of heterogeneous databases for collaborative storage management based on the inherent characteristics of different types of knowledge in the "heterogeneous data corpus to be stored." First, for the knowledge graph data representing entities and their complex relationship networks (i.e., the knowledge graph subgraph features dynamically aggregated and updated by the graph classification model in step 103), a graph database (such as Neo4j) is selected for storage. The core advantage of graph databases lies in their native support for the Property Graph Model, which can efficiently store entities, relationships, and their properties, and directly map the network structure between entities. This storage method provides natural and powerful support for subsequent complex queries based on graph traversal (such as using SPARQL or Cypher query languages to find multi-hop relationship paths between entities) and graph reasoning calculations (such as path analysis, community discovery, and influence assessment). For example, in an enterprise knowledge base, company entities, product entities, technology entities, and the relationship networks between them, such as "belonging to," "collaboration," and "application," can be efficiently stored in a graph database, facilitating industry chain analysis or technology association mining.

[0053] Secondly, for the text corpus data obtained through the mixed-grained text segmentation algorithm (including multi-grained text blocks at the paragraph level, sentence level, phrase level, etc.), the core of its storage strategy lies in achieving efficient retrieval based on semantics rather than literal content. To this end, the present invention utilizes the powerful semantic understanding ability of pre-trained language models (such as BERT, RoBERTa, or their domain-fine-tuned versions) to encode (Embedding) the content of text blocks at each granularity into high-dimensional dense semantic vectors. These semantic vectors capture the deep semantic information of the text blocks. Subsequently, these semantic vectors are stored in a vector database (such as Milvus) optimized for high-dimensional vector similarity search. The core value of the vector database lies in its ability to construct an efficient index structure (such as based on the ANN approximate nearest neighbor algorithm), supporting the rapid retrieval of the most semantically similar entries among a large number of vectors. This enables the system to respond to users' natural language queries, directly find relevant text fragments according to semantic similarity (rather than traditional keyword matching) (for example, find all paragraphs that semantically describe "supply chain risk mitigation strategies"), and support clustering analysis of the text semantic space (for example, automatically classify all feedback sentences discussing "customer satisfaction improvement").

[0054] Finally, for the image and table data knowledge extracted from images and tables and processed through normalization (mainly embodied as structured or semi-structured information), a mature relational database (such as MySQL or PostgreSQL) is selected for storage. Before storage, image data usually needs to extract key descriptive text and metadata with the help of OCR technology or multi-modal models such as ViLT; table data needs to be normalized, including identifying table headers, determining data types, parsing row and column structures, and possibly converting them into a form suitable for storage in a relational model (such as data rows, columns, key-value pairs). Relational databases, with their mature, stable, and powerful capabilities in processing structured data, can efficiently store and manage this normalized data. Their core advantage lies in supporting the execution of complex query operations using standardized SQL query languages (such as aggregating and statistical SUM / AVG of sales data, multi-table joining JOIN to associate different data sources) and flexible cross-table association analysis (for example, associating the product specification table stored in the relational database with the customer order table to analyze the sales trends of products with different specifications).

[0055] Through the above classification storage strategy, the "heterogeneous data corpus to be stored" generated in step 103 - that is, including knowledge graph sub - graph features (graph database), multi - granularity text block semantic vectors (vector database), and normalized image table data (relational database) - is persistently stored in the database system that best suits its characteristics respectively. Importantly, these physically dispersed knowledge fragments do not exist in isolation. They are closely associated at the semantic level through the "cross - modal semantic association index" constructed in step 103. This index is like a global semantic map, recording the deep semantic connections between different databases and different types of data fragments (such as the association between a text describing experimental results and its corresponding data table summary, or the association between an entity definition and its node in the knowledge graph). Finally, this design of optimizing storage according to knowledge types (structured data stored in relational databases, semi - structured graph data stored in graph databases, and unstructured text semantics stored in vector databases) and achieving semantic unified association through cross - modal indexes together constitute a multi - level knowledge base system with dynamic fusion capabilities (corresponding to the "multi - type knowledge structure layer" clearly shown in the attached figure). This system successfully overcomes the fragmentation problems of multi - source heterogeneous data in physical storage and semantic association, provides a solid, efficient, and semantically interconnected knowledge base for advanced applications such as intelligent data querying and knowledge answering based on large language models (LLMs), and significantly improves the efficiency and intelligence level of information retrieval.

[0056] In an embodiment of the present application, the method further includes: based on the constructed knowledge base system of multi - source heterogeneous data, in response to a user query request, the graph database, vector database, and relational database are collaboratively called through the cross - modal semantic association index; the graph neural network is used to perform semantic fusion reasoning on the cross - modal semantic association index, and intelligent retrieval is performed in combination with the query results of each database, and a multi - source result feedback integrating knowledge graph entities, semantically similar texts, and associated structured data is returned.

[0057] In this embodiment, relying on the multi-level knowledge base system successfully constructed in steps 101 to 104 (i.e., including the original document relationship layer, the basic semantic layer, and the multi-type knowledge structure layer), an intelligent and integrated response to complex user query requests is realized, fully demonstrating the core value of this knowledge base system in breaking information barriers and improving retrieval effectiveness. When a user submits a query request (usually in the form of a natural language question or a mixed instruction), the system initiates a collaborative intelligent retrieval process, which is driven by the "cross-modal semantic association index" as the core link. The system first parses the user query and uses the cross-modal semantic association index established in step 103 to understand the core intention of the query and the different knowledge types potentially involved. This is similar to locating the key areas that the query may touch with the help of a global semantic map. Guided by the index, the system collaboratively and on-demand invokes three types of heterogeneous databases deployed in step 104: The graph database (such as Neo4j) is responsible for handling the query part that requires graph traversal and relationship reasoning (for example, tracing the multi-level equity relationship chain among enterprises); the vector database (such as Milvus) deals with the part that requires in-depth semantic understanding or similarity matching (for example, finding text descriptions that are close in meaning to "market risk assessment methods"); and the relational database (such as MySQL or PostgreSQL) performs tasks that require precise calculations and structured data extraction (for example, summarizing sales data within a specific time period or executing a JOIN query on related tables). This on-demand scheduling mechanism ensures the optimal efficiency of each performing its own functions.

[0058] After obtaining the preliminary sub-result sets from various databases, the retrieval process does not end but enters a deep semantic fusion and reasoning stage. At this time, the system invokes the graph neural network (GNN) model used when constructing the index in step 103 again. This model runs on the cross-modal semantic association index, a global semantic network, and comprehensively processes the current user query intention, the heterogeneous result fragments returned by each database (these fragments are regarded as nodes in the index), and the rich association relationships contained in the index itself. Through its powerful information transmission and aggregation capabilities, GNN performs deeper semantic fusion and reasoning on these preliminary results. It can reveal the hidden deep connections between different result fragments (such as identifying the causal relationship between a text describing experimental results and a corresponding data table), eliminate the ambiguity between results (such as distinguishing the meanings of the same term in different contexts), and even complete the information chain (such as inferring the missing key links), thereby generating a more comprehensive and coherent knowledge view than simply adding the result sets.

[0059] Based on the fusion inference results of the GNN, the system conducts the final integration of intelligent retrieval results. This process combines the deep semantic association conclusions derived from the fusion inference of the graph neural network and the specific query result data directly returned by each database. The system will perform intelligent sorting and refinement based on relevance, confidence, and the degree of satisfaction of the user's query intent, filtering out redundancies and strengthening core associated information. Finally, the system returns to the user a multi-source result feedback that integrates knowledge graph entities and their relationships, semantically similar text content, and associated structured data. This feedback completely changes the presentation method of traditional fragmented information, and an answer can organically integrate various knowledge forms. For example, in response to a complex query such as "Company X's technology layout in the field of intelligent driving", the feedback may include: the network of Company X's key technology partners and their subsidiaries returned from the graph database (showing its technology ecosystem), the detailed description paragraphs of Company X's R & D strategy for intelligent driving in annual reports or technical patents retrieved from the vector database (providing background explanations and qualitative analysis), and the financial data table of the number of patent applications related to autonomous driving by Company X and the key technology investments accurately extracted from the relational database (providing specific quantitative support). Such retrieval results that break through modal boundaries and deeply integrate knowledge significantly improve the breadth, depth, and accuracy of information acquisition, fully verifying the powerful capabilities of the multi-level knowledge base system constructed by the present invention in supporting a new generation of intelligent applications based on large language models (such as advanced knowledge Q&A and intelligent data query).

[0060] The above is the method embodiment proposed in this application. Based on the same inventive concept, the embodiment of this application also provides a device for constructing a knowledge base system for multi-source heterogeneous data, and its structure is as Figure 2 shown.

[0061] Figure 2 It is a schematic internal structure diagram of a device for constructing a knowledge base system for multi-source heterogeneous data provided by the embodiment of this application. As Figure 2 shown, the device includes:

[0062] At least one processor 201;

[0063] And a memory 202 communicatively connected to the at least one processor;

[0064] Wherein, the memory 202 stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor 201 so that the at least one processor 201 can:

[0065] Integrate a dynamic graph convolutional network and a hierarchical attention mechanism to construct a multi-modal document parsing engine; based on the multi-modal document parsing engine, perform semantic structure parsing on the original heterogeneous documents to extract document structure features and content semantic features, and construct an original document relationship model based on the document structure features and content semantic features; based on the original document relationship model, classify and fuse the heterogeneous data in the original heterogeneous documents to obtain a heterogeneous data corpus to be stored, and use a graph neural network to process the heterogeneous data corpus to be stored to establish a cross-modal semantic association index; wherein, the heterogeneous data corpus to be stored includes knowledge graph data, text corpus data, images and table data; based on the cross-modal semantic association index, use a preset heterogeneous database to classify and store the heterogeneous data corpus to be stored to complete the construction of a knowledge base system for multi-source heterogeneous data.

[0066] Some embodiments of the present application provide a non-volatile computer storage medium corresponding to Figure 1 and storing computer-executable instructions, where the computer-executable instructions are set as:

[0067] Integrate a dynamic graph convolutional network and a hierarchical attention mechanism to construct a multi-modal document parsing engine; based on the multi-modal document parsing engine, perform semantic structure parsing on the original heterogeneous documents to extract document structure features and content semantic features, and construct an original document relationship model based on the document structure features and content semantic features; based on the original document relationship model, classify and fuse the heterogeneous data in the original heterogeneous documents to obtain a heterogeneous data corpus to be stored, and use a graph neural network to process the heterogeneous data corpus to be stored to establish a cross-modal semantic association index; wherein, the heterogeneous data corpus to be stored includes knowledge graph data, text corpus data, images and table data; based on the cross-modal semantic association index, use a preset heterogeneous database to classify and store the heterogeneous data corpus to be stored to complete the construction of a knowledge base system for multi-source heterogeneous data.

[0068] The embodiments in the present application are all described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the embodiments of the Internet of Things devices and media, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0069] The systems and media provided by the embodiments of the present application correspond one-to-one with the methods. Therefore, the systems and media also have beneficial technical effects similar to those of the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be elaborated here.

[0070] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0071] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more of the processes or Figure 1 blocks or combinations of blocks.

[0072] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one or more of the processes or Figure 1 blocks or combinations of blocks.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the processes or Figure 1 blocks or combinations of blocks.

[0074] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0075] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flashRAM). The memory is an example of computer-readable media.

[0076] A computer-readable medium includes permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0077] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0078] The above description is only for the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for constructing a knowledge base system for multi-source heterogeneous data, characterized in that, The method includes: Integrating a dynamic graph convolutional network and a hierarchical attention mechanism to construct a multimodal document parsing engine; Based on the multimodal document parsing engine, performing semantic structure parsing on the original heterogeneous document to extract document structure features and content semantic features, and constructing an original document relationship model based on the document structure features and the content semantic features; Based on the original document relationship model, classifying and fusing the heterogeneous data in the original heterogeneous document to obtain a heterogeneous data corpus to be stored, and processing the heterogeneous data corpus to be stored using a graph neural network to establish a cross-modal semantic association index; wherein, the heterogeneous data corpus to be stored includes knowledge graph data, text corpus data, images, and table data; Based on the cross-modal semantic association index, classifying and storing the heterogeneous data corpus to be stored using a preset heterogeneous database to complete the construction of a knowledge base system for multi-source heterogeneous data.

2. The method for constructing a knowledge base system for multi-source heterogeneous data according to claim 1, wherein Based on the multimodal document parsing engine, performing semantic structure parsing on the original heterogeneous document, specifically including: Using the dynamic graph convolutional network and combining with a document structure attention mechanism to parse the layout information of the original heterogeneous document to extract document structure features; Using the dynamic graph convolutional network and combining with a text semantic attention mechanism to parse the entities, themes, and semantic relationships of the text content in the original heterogeneous document to extract content semantic features.

3. The method for constructing a knowledge base system for multi-source heterogeneous data according to claim 2, wherein The original document relationship model includes: a document citation network, a document evolution path, and a theme association graph; Based on the document structure features and the content semantic features, constructing an original document relationship model, specifically including: Using the dynamic graph convolutional network and combining with the hierarchical attention mechanism to trace the citation relationships between documents and construct a document citation network; Analyzing the version association information in the document citation network to determine the evolution path of the document; Using the hierarchical attention mechanism to fuse the document structure features, the content semantic features, and external domain knowledge graph information to identify key theme nodes and construct a cross-document theme association graph.

4. A method for constructing a knowledge base system for multi-source heterogeneous data according to claim 2, characterized in that, Based on the original document relationship model, classifying and fusing the heterogeneous data in the original heterogeneous document to obtain a heterogeneous data corpus to be stored, specifically including: For knowledge graph data, constructing a graph classification model based on subgraph topological features and the content semantic features, dynamically aggregating entity and relationship information, and updating the knowledge graph subgraph features; For text corpus data, using a preset mixed-granularity text segmentation algorithm, combining the logical structure information in the document structure features and the semantic coherence information in the content semantic features, and segmenting the text content into paragraph-level, sentence-level, and phrase-level text blocks; For image and table data, using an improved ViLT model architecture, adding an embedding layer for processing table structure information, and designing a cross-modal alignment loss function to map the content semantics of images and tables to a vector representation space consistent with text descriptions to generate a chart semantic summary.

5. The method for constructing a knowledge base system for multi-source heterogeneous data according to claim 4, characterized in that, Using a graph neural network to process the heterogeneous data corpus to be stored to establish a cross-modal semantic association index, specifically including: Input the knowledge graph sub-graph features, text block information at different granularities, and the chart semantic summary vector into a graph neural network; Use the graph neural network to model the potential semantic associations among the knowledge graph sub-graph features, text block information, and the chart semantic summary vector; Based on the modeling results, generate a cross-modal semantic association index that connects the knowledge graph data, text corpus data, and table data.

6. A method for constructing a knowledge base system for multi-source heterogeneous data according to claim 4, characterized in that Based on the cross-modal semantic association index, classify and store the heterogeneous data corpus to be stored using a preset heterogeneous database, specifically including: Store the entities and their relationship networks included in the knowledge graph data into a graph database; Use a pre-trained language model to convert the text blocks at the paragraph level, sentence level, and phrase level into semantic vectors, and store the semantic vectors into a vector database; Normalize the image and table data and then store them into a relational database.

7. A method for constructing a knowledge base system for multi-source heterogeneous data according to claim 1, characterized in that, Integrate a dynamic graph convolutional network and a hierarchical attention mechanism to construct a multi-modal document parsing engine, specifically including: Design the main architecture of the dynamic graph convolutional network that includes a dynamic graph update strategy; In the dynamic graph convolutional network architecture, incorporate a document structure attention module and a text semantic attention module respectively; Introduce the hierarchical attention mechanism module, and configure it to fuse layout structure features and external domain knowledge information, and capture the cross-reference relationships, version iteration relationships, and topic association relationships among documents.

8. A method for constructing a knowledge base system for multi-source heterogeneous data according to claim 6, characterized in that, The method further includes: Based on the constructed knowledge base system of multi-source heterogeneous data, in response to a user query request, cooperatively call the graph database, the vector database, and the relational database through the cross-modal semantic association index; Use the graph neural network to perform semantic fusion reasoning on the cross-modal semantic association index, and perform intelligent retrieval in combination with the query results of each database, and return a multi-source result feedback that integrates knowledge graph entities, semantically similar texts, and associated structured data.

9. A knowledge base system construction device for multi-source heterogeneous data, characterized in that, The device includes: At least one processor; And a memory communicatively connected to the at least one processor; Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method for constructing a knowledge base system of multi-source heterogeneous data according to any one of claims 1-8.

10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, a method for constructing a knowledge base system of multi-source heterogeneous data according to any one of claims 1-8 is implemented.

Images