A heterogeneous data fusion indexing method based on multi-source knowledge graphs
By using a heterogeneous data fusion indexing method based on multi-source knowledge graphs, the problems of cross-database semantic alignment and hidden association mining were solved, achieving efficient data consistency management and real-time retrieval, improving query efficiency and accuracy, and reducing system operation and storage costs.
Patent Information
- Application Number
- CN202611276382.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-21
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies suffer from problems such as low accuracy of cross-database semantic alignment, insufficient discovery of hidden associations, inconsistency of multiple independent indexes, single retrieval and sorting dimensions, and low efficiency of full index updates, which cannot meet the real-time ingestion requirements of massive heterogeneous data.
It adopts a multi-source knowledge graph approach, realizes semantic fusion of the same concept across databases through a concept mapping model, uses an improved TransH algorithm to mine N-hop indirect relationships, constructs a three-layer graph native index and stores cold and hot data in a hierarchical manner, and combines precise ID matching, semantic similarity and topological importance scores for retrieval ranking, supporting dynamic incremental updates.
It improves the accuracy of cross-database concept matching, ensures the global uniqueness of entity IDs, enhances query efficiency and data consistency, improves retrieval accuracy and recall, and reduces system operation and maintenance complexity and storage costs.
Smart Images

Figure CN123115472A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of heterogeneous data fusion indexing methods, specifically a heterogeneous data fusion indexing method based on multi-source knowledge graphs. Background Technology
[0002] With the rapid implementation of big data and artificial intelligence technologies, various industries have accumulated a large amount of heterogeneous data from different business systems and collection channels. This data is scattered and stored in independent global ontology libraries and multiple domain ontology libraries. Different data sources often use different naming methods for the same entity or concept, and there is a lack of unified semantic alignment standards between cross-database data.
[0003] Traditional heterogeneous data fusion methods mostly rely on a single string matching rule to complete entity alignment. The matching accuracy is greatly affected by aliases, abbreviations, and domain-specific terms, which can easily lead to the same entity being identified as multiple different objects. This makes it impossible to achieve deep normalization of cross-database data. At the same time, conventional global ID allocation schemes require the deployment of independent distributed ID generation services, which not only increases the complexity of system operation and maintenance, but also easily leads to ID conflicts in high-concurrency data ingestion scenarios, seriously affecting the accuracy of unique identification of cross-source entities.
[0004] In existing heterogeneous data fusion processes, most can only uncover explicit relationships directly labeled between heterogeneous data, failing to effectively discover N-hop indirect hidden relationships across data sources. A large number of valuable relationship edges are missed, resulting in significant structural sparsity in the final fused graph. Conventional graph reasoning models mostly use the basic TransE algorithm, which cannot effectively handle complex semantic scenarios where different relationships are located on different hyperplanes in entity space. The accuracy of relationship prediction is low, and the lack of multi-source cross-validation makes it easy to introduce erroneous and invalid relationship edges into the fused graph, further reducing the overall quality of the graph data.
[0005] Currently, the industry commonly uses a binary retrieval architecture that combines vector indexes and inverted indexes. Structured data, edge relation data, and unstructured text data are stored in three separate index systems. Data synchronization between different indexes relies on scheduled batch processing tasks, which easily leads to data inconsistency issues. Furthermore, traditional index architectures do not perform layered optimization for the native graph structure, and single entity location typically takes more than 10ms. Query efficiency in high-frequency access scenarios cannot meet business needs, and the inability to store hot and cold data in a layered manner results in a large amount of historical cold data consuming valuable memory resources, significantly increasing the system's storage costs.
[0006] Conventional fusion retrieval ranking schemes only calculate result weights based on two dimensions: keyword precision matching score and semantic vector similarity score. This completely ignores the importance of the knowledge graph's topological structure, failing to identify high-value core entities within the graph. This often results in low-relevance results being ranked higher while high-value entities are buried. Furthermore, traditional schemes require a full rebuild of all indexes when new data is added, unable to support local dynamic incremental updates. The time required for index updates increases linearly with data volume, making it difficult to adapt to the real-time ingestion requirements of massive amounts of heterogeneous data.
[0007] Therefore, there is an urgent need for a new heterogeneous data fusion indexing method based on multi-source knowledge graphs to solve the problems of low cross-database semantic alignment accuracy, insufficient hidden association mining, inconsistency of multiple independent index data, single retrieval and sorting dimension, and low efficiency of full index update in the existing technologies. Summary of the Invention
[0008] The purpose of this invention is to provide a heterogeneous data fusion indexing method based on multi-source knowledge graphs. The method comprises the following steps:
[0009] Step 1: Semantic Alignment and Entity Normalization of Multi-Source Heterogeneous Data: Collect multi-source heterogeneous data from the global ontology library and at least two domain ontology libraries. Construct a concept mapping model to achieve semantic fusion of the same concept with different names across libraries. Assign a globally unique ID to each fused data point. The generation rule for the unique ID in the global ontology library is domain identifier + timestamp + random check bit. The domain identifier is used to mark the original data source type to which the entity belongs, the timestamp is used to record the time when the entity was first entered into the library, and the random check bit is used to avoid ID duplication and conflict, ensuring the global uniqueness of cross-source entity IDs. The matching degree calculation formula of the concept mapping model is:
[0010] ;
[0011] in, A concept in the global ontology library. For concepts in the domain ontology library, For string similarity, For conceptual structural similarity, For domain business rule similarity, All are weighted values. The weights of the three are dynamically adjusted according to different business scenarios. When the matching degree is greater than the preset threshold of 0.85, they are judged to be the same concept.
[0012] Step Two: Hidden Association Mining and Fusion Graph Construction of Heterogeneous Data: Based on a representation learning graph reasoning model of knowledge graph, this step mines N-hop indirect associations between different heterogeneous data, fills in the missing association edges in traditional fusion methods, and constructs a unified knowledge graph after full data fusion. The representation learning graph reasoning model adopts the TransH improved algorithm, and the scoring function for entity relationships is:
[0013] ;
[0014] in For the head entity vector, For the tail entity vector, For relation vectors, For relationship The corresponding normal vector is used to predict the association by minimizing the loss value of the scoring function, and hidden indirect associations with a confidence level greater than 0.7 are discovered.
[0015] After the hidden relationship mining is completed, a new relationship confidence verification step is added. The authenticity of the relationship is determined by cross-validation of at least two different domain ontology libraries. Invalid relationship edges with a confidence of less than 0.7 are removed to avoid introducing incorrect relationship information into the fused graph.
[0016] Step 3: Layered Construction of the Three-Layer Graph Native Index: A point attribute index is constructed using the fused unified knowledge graph as the underlying data carrier. Heterogeneous data associated with the unified knowledge graph is fused and used as edge data to construct an edge relationship index. The original multi-source heterogeneous unstructured data is used as the full-text search index, completing the collaborative construction of the three-layer graph native index. The query efficiency optimization formula for the point attribute index is:
[0017] ;
[0018] in, The time required for direct location of a globally unique ID. Number of filter conditions for attributes. To reduce the time spent filtering single attributes, this formula optimizes the query path of the point index, controlling the time spent locating a single entity to within 1ms.
[0019] A cold and hot data tiered storage strategy is adopted. The point index and edge index of the frequently accessed entities in the past three months are stored in the memory cache, the historical full index data is stored in SSD storage, and the original unstructured full-text index data is stored in object storage.
[0020] Step 4: Keyword Fusion Retrieval and Weighted Ranking: The user-input main keyword is mapped to a globally unique ID to complete precise retrieval, and fuzzy keywords are converted into knowledge embedding vectors to complete semantic similarity retrieval. The two search results are weighted and ranked according to pre-set reliability weights, and finally, the fused retrieval result is output, realizing the collaborative support of fuzzy keyword and precise ID retrieval. The comprehensive score calculation formula for the weighted ranking is as follows:
[0021] ;in, A score is awarded for a precise match to a globally unique ID. To score the semantic similarity of fuzzy keywords, Assign a score to the topological importance of an entity in the knowledge graph. Finally, the search results are output in descending order of comprehensive score. After the two search results are merged, the entities are automatically deduplicated using the globally unique ID. If the same entity is simultaneously matched by both precise ID search and semantic vector search, the comprehensive ranking weight of the entity is automatically increased to ensure that highly relevant results are displayed first.
[0022] Step 5: Dynamic Incremental Update: When new multi-source heterogeneous data is added to the database, there is no need to rebuild the three-layer graph native index in its entirety. Instead, only the newly added entities and relationships are updated locally, ensuring the real-time performance of the merged index.
[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0024] This invention achieves semantic fusion of different names for the same concept across different databases by setting up a concept mapping model. It breaks through the entity alignment approach of single string matching and dynamically weights three dimensions: string similarity, concept structure similarity, and domain business rule similarity. Compared with traditional single matching methods, the accuracy of cross-database concept matching in this application is significantly improved. Furthermore, it adopts a custom rule of "domain identifier + timestamp + random check bit" to ensure the global uniqueness of cross-source entity IDs from the root, eliminating the need to deploy an independent ID allocation service and greatly reducing the complexity of system operation and maintenance.
[0025] This invention breaks through the traditional binary retrieval architecture that combines vector indexes and inverted indexes. For the first time, it deeply binds point attribute indexes, edge relation indexes, and full-text retrieval indexes onto a unified knowledge graph carrier, realizing integrated indexing of structured related data, unstructured text, and vector semantic data, thus avoiding data consistency issues between multiple independent indexing systems.
[0026] This invention differs from the conventional keyword + vector two-way fusion ranking scheme by adding a knowledge graph topology importance score dimension. It calculates the final score by comprehensively considering three dimensions: accurate ID matching, semantic similarity, and graph topology weight, which not only ensures the accuracy of precise retrieval but also takes into account the recall rate of fuzzy semantic retrieval. Attached Figure Description
[0027] Figure 1 is a flowchart of the method of the present invention. Detailed Implementation
[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0029] As attached Figure 1 As shown, the present invention provides a heterogeneous data fusion indexing method based on multi-source knowledge graphs, and specific embodiments are as follows:
[0030] The application scenario of this embodiment is a provincial government data sharing platform. The platform simultaneously connects to three independent domain business databases: the public security household registration database, the human resources and social security database, and the real estate registration database. Previously, the data in the three databases were completely independent, and the identity identifiers and attribute field names of the same natural person were completely inconsistent. Traditional relational database association queries took more than 12 seconds, which could not meet the business requirements of "second-level response" for government services. Therefore, this method is used to complete the heterogeneous data fusion index transformation.
[0031] Step 1: Semantic alignment and entity normalization of multi-source heterogeneous data: First, sort out the global government ontology database, and then import the full metadata of the three domain ontology databases of public security household registration, human resources and social security, and real estate registration. First, include all the different names of natural persons in the three databases, such as "XM" in the public security database, "GRXM" in the human resources and social security database, and "YHXM" in the real estate database, into the concept mapping model calculation.
[0032] According to the preset similarity formula, string similarity The edit distance calculation yielded a similarity of 0.94, indicating a conceptual structure similarity. By comparing the field length, data type, and non-null constraint of the three fields, a similarity of 0.92 was obtained, indicating a similarity of domain business rules. The value was 0.97 after verification using government data standards and specifications. Substituting α=0.4, β=0.3, and γ=0.3 into the formula:
[0033]
[0034] After calculation, the final comprehensive matching degree is 0.943, which is much higher than the preset threshold of 0.85. It is determined that the three are the same concept and semantic fusion is completed. Then, according to the globally unique ID generation rules, a global ID with domain identifier bit + timestamp + random check bit is assigned to each natural person entity. For example, the ID prefix of natural person entity in the public security household registration database is set to GA, that in the human resources and social security database is set to RS, and that in the real estate registration database is set to BDC. This marks the original data source of the entity and completely avoids cross-database ID duplication through timestamp and 6-bit random check bit.
[0035] Step Two: Hidden Association Mining and Fusion Graph Construction in Heterogeneous Data: We selected the improved TransH algorithm as the representation learning graph inference model. First, we imported all the explicit association edges from the three libraries into the model for training. The training epochs were set to 800, and the batch size to 256. After training, the association confidence between entities was calculated using a scoring function, uncovering a large number of previously missed N-hop indirect associations.
[0036] For example, the two-hop relationship between a natural person's social security participation unit and the real estate registration record under that unit's name was previously impossible to link directly among three independent business databases. This time, a total of 720,000 such hidden relationship edges were discovered. All the discovered relationship edges entered the cross-validation stage. At the same time, cross-validation was performed by calling two data sources from different fields: unit registration information from the public security database and participation relationship information from the human resources and social security database. A total of 41,000 invalid relationship edges with a confidence level of less than 0.7 were directly removed. The final unified government knowledge graph contains 12 million natural person entities, 2.3 million institutional entities, and 120 million valid relationship edges, without introducing any incorrect relationship information.
[0037] Step 3: Layered Construction of the Three-Layer Graph Native Index: Using the integrated unified knowledge graph as the underlying carrier, first construct the point attribute index, incorporating the globally unique IDs and corresponding attribute fields of all natural persons and institutional entities into the point index management; then, incorporate all heterogeneous edge data such as social security relationships and real estate registration relationships across databases into the edge relationship index; finally, incorporate all unstructured raw data such as scanned copies of ID cards, social security payment vouchers, and scanned copies of real estate ownership certificates stored in the three business databases into the full-text search index;
[0038] At the same time, a cold and hot data tiered storage strategy was implemented: the point index and edge index of 2.1 million entities that were frequently accessed by government services in the past three months were all stored in memory cache, the historical full index data was stored in SSD storage, and the original unstructured full-text index data was stored in object storage. This not only controlled hardware costs but also ensured the response speed of high-frequency access scenarios.
[0039] Step 4: Keyword Fusion Retrieval and Weighted Ranking: After the platform goes live, when users enter search keywords, such as "Zhang San," "social security," and "real estate," the system first maps out the globally unique ID of "Zhang San" that matches precisely, completing the precise ID retrieval to obtain the first result. Then, it converts the fuzzy keywords "social security" and "real estate" into knowledge embedding vectors, completing the semantic similarity retrieval to obtain the second result.
[0040] Both results are imported into the comprehensive score calculation formula for calculation, where Set to 0.5 Set to 0.3 Setting it to 0.2 gives the highest weight to the global ID exact match score, ensuring that accurate search results are displayed first. It also incorporates semantic similarity and entity topological importance scores. If the same individual entity matches both the exact ID search and the semantic vector search, the system will automatically increase the entity's ranking weight, ultimately outputting results in descending order of the overall score.
[0041] Step 5: Dynamic Incremental Update: During the daily operation of the platform, when new data such as household registration transfer, social security enrollment, and real estate registration are added to the database, the system does not need to rebuild the entire three-level index. It only completes the partial index update for the newly added entities and their corresponding related edges, which does not affect the platform's online retrieval service and ensures the real-time performance of the fused index.
[0042] Any technical solution that achieves the above-mentioned technical effects by utilizing the technical solutions described in this invention, or by designing similar technical solutions by those skilled in the art under the inspiration of the technical solutions described in this invention, falls within the protection scope of this invention.
Claims
1. A heterogeneous data fusion indexing method based on multi-source knowledge graphs, characterized in that: The steps of this method are as follows: Step 1: Semantic alignment and entity normalization of multi-source heterogeneous data: Collect multi-source heterogeneous data from the global ontology library and at least two domain ontology libraries, construct a concept mapping model to complete the semantic fusion of the same concept with different names across libraries, and assign a globally unique ID to each fused data. Step 2: Hidden Association Mining and Fusion Graph Construction of Heterogeneous Data: Based on the representation learning graph reasoning model of knowledge graph, we mine the N-hop indirect associations between different heterogeneous data, fill in the association edges missed by traditional fusion methods, and construct a unified knowledge graph after the fusion of all data. Step 3: Layered construction of the three-layer graph native index: The point attribute index is constructed using the fused unified knowledge graph as the underlying data carrier, the edge relationship index is constructed using the fused heterogeneous data associated with the unified knowledge graph as edge data, and the original multi-source heterogeneous unstructured data is used as the full-text search index to complete the collaborative construction of the three-layer graph native index; Step 4: Keyword Fusion Retrieval and Weighted Ranking: The user-input main keyword is mapped to a globally unique ID to complete the precise retrieval, and the fuzzy keyword is converted into a knowledge embedding vector to complete the semantic similarity retrieval. The two search results are weighted and ranked according to the pre-set reliability weights, and finally the fusion retrieval result is output, realizing the collaborative support of fuzzy keyword and precise ID retrieval. Step 5: Dynamic Incremental Update: When new multi-source heterogeneous data is added to the database, there is no need to rebuild the three-layer graph native index in its entirety. Instead, only the newly added entities and relationships are updated locally, ensuring the real-time performance of the merged index.
2. The heterogeneous data fusion indexing method based on multi-source knowledge graphs according to claim 1, characterized in that: The formula for calculating the matching degree of the concept mapping model in step one is as follows: ; in, A concept in the global ontology library. For concepts in the domain ontology library, For string similarity, For conceptual structural similarity, For domain business rule similarity, All are weighted values. The weights of the three are dynamically adjusted according to different business scenarios. When the matching degree is greater than the preset threshold of 0.85, they are judged to be the same concept.
3. The heterogeneous data fusion indexing method based on multi-source knowledge graphs according to claim 1, characterized in that: The representation learning graph reasoning model employs an improved TransH algorithm, and the scoring function for entity relations is: ; in For the head entity vector, For the tail entity vector, For relation vectors, For relationship The corresponding normal vector is used to predict the association by minimizing the loss value of the scoring function, and hidden indirect associations with a confidence level greater than 0.7 are discovered.
4. The heterogeneous data fusion indexing method based on multi-source knowledge graphs according to claim 1, characterized in that: The query efficiency optimization formula for the point attribute index is as follows: ; in, The time required for direct location of a globally unique ID. Number of filter conditions for attributes. To reduce the time spent filtering single attributes, this formula optimizes the query path of the point index, controlling the time spent locating a single entity to within 1ms.
5. The heterogeneous data fusion indexing method based on multi-source knowledge graphs according to claim 1, characterized in that: The formula for calculating the overall score of the weighted ranking is as follows: ;in, A score is awarded for a precise match to a globally unique ID. To score the semantic similarity of fuzzy keywords, Assign a score to the topological importance of an entity in the knowledge graph. Finally, the search results are output in descending order of the overall score.
6. The heterogeneous data fusion indexing method based on multi-source knowledge graphs according to claim 1, characterized in that: The unique ID of the global ontology is generated by the following rule: domain identifier + timestamp + random check bit. The domain identifier is used to mark the original data source type to which the entity belongs, the timestamp is used to record the time when the entity was first entered into the database, and the random check bit is used to avoid ID duplication and conflict, thus ensuring the global uniqueness of cross-source entity IDs.
7. The heterogeneous data fusion indexing method based on multi-source knowledge graphs according to claim 1, characterized in that: After the hidden relationship mining is completed in step two, a new relationship confidence verification step is added. The authenticity of the relationship is determined by cross-validation of at least two different domain ontology libraries, and invalid relationship edges with a confidence of less than 0.7 are removed to avoid introducing incorrect relationship information into the fused graph.
8. The heterogeneous data fusion indexing method based on multi-source knowledge graphs according to claim 1, characterized in that: In step three, the three-layer graph native index adopts a cold and hot data tiered storage strategy. The point index and edge index of the frequently accessed entities in the past three months are stored in the memory cache, while the historical full index data is stored in SSD storage and the original unstructured full-text index data is stored in object storage.
9. The heterogeneous data fusion indexing method based on multi-source knowledge graphs according to claim 1, characterized in that: After the two search results are merged, entities are automatically deduplicated using globally unique IDs. If the same entity is matched by both precise ID search and semantic vector search, the overall ranking weight of that entity is automatically increased to ensure that highly relevant results are displayed first.
