Knowledge graph-based heterogeneous database construction method

Abstract

The present application belongs to the technical field of knowledge graph construction and database construction, and particularly relates to a heterogeneous database construction method based on a knowledge graph, aiming to solve the problems of the existing heterogeneous database construction method, such as great difficulty in data integration, lack of context information, difficulty in consistency maintenance, and weak dynamic updating capability. The method comprises: obtaining multi-modal data of a heterogeneous database to be constructed; pre-processing the data, extracting feature vectors, and performing multi-modal feature fusion to obtain fused features; inputting the fused features into a multi-modal neural network based on context perception to automatically extract the relationship between entities; constructing a knowledge graph based on the relationship between entities; and mapping the knowledge graph to the table structure of the heterogeneous database to complete the construction of the heterogeneous database. The present application solves the problems of the existing heterogeneous database construction method, such as great difficulty in data integration, lack of context information, difficulty in consistency maintenance, and weak dynamic updating capability.

Description

Technical Field

[0001] This invention belongs to the field of knowledge graph construction and database construction technology, specifically relating to a method, system, electronic device, and computer-readable storage medium for constructing heterogeneous databases based on knowledge graphs. Background Technology

[0002] In the era of big data, the construction and management of heterogeneous databases have become particularly important. Heterogeneous databases typically contain multiple types of data, such as structured data (e.g., relational databases), semi-structured data (e.g., XML, JSON), and unstructured data (e.g., text, images). Traditional methods for constructing heterogeneous databases mainly rely on data integration, data cleaning, and data mapping, which have the following shortcomings:

[0003] 1) Data formats and structures vary significantly from data sources, making it difficult for traditional data fusion methods to efficiently integrate these data; 2) Traditional methods often overlook the importance of contextual information for data understanding and correlation, resulting in weak semantic understanding capabilities; 3) Heterogeneous databases may contain a large number of contradictions and redundancies, and existing methods struggle to efficiently detect and repair these problems; 4) As new data is continuously added, heterogeneous databases require constant updates and optimizations, and existing methods typically rely on periodic reconstruction, which is inefficient and difficult to update in real time.

[0004] In recent years, graph neural networks (GNNs) have made significant progress in processing graph-structured data. However, existing GNN methods still have shortcomings in multimodal data processing, context awareness, and dynamic updates, resulting in low efficiency and poor robustness in heterogeneous database construction.

[0005] Based on this, the present invention proposes a method for constructing heterogeneous databases based on knowledge graphs. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, namely, the difficulties in data integration, lack of contextual information, challenges in maintaining consistency, and weak dynamic update capabilities in existing heterogeneous database construction methods, this invention, in its first aspect, proposes a heterogeneous database construction method based on knowledge graphs, comprising:

[0007] S10, Obtain multimodal data of the heterogeneous database to be constructed as input data; the input data includes text data, image data, and tabular data;

[0008] S20, preprocess the input data to obtain preprocessed data; extract the feature vector of the preprocessed data and perform multimodal feature fusion to obtain fused features;

[0009] S30, the fused features are input into a pre-constructed context-aware multimodal neural network to automatically extract the relationships between entities;

[0010] S40, Construct a knowledge graph based on the entities and the relationships between them;

[0011] S50, the knowledge graph is mapped to the table structure of the heterogeneous database to complete the construction of the heterogeneous database;

[0012] The context-aware multimodal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence.

[0013] All three context-aware modules are built upon a sequentially connected multi-head attention mechanism and a Transformer encoder, and are used to obtain context-aware feature vectors. The multi-scale feature extraction layer is used to extract features at different scales. The graph construction layer is used to construct the graph structure. The hierarchical neural network layer is used to extract node features. The feature fusion layer is used to fuse the input features through an attention mechanism. The adaptive gating mechanism layer is used to adjust features through one or more GRU units. The feature reconstruction layer is used for feature reconstruction and fusion. The output layer is built upon a sequentially connected fully connected layer and a softmax function.

[0014] In some preferred embodiments, the preprocessing includes format conversion, data cleaning, and noise removal.

[0015] In some preferred embodiments, the fused features are input into a pre-constructed context-aware multimodal neural network to automatically extract the relationships between entities. The method is as follows:

[0016] The fused features are processed by the first context-aware module to obtain a context-aware feature vector, which is used as the first vector.

[0017] The first vector is subjected to feature extraction by convolution kernels of different scales in the multi-scale feature extraction layer to obtain a multi-scale feature vector, which is used as the second vector.

[0018] Based on the second vector, a graph structure is constructed through the graph construction layer, and the graph structure and node features are output as the first node features;

[0019] Based on the graph structure, a graph attention network is used for message passing to output a second node feature; the first node feature is added to the second node feature to obtain a third node feature; based on the graph structure, a graph convolutional network is used for message passing to output a fourth node feature; the third node feature is added to the fourth node feature to obtain a fifth node feature; based on the graph structure, a graph SAGE is used for message passing to output a sixth node feature; the fifth node feature is added to the sixth node feature to obtain the final node feature; the hierarchical neural network layer includes a graph attention network, a graph convolutional network, and a graph SAGE;

[0020] The final node features are processed by the second context-aware module to obtain a context-aware feature vector, which is used as the third vector.

[0021] The weight of each relation type in the third vector is calculated through the attention mechanism of the feature fusion layer, and the third vector is weighted to obtain the fourth vector; the fourth vector is fused with the first vector and processed through a fully connected layer to obtain the fifth vector;

[0022] Based on the fifth vector, the GRU unit of the adaptive gating mechanism layer dynamically adjusts the importance of the features according to the context information and outputs the sixth vector.

[0023] The sixth vector is processed by the third context-aware module to obtain a context-aware feature vector, which is then used as the seventh vector.

[0024] The seventh vector is reconstructed by the autoencoder of the feature reconstruction layer, and the reconstructed vector is fused with the fused features to obtain the eighth vector; the feature reconstruction layer includes an autoencoder.

[0025] The output layer performs classification prediction on the eighth vector to generate prediction results of the relationships between entities.

[0026] In some preferred embodiments, the loss function of the context-aware multimodal neural network during training is:

[0027]

[0028] Where L represents the loss function, N represents the number of entity samples, C represents the number of entity categories, and y ij p represents the truth label corresponding to the entity prediction result. ij This represents the entity prediction result, where M represents the number of samples relating the entities, R represents the number of relation categories, and z ijq represents the truth label corresponding to the prediction result of the relationship between entities. ij This indicates the prediction results regarding the relationships between entities. Represents the feature vector of text data. Represents the feature vector of image data. Represents the feature vector of the tabular data, ⊙ represents element-wise multiplication, T represents transpose, A represents the defined interaction matrix, and ω i Indicates context weight, This represents the context feature vector of the i-th sample. W represents the truth label corresponding to the context feature vector. k Let represent the k-th parameter matrix, where λ, η, and θ represent regularization coefficients, and α1, α2, α3, α4, and α5 represent weight coefficients.

[0029] In some preferred embodiments, the weighting coefficients are obtained by:

[0030] Calculate the exponential decay function value of the loss function term corresponding to the weight coefficient to be obtained, and use it as the first value;

[0031] Calculate the sum of the exponential decay function values ​​corresponding to all loss function terms, and use this as the second value;

[0032] The ratio of the first value to the second value is used as the weighting coefficient.

[0033] In some preferred embodiments, a knowledge graph is constructed based on the entities and the relationships between them, using the following method:

[0034] A graph is constructed based on the entities and the relationships between them, serving as an initial knowledge graph;

[0035] The initial knowledge graph is processed by a graph neural network to perform message passing and extract the feature representation of each entity.

[0036] By combining the feature representations of each entity, the similarity between each pair of entities is calculated, and clustering algorithms are used to cluster entities with similarity values ​​higher than a set value.

[0037] After clustering, relationships between entities are extracted using a graph traversal algorithm, and relationship conflict detection is performed using a constraint propagation algorithm. If conflicting relationships exist, relationships between entities in the initial knowledge graph are deleted or retained using a heuristic algorithm. The knowledge graph processed by the heuristic algorithm is used as the final constructed knowledge graph. The constraint propagation algorithm includes arc consistency and path consistency.

[0038] In some preferred embodiments, relationships between entities in the initial knowledge graph are deleted or retained using heuristic algorithms, the method being as follows:

[0039] The context information of the conflict relationship is obtained and encoded to obtain an encoding vector; the encoding vector is weighted by a set first learnable parameter, and the weighted encoding vector is activated to obtain a context feature vector;

[0040] The feature vectors of the multimodal data corresponding to the conflict relationship are fused to obtain a multimodal feature vector; the multimodal feature vector is weighted by a set second learnable parameter, and the weighted multimodal feature vector is activated to obtain a multimodal information feature vector;

[0041] By combining the multimodal data corresponding to the conflict relationship and the contextual information of the conflict relationship, the weights are dynamically adjusted through an attention mechanism.

[0042] Based on the dynamically adjusted weights, the context feature vector, the multimodal information feature vector, and the confidence scores corresponding to the conflict relationships are weighted and summed, and conflict relationships with sums less than a set threshold are deleted.

[0043] In a second aspect, the present invention proposes a heterogeneous database construction system based on knowledge graphs, the system comprising:

[0044] The data acquisition module is configured to acquire multimodal data from the heterogeneous database to be constructed as input data; the input data includes text data, image data, and tabular data.

[0045] The feature fusion module is configured to preprocess the input data to obtain preprocessed data; extract the feature vectors of the preprocessed data and perform multimodal feature fusion to obtain fused features;

[0046] The relationship extraction module is configured to input the fused features into a pre-constructed context-aware multimodal neural network to automatically extract the relationships between entities.

[0047] The knowledge graph construction module is configured to construct a knowledge graph based on the entities and the relationships between them.

[0048] The database construction module is configured to map the knowledge graph to the table structure of the heterogeneous database, thereby completing the construction of the heterogeneous database.

[0049] The context-aware multimodal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence.

[0050] All three context-aware modules are built upon a sequentially connected multi-head attention mechanism and a Transformer encoder, and are used to obtain context-aware feature vectors. The multi-scale feature extraction layer is used to extract features at different scales. The graph construction layer is used to construct the graph structure. The hierarchical neural network layer is used to extract node features. The feature fusion layer is used to fuse the input features through an attention mechanism. The adaptive gating mechanism layer is used to adjust features through one or more GRU units. The feature reconstruction layer is used for feature reconstruction and fusion. The output layer is built upon a sequentially connected fully connected layer and a softmax function.

[0051] In a third aspect, the present invention provides an electronic device, the device comprising:

[0052] At least one processor, and a memory communicatively connected to at least one of the processors;

[0053] The memory stores instructions that can be executed by the processor to implement the above-described method for constructing a heterogeneous database based on a knowledge graph.

[0054] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions for execution by a computer to implement the above-described method for constructing a heterogeneous database based on a knowledge graph.

[0055] The beneficial effects of this invention are:

[0056] This invention solves the problems of existing heterogeneous database construction methods, such as high difficulty in data integration, lack of context information, difficulty in maintaining consistency, and weak dynamic update capability.

[0057] 1) This invention effectively integrates data from multiple modalities such as text, images, and tables through a multimodal input fusion layer and a multi-scale feature extraction layer, generating high-quality feature vectors and improving the richness and accuracy of the knowledge graph.

[0058] 2) This invention significantly improves the quality and efficiency of knowledge graph construction by introducing mechanisms such as context awareness and adaptive gating.

[0059] 3) This invention ensures the logical consistency of the knowledge graph through constraint propagation algorithm and conflict resolution strategy, reduces human intervention, avoids contradictions and redundancy, improves the reliability and credibility of the knowledge graph, and thus realizes the efficient, accurate and reliable construction of heterogeneous databases. Attached Figure Description

[0060] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0061] Figure 1 This is a flowchart illustrating a method for constructing a heterogeneous database based on a knowledge graph, according to one embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0063] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0064] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0065] A method for constructing a heterogeneous database based on a knowledge graph, as described in the first embodiment of the present invention, is as follows: Figure 1 As shown, it includes the following steps:

[0066] S10, Obtain multimodal data of the heterogeneous database to be constructed as input data; the input data includes text data, image data, and tabular data;

[0067] S20, preprocess the input data to obtain preprocessed data; extract the feature vector of the preprocessed data and perform multimodal feature fusion to obtain fused features;

[0068] S30, the fused features are input into a pre-constructed context-aware multimodal neural network to automatically extract the relationships between entities;

[0069] S40, Construct a knowledge graph based on the entities and the relationships between them;

[0070] S50, the knowledge graph is mapped to the table structure of the heterogeneous database to complete the construction of the heterogeneous database;

[0071] The context-aware multimodal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence.

[0072] All three context-aware modules are built upon a sequentially connected multi-head attention mechanism and a Transformer encoder, and are used to obtain context-aware feature vectors. The multi-scale feature extraction layer is used to extract features at different scales. The graph construction layer is used to construct the graph structure. The hierarchical neural network layer is used to extract node features. The feature fusion layer is used to fuse the input features through an attention mechanism. The adaptive gating mechanism layer is used to adjust features through one or more GRU units. The feature reconstruction layer is used for feature reconstruction and fusion. The output layer is built upon a sequentially connected fully connected layer and a softmax function.

[0073] To more clearly illustrate the method for constructing a heterogeneous database based on a knowledge graph according to the present invention, the steps of one embodiment of the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0074] S10, Obtain multimodal data of the heterogeneous database to be constructed as input data; the input data includes text data, image data, and tabular data;

[0075] In this embodiment, multimodal data of the heterogeneous database to be constructed is first collected; taking the material and equipment management system as an example, financial data, personnel data, etc. in the system are collected, including text (such as contracts and reports), images (such as invoices and ID photos), and tables (such as financial statements and personnel files).

[0076] S20, preprocess the input data to obtain preprocessed data; extract the feature vector of the preprocessed data and perform multimodal feature fusion to obtain fused features;

[0077] In this embodiment, the collected multimodal data undergoes preprocessing such as format conversion, data cleaning, and noise removal. After preprocessing, features of each modality are extracted. For example, a pre-trained BERT model is used to convert text into word vectors, a pre-trained ResNet model is used to convert images into feature vectors, and a TabNet model is used to convert tabular data into feature vectors. Then, the multimodal features are fused through a fully connected layer after weighted averaging or concatenation to generate fused features.

[0078] S30, the fused features are input into a pre-constructed context-aware multimodal neural network to automatically extract the relationships between entities;

[0079] Traditional entity and relation extraction methods primarily rely on textual features, neglecting contextual information beyond the text (such as images and tables) and complex interaction patterns between entities. Furthermore, existing methods often assume that all entity and relation types are of equal importance, which is unrealistic, as different entity and relation types have varying values ​​in different scenarios. Therefore, this embodiment constructs a context-aware multimodal graph neural network and introduces a cross-layer connection mechanism to enhance the model's expressive power and learning efficiency, thereby improving the accuracy and robustness of entity and relation extraction.

[0080] Context-aware multimodal neural networks are constructed by sequentially connecting the following modules: a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer.

[0081] The first context-aware module, the second context-aware module, and the third context-aware module are all constructed based on a multi-head attention mechanism and a Transformer encoder connected in sequence. The multi-head attention mechanism is used to capture long-distance dependencies and generate context-aware feature vectors. The context information is further encoded by a single-layer or multi-layer Transformer encoder to generate context-aware feature vectors.

[0082] The fused features are processed by the first context-aware module to obtain a context-aware feature vector, which is used as the first vector.

[0083] The multi-scale feature extraction layer is used to extract features from the first vector using convolutional kernels of different scales (such as 3x3, 5x5, 7x7) to obtain a multi-scale feature vector, which is then used as the second vector.

[0084] The graph construction layer is used to construct a graph structure based on the second vector, and output the graph structure and node features as the first node features;

[0085] The hierarchical neural network layer includes a graph attention network, a graph convolutional network, and GraphSAGE. This hierarchical neural network layer is used to perform message passing through the graph attention network based on the graph structure, outputting second node features. The first node features are added to the second node features to obtain third node features. This invention captures the relationship between nodes (i.e., entities) and their neighbors through the graph attention mechanism in the graph attention network; it preserves initial features through residual connections, preventing gradient vanishing and improving the training stability of the model.

[0086] Based on the graph structure, message passing is performed through a graph convolutional network to output the fourth node feature; the third node feature is added to the fourth node feature to obtain the fifth node feature; that is, the present invention further aggregates the neighborhood information of the nodes through graph convolution operations and retains the features of the previous layer through residual connections, thereby further improving the training stability of the model.

[0087] Based on the graph structure, message passing is performed through GraphSAGE to output the features of the sixth node; that is, the present invention further aggregates the neighborhood information of the node through GraphSAGE (Graph Sample and Aggregate).

[0088] The fifth node feature is added to the sixth node feature to obtain the final node feature;

[0089] The final node features are processed by the second context-aware module to obtain a context-aware feature vector, which is used as the third vector.

[0090] The feature fusion layer is used to calculate the weight of each relation type in the third vector through an attention mechanism, and to perform weighted processing on the third vector to obtain a fourth vector; the fourth vector is then fused with the first vector and processed through a fully connected layer to obtain a fifth vector;

[0091] The adaptive gating mechanism layer is used to dynamically adjust the importance of features based on the fifth vector through a GRU unit (or an LSTM unit) according to context information, and output a sixth vector.

[0092] The sixth vector is processed by the third context-aware module to obtain a context-aware feature vector, which is then used as the seventh vector.

[0093] The feature reconstruction layer is used to reconstruct the features of the seventh vector through an autoencoder, and then perform a secondary fusion of the reconstructed vector with the fused features to obtain the eighth vector.

[0094] The output layer is constructed based on a fully connected layer and a softmax function connected in sequence; the output layer is used to generate prediction results of the relationship between entities by classifying and predicting the eighth vector.

[0095] The loss function of the context-aware multimodal neural network during training is:

[0096]

[0097] Where L represents the loss function, N represents the number of entity samples, C represents the number of entity categories, and y ijp represents the truth label corresponding to the entity prediction result. ij This represents the entity prediction result, where M represents the number of samples relating the entities, R represents the number of relation categories, and z ij q represents the truth label corresponding to the prediction result of the relationship between entities. ij This indicates the prediction results regarding the relationships between entities. Represents the feature vector of text data. Represents the feature vector of image data. Represents the feature vector of the tabular data, ⊙ represents element-wise multiplication, T represents transpose, A represents the defined interaction matrix, and ω i This represents the context weights, a term in the loss function used to capture higher-order relationships between different modalities of data. This represents the context feature vector of the i-th sample. W represents the ground truth label corresponding to the context feature vector. This loss function term is used to enhance the network's sensitivity to context. k Let represent the k-th parameter matrix. An adaptive regularization term is calculated from this parameter matrix to prevent overfitting. λ, η, and θ represent regularization coefficients, and α1, α2, α3, α4, and α5 represent weight coefficients. The weight coefficients are obtained as follows:

[0098] Calculate the exponential decay function value of the loss function term corresponding to the current weight coefficient to be acquired, as the first value; calculate the sum of the exponential decay function values ​​corresponding to all loss function terms, as the second value; and use the ratio of the first value to the second value as the weight coefficient. That is... Where ξ represents the difficulty coefficient, L1 is the loss function term corresponding to the weight coefficient to be obtained, and Li is the loss function term corresponding to each weight coefficient, such as the one mentioned above. All of these are used as loss function terms, and W is the number of loss function terms.

[0099] S40, Construct a knowledge graph based on the entities and the relationships between them;

[0100] In this embodiment, the process of constructing the knowledge graph is as follows:

[0101] A graph is constructed based on the entities and the relationships between them, serving as an initial knowledge graph;

[0102] The initial knowledge graph is message-passed through a graph neural network (preferably GNN, but in other embodiments, GCN, GAT, etc. can be selected for message passing) to extract the feature representation of each entity.

[0103] By combining the feature representations of each entity, the similarity between each pair of entities is calculated (such as cosine similarity or Jaccard similarity), and clustering algorithms (such as DBSCAN clustering algorithm) are used to cluster entities with similarity values ​​higher than a set value.

[0104] After clustering, relationships between entities are extracted using graph traversal algorithms (such as BFS, DFS, etc.), and relationship conflict detection is performed using constraint propagation algorithms. If conflicting relationships exist (i.e., based on domain knowledge and common sense, a series of logical constraints are defined to ensure that the knowledge in the knowledge graph is logically consistent, such as mutual exclusion relationships, where certain relationships cannot be true simultaneously. For example, one person cannot be both the father and son of another person; transitive relationships: some relationships are transitive. For example, if A is the parent class of B, and B is the parent class of C, then A is also the parent class of C), then heuristic algorithms are used to delete or retain the relationships between entities in the initial knowledge graph (including arc consistency (e.g., checking whether one person is both the father and son of another person, and if so, deleting one of the relationships) and path consistency (e.g., whether A is the parent class of B, and B is the parent class of C exists, and if so, adding A as the parent class of C)). The knowledge graph processed by heuristic algorithms is used as the final constructed knowledge graph.

[0105] The method for deleting or retaining relationships between entities in the initial knowledge graph using a heuristic algorithm is as follows:

[0106] The context information of the conflict relationship is obtained and encoded to obtain an encoded vector. This encoded vector is then weighted using a first learnable parameter, and the weighted encoded vector is activated to obtain a context feature vector; commonly, σ(W1×x+b1), where W1 and b1 are learnable parameters, and x is the weighted vector.

[0107] The feature vectors of the multimodal data corresponding to the conflict relationship are fused to obtain a multimodal feature vector; the multimodal feature vector is weighted by a set second learnable parameter, and the weighted multimodal feature vector is activated to obtain a multimodal information feature vector;

[0108] Combining the multimodal data corresponding to the conflict relationship and the contextual information of the conflict relationship, a dynamic adjustment weight is calculated through an attention mechanism; that is, the multimodal data and contextual information are used as key vectors, the features of the current relationship are used as query vectors, and the dynamic adjustment weight is calculated through the formula of the attention mechanism. This is an existing technology and will not be elaborated here.

[0109] Based on the dynamically adjusted weights, the context feature vector, the multimodal information feature vector, and the confidence corresponding to the conflict relationship are weighted and summed respectively. Conflict relationships with a sum less than a set threshold are deleted. If multiple conflict relationships have the same sum, the relationship with the fewest number of relationships to delete can be selected. That is, the optimal solution is selected by minimizing conflicts to maintain the consistency of the knowledge graph.

[0110] This invention automates the consistency maintenance of knowledge graphs through constraint propagation and conflict resolution algorithms, reduces manual intervention, ensures logical consistency of knowledge in the knowledge graph, and improves the accuracy and reliability of the knowledge graph.

[0111] In addition, incremental fusion can be performed on the constructed knowledge graph, that is, new data is incrementally extracted to generate new entities and relations, and the newly extracted entities and relations are merged with the existing knowledge graph to update the graph structure.

[0112] S50, the knowledge graph is mapped to the table structure of the heterogeneous database to complete the construction of the heterogeneous database;

[0113] In this embodiment, entities in the knowledge graph are mapped to the table structure of the heterogeneous database. Relationships in the graph are used as foreign keys to establish connections between different tables, thus constructing a heterogeneous database. The constructed heterogeneous database can be used in multiple scenarios, such as financial data in a materials and equipment management system. After the heterogeneous database is constructed, financial data can be integrated to generate financial reports and perform budget control (such as timely detection of overspending issues by analyzing the expenditure of various projects).

[0114] In summary, this invention effectively integrates data from multiple modalities, such as text, images, and tables, through multimodal input fusion, multi-scale feature extraction, the introduction of context awareness, and adaptive gating mechanisms. This generates high-quality feature vectors, improving the efficiency, richness, and accuracy of knowledge graph construction. Furthermore, by employing constraint propagation algorithms and conflict resolution strategies, it ensures the logical consistency of the knowledge graph, avoiding contradictions and redundancy. This enhances the reliability and credibility of knowledge graphs and the heterogeneous databases built upon them, thereby addressing the problems of high data integration difficulty, lack of contextual information, difficulty in maintaining consistency, and weak dynamic update capabilities in existing heterogeneous database construction methods.

[0115] A second embodiment of the present invention provides a heterogeneous database construction system based on knowledge graphs, the system comprising:

[0116] The data acquisition module is configured to acquire multimodal data from the heterogeneous database to be constructed as input data; the input data includes text data, image data, and tabular data.

[0117] The feature fusion module is configured to preprocess the input data to obtain preprocessed data; extract the feature vectors of the preprocessed data and perform multimodal feature fusion to obtain fused features;

[0118] The relationship extraction module is configured to input the fused features into a pre-constructed context-aware multimodal neural network to automatically extract the relationships between entities.

[0119] The knowledge graph construction module is configured to construct a knowledge graph based on the entities and the relationships between them.

[0120] The database construction module is configured to map the knowledge graph to the table structure of the heterogeneous database, thereby completing the construction of the heterogeneous database.

[0121] The context-aware multimodal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence.

[0122] All three context-aware modules are built upon a sequentially connected multi-head attention mechanism and a Transformer encoder, and are used to obtain context-aware feature vectors. The multi-scale feature extraction layer is used to extract features at different scales. The graph construction layer is used to construct the graph structure. The hierarchical neural network layer is used to extract node features. The feature fusion layer is used to fuse the input features through an attention mechanism. The adaptive gating mechanism layer is used to adjust features through one or more GRU units. The feature reconstruction layer is used for feature reconstruction and fusion. The output layer is built upon a sequentially connected fully connected layer and a softmax function.

[0123] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the system described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0124] It should be noted that the knowledge graph-based heterogeneous database construction system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further split into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0125] A third embodiment of the present invention provides an electronic device comprising at least one processor and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to implement the above-described method for constructing a heterogeneous database based on a knowledge graph.

[0126] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, which are executed by the computer to implement the above-described method for constructing a heterogeneous database based on a knowledge graph.

[0127] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the electronic device and readable storage medium described above can be referred to the corresponding process in the foregoing method examples, and will not be repeated here.

[0128] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

[0129] The terms “first,” “second,” “third,” etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0130] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for constructing a heterogeneous database based on knowledge graphs, characterized in that, The method includes: S10, Obtain multimodal data of the heterogeneous database to be constructed as input data; the input data includes text data, image data, and tabular data; S20, preprocess the input data to obtain preprocessed data; extract the feature vector of the preprocessed data and perform multimodal feature fusion to obtain fused features; S30, the fused features are input into a pre-constructed context-aware multimodal neural network to automatically extract the relationships between entities; S40, Construct a knowledge graph based on the entities and the relationships between them; S50, the knowledge graph is mapped to the table structure of the heterogeneous database to complete the construction of the heterogeneous database; The context-aware multimodal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence. All three context-aware modules are built upon a sequentially connected multi-head attention mechanism and a Transformer encoder, and are used to obtain context-aware feature vectors. The multi-scale feature extraction layer is used to extract features at different scales. The graph construction layer is used to construct the graph structure. The hierarchical neural network layer is used to extract node features. The feature fusion layer is used to fuse the input features through an attention mechanism. The adaptive gating mechanism layer is used to adjust features through one or more GRU units. The feature reconstruction layer is used for feature reconstruction and fusion. The output layer is built upon a sequentially connected fully connected layer and a softmax function.

2. The method for constructing a heterogeneous database based on knowledge graphs according to claim 1, characterized in that, The preprocessing includes format conversion, data cleaning, and noise removal.

3. The method for constructing a heterogeneous database based on a knowledge graph according to claim 2, characterized in that, The fused features are input into a pre-constructed context-aware multimodal neural network to automatically extract the relationships between entities. The method is as follows: The fused features are processed by the first context-aware module to obtain a context-aware feature vector, which is used as the first vector. The first vector is subjected to feature extraction by convolution kernels of different scales in the multi-scale feature extraction layer to obtain a multi-scale feature vector, which is used as the second vector. Based on the second vector, a graph structure is constructed through the graph construction layer, and the graph structure and node features are output as the first node features; Based on the graph structure, message passing is performed through a graph attention network to output the second node feature; the first node feature and the second node feature are added together to obtain the third node feature; Based on the graph structure, message passing is performed through a graph convolutional network to output the fourth node feature; the third node feature is added to the fourth node feature to obtain the fifth node feature; based on the graph structure, message passing is performed through GraphSAGE to output the sixth node feature; The fifth node feature is added to the sixth node feature to obtain the final node feature; The hierarchical neural network layers include graph attention networks, graph convolutional networks, and GraphSAGE; The final node features are processed by the second context-aware module to obtain a context-aware feature vector, which is used as the third vector. The weight of each relation type in the third vector is calculated through the attention mechanism of the feature fusion layer, and the third vector is weighted to obtain the fourth vector; the fourth vector is fused with the first vector and processed through a fully connected layer to obtain the fifth vector; Based on the fifth vector, the GRU unit of the adaptive gating mechanism layer dynamically adjusts the importance of the features according to the context information and outputs the sixth vector. The sixth vector is processed by the third context-aware module to obtain a context-aware feature vector, which is then used as the seventh vector. The seventh vector is reconstructed by the autoencoder of the feature reconstruction layer, and the reconstructed vector is fused with the fused features to obtain the eighth vector; the feature reconstruction layer includes an autoencoder. The output layer performs classification prediction on the eighth vector to generate prediction results of the relationships between entities.

4. The method for constructing a heterogeneous database based on a knowledge graph according to claim 3, characterized in that, The loss function of the context-aware multimodal neural network during training is: Where L represents the loss function, N represents the number of entity samples, C represents the number of entity categories, and y ij p represents the truth label corresponding to the entity prediction result. ij This represents the entity prediction result, where M represents the number of samples relating entities, R represents the number of relation categories, and Z represents the number of samples relating entities. ij q represents the truth label corresponding to the prediction result of the relationship between entities. ij This indicates the prediction results regarding the relationships between entities. Represents the feature vector of text data. Represents the feature vector of image data. Represents the feature vector of the tabular data, ⊙ represents element-wise multiplication, T represents transpose, A represents the defined interaction matrix, and ω i Indicates context weight, This represents the context feature vector of the i-th sample. W represents the truth label corresponding to the context feature vector. k Let represent the k-th parameter matrix, where λ, η, and θ represent regularization coefficients, and α1, α2, α3, α4, and α5 represent weight coefficients.

5. The method for constructing a heterogeneous database based on a knowledge graph according to claim 4, characterized in that, The weighting coefficients are obtained as follows: Calculate the exponential decay function value of the loss function term corresponding to the weight coefficient to be obtained, and use it as the first value; Calculate the sum of the exponential decay function values ​​corresponding to all loss function terms, and use this as the second value; The ratio of the first value to the second value is used as the weighting coefficient.

6. The method for constructing a heterogeneous database based on a knowledge graph according to claim 5, characterized in that, Based on the entities and the relationships between them, a knowledge graph is constructed using the following method: A graph is constructed based on the entities and the relationships between them, serving as an initial knowledge graph; The initial knowledge graph is processed by a graph neural network to perform message passing and extract the feature representation of each entity. By combining the feature representations of each entity, the similarity between each pair of entities is calculated, and clustering algorithms are used to cluster entities with similarity values ​​higher than a set value. After clustering, relationships between entities are extracted using a graph traversal algorithm, and relationship conflict detection is performed using a constraint propagation algorithm. If conflicting relationships exist, relationships between entities in the initial knowledge graph are deleted or retained using a heuristic algorithm. The knowledge graph processed by the heuristic algorithm is used as the final constructed knowledge graph. The constraint propagation algorithm includes arc consistency and path consistency.

7. The method for constructing a heterogeneous database based on a knowledge graph according to claim 6, characterized in that, The method for deleting or retaining relationships between entities in the initial knowledge graph using a heuristic algorithm is as follows: The context information of the conflict relationship is obtained and encoded to obtain an encoding vector; the encoding vector is weighted by a set first learnable parameter, and the weighted encoding vector is activated to obtain a context feature vector; The feature vectors of the multimodal data corresponding to the conflict relationship are fused to obtain a multimodal feature vector; the multimodal feature vector is weighted by a set second learnable parameter, and the weighted multimodal feature vector is activated to obtain a multimodal information feature vector; By combining the multimodal data corresponding to the conflict relationship and the contextual information of the conflict relationship, the weights are dynamically adjusted through an attention mechanism. Based on the dynamically adjusted weights, the context feature vector, the multimodal information feature vector, and the confidence scores corresponding to the conflict relationships are weighted and summed, and conflict relationships with sums less than a set threshold are deleted.

8. A heterogeneous database construction system based on knowledge graphs, characterized in that, The system includes: The data acquisition module is configured to acquire multimodal data from the heterogeneous database to be constructed as input data; the input data includes text data, image data, and tabular data. The feature fusion module is configured to preprocess the input data to obtain preprocessed data; extract the feature vectors of the preprocessed data and perform multimodal feature fusion to obtain fused features; The relationship extraction module is configured to input the fused features into a pre-constructed context-aware multimodal neural network to automatically extract the relationships between entities. The knowledge graph construction module is configured to construct a knowledge graph based on the entities and the relationships between them. The database construction module is configured to map the knowledge graph to the table structure of the heterogeneous database, thereby completing the construction of the heterogeneous database. The context-aware multimodal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence. All three context-aware modules are built upon a sequentially connected multi-head attention mechanism and a Transformer encoder, and are used to obtain context-aware feature vectors. The multi-scale feature extraction layer is used to extract features at different scales. The graph construction layer is used to construct the graph structure. The hierarchical neural network layer is used to extract node features. The feature fusion layer is used to fuse the input features through an attention mechanism. The adaptive gating mechanism layer is used to adjust features through one or more GRU units. The feature reconstruction layer is used for feature reconstruction and fusion. The output layer is built upon a sequentially connected fully connected layer and a softmax function.

9. An electronic device, characterized in that, The electronic device includes: At least one processor, and a memory communicatively connected to at least one of the processors; The memory stores instructions that can be executed by the processor to implement the knowledge graph-based heterogeneous database construction method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are executed by a computer to implement the knowledge graph-based heterogeneous database construction method according to any one of claims 1-7.

Images