Heterogeneous database construction method based on knowledge graph

Abstract

The invention belongs to the technical field of knowledge graph construction and database construction, particularly relates to a heterogeneous database construction method based on a knowledge graph, and aims to solve the problems that an existing heterogeneous database construction method is high in data integration difficulty, short in context information, difficult in consistency maintenance and weak in dynamic updating capability. The method comprises the following steps: acquiring multi-modal data of a to-be-constructed heterogeneous database; preprocessing the data, and extracting feature vectors for multi-modal feature fusion to obtain fusion features; inputting the fusion features into a multi-modal neural network based on context sensing, and automatically extracting a relationship between entities; constructing a knowledge graph based on the relationship between the entities; and mapping the knowledge graph into a table structure of the heterogeneous database to complete construction of the heterogeneous database. The problems that an existing heterogeneous database construction method is large in data integration difficulty, lack of context information, difficult in consistency maintenance and weak in dynamic updating capacity are solved.

Description

Technical Field

[0001] The present invention belongs to the technical fields of knowledge graph construction and database construction, and particularly relates to a method, a system, an electronic device, and a computer-readable storage medium for constructing a heterogeneous database based on a knowledge graph. Background Art

[0002] In the big data era, the construction and management of heterogeneous databases have become particularly important. Heterogeneous databases usually contain various types of data, such as structured data (such as relational databases), semi-structured data (such as XML, JSON), and unstructured data (such as text, images). Traditional methods for constructing heterogeneous databases mainly rely on means such as data integration, data cleaning, and data mapping, and have the following deficiencies:

[0003] 1) The data formats and structures of different data sources vary greatly, and traditional data fusion methods are difficult to efficiently integrate this data together; 2) Traditional methods often ignore the importance of context information for data understanding and association, resulting in weak semantic understanding ability of data; 3) There may be a large amount of contradictions and redundancies in heterogeneous databases, and existing methods are difficult to efficiently detect and repair these problems; 4) As new data is continuously added, heterogeneous databases need to be continuously updated and optimized, and existing methods usually rely on periodic reconstruction, with low efficiency and difficulty in real-time updating.

[0004] In recent years, graph neural networks (GNNs) have made remarkable progress in processing graph-structured data, but existing GNN methods still have deficiencies in multi-modal data processing, context awareness, and dynamic updating, resulting in low efficiency and poor robustness in constructing heterogeneous databases.

[0005] Based on this, the present invention proposes a method for constructing a heterogeneous database based on a knowledge graph. Summary of the Invention

[0006] In order to solve the above problems in the prior art, that is, to solve the problems of difficult data integration, lack of context information, difficult consistency maintenance, and weak dynamic updating ability existing in the existing methods for constructing heterogeneous databases, in the first aspect of the present invention, a method for constructing a heterogeneous database based on a knowledge graph is proposed, and the method includes:

[0007] S10, obtaining multi-modal data of the heterogeneous database to be constructed as input data; the input data includes text data, image data, and table data;

[0008] S20, preprocessing the input data to obtain preprocessed data; extracting feature vectors of the preprocessed data and performing multi-modal feature fusion to obtain fused features;

[0009] S30. Input the fusion features into a pre-constructed context-aware multi-modal neural network to automatically extract the relationships between entities.

[0010] S40. Construct a knowledge graph based on the entities and the relationships between the entities.

[0011] S50. Map the knowledge graph to the table structure of a heterogeneous database to complete the construction of the heterogeneous database.

[0012] The context-aware multi-modal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence.

[0013] All three context-aware modules are based on a multi-head attention mechanism and a Transformer encoder connected in sequence, and are all used to obtain context-aware feature vectors; the multi-scale feature extraction layer is used to extract features of different scales; the graph construction layer is used to construct a graph structure; the hierarchical neural network layer is used to extract node features; the feature fusion layer is used to fuse the input features through an attention mechanism; the adaptive gating mechanism layer is used to adjust features through one or more GRU units; the feature reconstruction layer is used for feature reconstruction and fusion; the output layer is based on a fully connected layer and a softmax function connected in sequence.

[0014] In some preferred embodiments, the preprocessing includes format conversion, data cleaning, and noise removal.

[0015] In some preferred embodiments, the method of inputting the fusion features into a pre-constructed context-aware multi-modal neural network to automatically extract the relationships between entities is as follows:

[0016] Process the fusion features through the first context-aware module to obtain a context-aware feature vector as the first vector.

[0017] Extract features from the first vector through convolutional kernels of different scales in the multi-scale feature extraction layer to obtain a multi-scale feature vector as the second vector.

[0018] Based on the second vector, construct a graph structure through the graph construction layer, and output the graph structure and node features as the first node features.

[0019] Based on the said graph structure, message passing is performed through a graph attention network to output second node features; the first node features and the second node features are added to obtain third node features; based on the graph structure, message passing is performed through a graph convolutional network to output fourth node features; the third node features and the fourth node features are added to obtain fifth node features; based on the graph structure, message passing is performed through GraphSAGE to output sixth node features; the fifth node features and the sixth node features are added to obtain final node features; the hierarchical neural network layer includes a graph attention network, a graph convolutional network, and GraphSAGE;

[0020] The second context-aware module processes the final node features to obtain a context-aware feature vector as the third vector;

[0021] The attention mechanism of the feature fusion layer calculates the weights of each relationship type in the third vector, and performs weighted processing on the third vector to obtain a fourth vector; the fourth vector and the first vector are feature-fused and processed through a fully connected layer to obtain a fifth vector;

[0022] Based on the fifth vector, the GRU unit of the adaptive gating mechanism layer dynamically adjusts the importance of features according to context information and outputs a sixth vector;

[0023] The third context-aware module processes the sixth vector to obtain a context-aware feature vector as the seventh vector;

[0024] The autoencoder of the feature reconstruction layer performs feature reconstruction on the seventh vector, and performs secondary fusion on the vector after feature reconstruction and the fusion feature to obtain an eighth vector; the feature reconstruction layer includes an autoencoder;

[0025] The output layer performs classification prediction on the eighth vector to generate a prediction result of the relationship between entities.

[0026] In some preferred embodiments, for the context-aware multi-modal neural network, the loss function during training is:

[0027] where L represents the loss function, N represents the number of entity samples, C represents the number of entity categories, y ij represents the true label corresponding to the entity prediction result, p ij represents the entity prediction result, M represents the number of samples of the relationship between entities, R represents the number of relationship categories, z ij represents the true label corresponding to the relationship prediction result between entities, qij Represents the relationship prediction result between entities, Represents the feature vector of text data, Represents the feature vector of image data, Represents the feature vector of tabular data, ⊙ represents element-wise multiplication, T represents transpose, A represents the set interaction matrix, ω i Represents the context weight, Represents the context feature vector of the i-th sample, Represents the true label corresponding to the context feature vector, W k Represents the k-th parameter matrix, λ, η, θ all represent regularization coefficients, and α1, α2, α3, α4, α5 all represent weight coefficients.

[0028] In some preferred embodiments, the method for obtaining the weight coefficient is:

[0029] Calculate the exponential decay function value of the loss function term corresponding to the weight coefficient to be obtained currently as the first value;

[0030] Calculate the sum of the exponential decay function values of all loss function terms as the second value;

[0031] Take the ratio of the first value to the second value as the weight coefficient.

[0032] In some preferred embodiments, based on the entities and the relationships between the entities, a knowledge graph is constructed, and the method is:

[0033] Construct a graph based on the entities and the relationships between the entities as the initial knowledge graph;

[0034] Perform message passing on the initial knowledge graph through a graph neural network to extract the feature representation of each entity;

[0035] Combine the feature representations of each entity, calculate the similarity between each pair of entities, and cluster those with a similarity higher than a set value through a clustering algorithm;

[0036] After clustering, extract the relationships between entities through a graph traversal algorithm and perform relationship conflict detection through a constraint propagation algorithm. If there are conflicting relationships, delete or retain the relationships between entities in the initial knowledge graph through a heuristic algorithm, and use the knowledge graph processed by the heuristic algorithm as the finally constructed knowledge graph; the constraint propagation algorithm includes arc consistency and path consistency.

[0037] In some preferred embodiments, the method for deleting or retaining the relationships between entities in the initial knowledge graph through a heuristic algorithm is:

[0038] Obtain the context information of the conflict relationship and encode it to obtain an encoded vector; perform weighted processing on the encoded vector through a set first learnable parameter, and perform activation processing on the weighted encoded vector to obtain a context feature vector;

[0039] Fuse the feature vectors of the multi-modal data corresponding to the conflict relationship to obtain a multi-modal feature vector; perform weighted processing on the multi-modal feature vector through a set second learnable parameter, and perform activation processing on the weighted multi-modal feature vector to obtain a multi-modal information feature vector;

[0040] Combine the multi-modal data corresponding to the conflict relationship and the context information of the conflict relationship, and calculate the dynamically adjusted weights through an attention mechanism;

[0041] Based on the dynamically adjusted weights, respectively weight and sum the context feature vector, the multi-modal information feature vector, and the confidence corresponding to the conflict relationship, and delete the conflict relationships whose sum is less than the set sum threshold.

[0042] In the second aspect of the present invention, a heterogeneous database construction system based on a knowledge graph is proposed. The system includes:

[0043] A data acquisition module configured to acquire multi-modal data of the heterogeneous database to be constructed as input data; the input data includes text data, image data, and table data;

[0044] A feature fusion module configured to preprocess the input data to obtain preprocessed data; extract the feature vectors of the preprocessed data and perform multi-modal feature fusion to obtain a fusion feature;

[0045] A relationship extraction module configured to input the fusion feature into a pre-constructed context-aware multi-modal neural network to automatically extract the relationship between entities;

[0046] A graph construction module configured to construct a knowledge graph based on the entities and the relationships between the entities;

[0047] A database construction module configured to map the knowledge graph into the table structure of the heterogeneous database to complete the construction of the heterogeneous database;

[0048] The context-aware multi-modal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence;

[0049] All three context-aware modules are constructed based on the multi-head attention mechanism and the Transformer encoder connected in sequence, and are all used to obtain context-aware feature vectors; the multi-scale feature extraction layer is used to extract features of different scales; the graph construction layer is used to construct a graph structure; the hierarchical neural network layer is used to extract node features; the feature fusion layer is used to fuse the input features through the attention mechanism; the adaptive gating mechanism layer is used to adjust features through one or more GRU units; the feature reconstruction layer is used for feature reconstruction and fusion; the output layer is constructed based on the fully connected layer and the softmax function connected in sequence.

[0050] In the third aspect of the present invention, an electronic device is proposed, and the device includes:

[0051] At least one processor, and a memory communicatively connected to at least one of the processors;

[0052] Wherein, the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned method for constructing a heterogeneous database based on a knowledge graph.

[0053] In the fourth aspect of the present invention, a computer-readable storage medium is proposed, and the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by a computer to implement the above-mentioned method for constructing a heterogeneous database based on a knowledge graph.

[0054] Advantages of the present invention:

[0055] The present invention solves the problems of large data integration difficulty, lack of context information, difficult consistency maintenance, and weak dynamic update ability existing in the existing methods for constructing heterogeneous databases.

[0056] 1) Through the multi-modal input fusion layer and the multi-scale feature extraction layer, the present invention effectively fuses data of various modalities such as text, images, and tables, generates high-quality feature vectors, and improves the richness and accuracy of the knowledge graph.

[0057] 2) By introducing mechanisms such as context awareness and adaptive gating mechanism, the present invention significantly improves the construction quality and efficiency of the knowledge graph;

[0058] 3) Through the constraint propagation algorithm and conflict resolution strategy, the present invention ensures the logical consistency of the knowledge graph, reduces manual intervention, and avoids contradictions and redundancies, improves the reliability and credibility of the knowledge graph, and thus realizes the efficient, accurate, and reliable construction of the heterogeneous database. Description of the Drawings

[0059] Other features, objectives, and advantages of the present application will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0060] Figure 1 It is a schematic flowchart of a method for constructing a heterogeneous database based on a knowledge graph according to an embodiment of the present invention. Detailed implementation manners

[0061] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0062] The following further elaborates the present application with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention and are not intended to limit the invention. Additionally, it should be noted that, for the sake of description, only parts related to the relevant invention are shown in the accompanying drawings.

[0063] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0064] A method for constructing a heterogeneous database based on a knowledge graph according to the first embodiment of the present invention, as Figure 1 shown, includes the following steps:

[0065] S10. Obtain multimodal data of the heterogeneous database to be constructed as input data; the input data includes text data, image data, and table data.

[0066] S20. Preprocess the input data to obtain preprocessed data; extract the feature vectors of the preprocessed data and perform multimodal feature fusion to obtain fused features.

[0067] S30. Input the fused features into a pre-constructed context-aware multimodal neural network to automatically extract the relationships between entities.

[0068] S40. Construct a knowledge graph based on the entities and the relationships between the entities.

[0069] S50. Map the knowledge graph to the table structure of the heterogeneous database to complete the construction of the heterogeneous database.

[0070] The context-aware multi-modal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer that are connected in sequence;

[0071] The three context-aware modules are all constructed based on a multi-head attention mechanism and a Transformer encoder that are connected in sequence, and are all used to obtain context-aware feature vectors; the multi-scale feature extraction layer is used to extract features of different scales; the graph construction layer is used to construct a graph structure; the hierarchical neural network layer is used to extract node features; the feature fusion layer is used to fuse the input features through an attention mechanism; the adaptive gating mechanism layer is used to adjust features through one or more GRU units; the feature reconstruction layer is used for feature reconstruction and fusion; the output layer is constructed based on a fully connected layer and a softmax function that are connected in sequence.

[0072] To more clearly illustrate a method for constructing a heterogeneous database based on a knowledge graph of the present invention, the following will elaborate on each step in an embodiment of the method of the present invention with reference to the accompanying drawings.

[0073] S10. Obtain multi-modal data of the heterogeneous database to be constructed as input data; the input data includes text data, image data, and table data;

[0074] In this embodiment, first collect the multi-modal data of the heterogeneous database to be constructed; taking the material and equipment management system as an example, collect financial data, personnel data, etc. in the system, including text (such as contracts, reports), images (such as invoices, ID photos), and tables (such as financial statements, personnel files).

[0075] S20. Preprocess the input data to obtain preprocessed data; extract feature vectors of the preprocessed data and perform multi-modal feature fusion to obtain fused features;

[0076] In this embodiment, perform preprocessing such as format conversion, data cleaning, and noise removal on the collected multi-modal data. After preprocessing, extract the features of each modality's data respectively. For example, use a pre-trained BERT model to convert text into word vectors, use a pre-trained ResNet model to convert images into feature vectors, use a TabNet model to convert table data into feature vectors, etc. Then, fuse the multi-modal features through weighted average or concatenation and then through a fully connected layer to generate fused features.

[0077] S30. Input the fused features into a pre-constructed context-aware multi-modal neural network to automatically extract the relationships between entities;

[0078] Traditional entity and relation extraction methods mainly rely on text features, ignoring context information outside the text (such as images, tables, etc.), as well as complex interaction patterns between entities. In addition, existing methods often assume that all entity and relation types are equally important, which does not conform to the actual situation because different entity and relation types have different values in different scenarios. Therefore, in this embodiment, a context-aware multi-modal graph neural network is constructed, and a cross-layer connection mechanism is introduced to enhance the model's expressive power and learning efficiency, and improve the accuracy and robustness of entity and relation extraction.

[0079] The context-aware multi-modal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence;

[0080] The first context-aware module, the second context-aware module, and the third context-aware module are all constructed based on a multi-head attention mechanism and a Transformer encoder connected in sequence; the multi-head attention mechanism is used to capture long-range dependencies and generate context-aware feature vectors, and the context information is further encoded through a single-layer or multi-layer Transformer encoder to generate context-aware feature vectors;

[0081] The fused features are processed by the first context-aware module to obtain a context-aware feature vector as the first vector;

[0082] The multi-scale feature extraction layer is used to extract features from the first vector through convolutional kernels of different scales (such as 3x3, 5x5, 7x7) to obtain multi-scale feature vectors as the second vector;

[0083] The graph construction layer is used to construct a graph structure based on the second vector and output the graph structure and node features as the first node features;

[0084] The hierarchical neural network layer includes a graph attention network, a graph convolutional network, and GraphSAGE; the hierarchical neural network layer is used to perform message passing through the graph attention network based on the graph structure and output second node features; the first node features are added to the second node features to obtain third node features; in the present invention, the graph attention mechanism in the graph attention network captures the relationship between the receiving node (i.e., the entity) and its neighbors; the initial features are retained through residual connections to prevent gradient disappearance and improve the training stability of the model;

[0085] Based on the graph structure, message passing is performed through a graph convolutional network to output the fourth node feature; the third node feature and the fourth node feature are added together to obtain the fifth node feature; that is, the present invention further aggregates the neighborhood information of nodes through graph convolutional operations and retains the features of the previous layer through residual connections, further improving the training stability of the model.

[0086] Based on the graph structure, message passing is performed through GraphSAGE to output the sixth node feature; that is, the present invention further aggregates the neighborhood information of nodes through GraphSAGE (Graph Sample and Aggregate).

[0087] The fifth node feature and the sixth node feature are added together to obtain the final node feature;

[0088] The second context-aware module processes the final node feature to obtain a context-aware feature vector as the third vector;

[0089] The feature fusion layer is used to calculate the weight of each relationship type in the third vector through an attention mechanism and perform weighted processing on the third vector to obtain a fourth vector; the fourth vector and the first vector are feature-fused and processed through a fully connected layer to obtain a fifth vector;

[0090] The adaptive gating mechanism layer is used to dynamically adjust the importance of features based on the fifth vector through GRU units (or LSTM units can also be used) according to context information and output a sixth vector;

[0091] The third context-aware module processes the sixth vector to obtain a context-aware feature vector as the seventh vector;

[0092] The feature reconstruction layer is used to perform feature reconstruction on the seventh vector through an autoencoder, and perform secondary fusion on the vector after feature reconstruction and the fusion feature to obtain an eighth vector;

[0093] The output layer is constructed based on a sequentially connected fully connected layer and a softmax function; the output layer is used to perform classification prediction on the eighth vector to generate a prediction result of the relationship between entities.

[0094] The loss function of the context-aware multi-modal neural network during training is:

[0095] where L represents the loss function, N represents the number of entity samples, C represents the number of entity categories, y ij represents the true label corresponding to the entity prediction result, pij represents the entity prediction result, M represents the number of samples of the relationships between entities, R represents the number of relationship categories, z ij represents the true label corresponding to the relationship prediction result between entities, q ij represents the relationship prediction result between entities, represents the text data feature vector, represents the image data feature vector, represents the table data feature vector, ⊙ represents element-wise multiplication, T represents transpose, A represents the set interaction matrix, ω i represents the context weight, and this loss function term is used to capture the high-order relationships between different modal data, represents the context feature vector of the i-th sample, represents the true label corresponding to the context feature vector, and this loss function term is used to enhance the network's sensitivity to the context, W k represents the k-th parameter matrix. The adaptive regularization term is calculated through the parameter matrix to prevent overfitting. λ, η, and θ all represent regularization coefficients, and α1, α2, α3, α4, and α5 all represent weight coefficients. The weight coefficients are obtained as follows:

[0096] Calculate the exponential decay function value of the loss function term corresponding to the currently to-be-obtained weight coefficient as the first value; calculate the sum of the exponential decay function values of all loss function terms as the second value; take the ratio of the first value to the second value as the weight coefficient. That is where ξ represents the difficulty coefficient, L1 is the loss function term corresponding to the currently to-be-obtained weight coefficient, Li is the loss function term corresponding to each weight coefficient, such as the above are all used as loss function terms, and W is the number of loss function terms.

[0097] S40. Based on the entities and the relationships between the entities, construct a knowledge graph;

[0098] In this embodiment, the process of constructing the knowledge graph is as follows:

[0099] Construct a graph based on the entities and the relationships between the entities as the initial knowledge graph;

[0100] Perform message passing on the initial knowledge graph through a graph neural network (preferably GNN in the present invention, and in other embodiments, GCN, GAT, etc. can be selected for message passing) to extract the feature representation of each entity;

[0101] Calculate the similarity between each pair of entities (such as cosine similarity or Jaccard similarity) by combining the feature representations of the entities, and cluster those with similarity higher than the set value through a clustering algorithm (such as DBSCAN clustering algorithm, etc.);

[0102] After clustering, extract the relationships between entities through a graph traversal algorithm (such as BFS, DFS, etc.), and perform relationship conflict detection through a constraint propagation algorithm. If there are conflicting relationships (that is, according to domain knowledge and common sense, define a series of logical constraints to ensure the logical consistency of the knowledge in the knowledge graph, such as some relationships like mutually exclusive relationships cannot hold simultaneously. For example, a person cannot be both the father and son of another person; transitive relationships: some relationships are transitive. For example, if A is the parent class of B, and B is the parent class of C, then A is also the parent class of C), then delete or retain the relationships between entities in the initial knowledge graph through a heuristic algorithm (including arc consistency (such as checking whether a person is both the father and son of another person, if so, delete one of the relationships), path consistency (such as whether A is the parent class of B and B is the parent class of C exists, if it exists, then add A is the parent class of C)), and use the knowledge graph processed by the heuristic algorithm as the finally constructed knowledge graph.

[0103] Delete or retain the relationships between entities in the initial knowledge graph through a heuristic algorithm, and the method is as follows:

[0104] Obtain the context information of the conflict relationship and encode it to get an encoded vector; weight the encoded vector with a set first learnable parameter, and perform activation processing on the weighted encoded vector to obtain a context feature vector; that is, the commonly used σ(W1×x + b1), where W1 and b1 are both learnable parameters, and x is the weighted vector,

[0105] Fuse the feature vectors of the multi-modal data corresponding to the conflict relationship to obtain a multi-modal feature vector; weight the multi-modal feature vector with a set second learnable parameter, and perform activation processing on the weighted multi-modal feature vector to obtain a multi-modal information feature vector;

[0106] Combine the multi-modal data corresponding to the conflict relationship and the context information of the conflict relationship, and calculate the dynamically adjusted weight through an attention mechanism; that is, use the multi-modal data and context information as key vectors, and the feature of the current relationship as the query vector, and calculate the dynamically adjusted weight through the formula of the attention mechanism, which is prior art and will not be elaborated here.

[0107] Based on the dynamically adjusted weights, the context feature vector, the multi-modal information feature vector, and the confidence corresponding to the conflict relationship are weighted and summed respectively, and the conflict relationships whose sum is less than the set sum threshold are deleted. If the sums corresponding to multiple conflict relationships are the same, the relationship with the smallest number of deleted relationships can be selected, that is, the optimal solution is selected by minimizing conflicts to maintain the consistency of the knowledge graph.

[0108] Through the constraint propagation and conflict resolution algorithms, the present invention realizes the automation of maintaining the consistency of the knowledge graph, reduces manual intervention, ensures the logical consistency of the knowledge in the knowledge graph, and improves the accuracy and reliability of the knowledge graph.

[0109] In addition, the constructed knowledge graph can be incrementally fused, that is, new data is incrementally extracted to generate new entities and relationships, and the newly extracted entities and relationships are fused with the existing knowledge graph to update the graph structure.

[0110] S50, map the knowledge graph to the table structure of the heterogeneous database to complete the construction of the heterogeneous database;

[0111] In this embodiment, the entities in the knowledge graph are mapped to the table structure of the heterogeneous database, and the relationships in the graph are used as foreign keys to establish associations between different tables to construct a heterogeneous database. The constructed heterogeneous database can be applied in multiple scenarios. For example, for the financial data in the material and equipment management system, after constructing the heterogeneous database, the financial data can be integrated to generate financial reports and perform budget control (such as by analyzing the expenditure of each project to timely discover overspending problems).

[0112] In summary, through multi-modal input fusion, multi-scale feature extraction, introducing context awareness, adaptive gating mechanism, etc., the present invention effectively fuses data of various modalities such as text, images, and tables, generates high-quality feature vectors, improves the efficiency, richness, and accuracy of knowledge graph construction, and through the constraint propagation algorithm and conflict resolution strategy, ensures the logical consistency of the knowledge graph, avoids contradictions and redundancies, improves the reliability and credibility of the knowledge graph and the heterogeneous database constructed based on the knowledge graph, and thus solves the problems of large data integration difficulty, lack of context information, difficult consistency maintenance, and weak dynamic update ability existing in the existing heterogeneous database construction methods.

[0113] A heterogeneous database construction system based on a knowledge graph according to the second embodiment of the present invention, the system includes:

[0114] A data acquisition module configured to acquire multi-modal data of the heterogeneous database to be constructed as input data; the input data includes text data, image data, and table data;

[0115] A feature fusion module, configured to preprocess the input data to obtain preprocessed data; extract feature vectors of the preprocessed data and perform multi-modal feature fusion to obtain fused features;

[0116] A relation extraction module, configured to input the fused features into a pre-constructed context-aware multi-modal neural network to automatically extract the relations between entities;

[0117] A knowledge graph construction module, configured to construct a knowledge graph based on the entities and the relations between the entities;

[0118] A database construction module, configured to map the knowledge graph to the table structure of a heterogeneous database to complete the construction of the heterogeneous database;

[0119] The context-aware multi-modal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence;

[0120] The three context-aware modules are all based on a multi-head attention mechanism and a Transformer encoder connected in sequence, and are all used to obtain context-aware feature vectors; the multi-scale feature extraction layer is used to extract features of different scales; the graph construction layer is used to construct a graph structure; the hierarchical neural network layer is used to extract node features; the feature fusion layer is used to fuse the input features through an attention mechanism; the adaptive gating mechanism layer is used to adjust features through one or more GRU units; the feature reconstruction layer is used for feature reconstruction and fusion; the output layer is constructed based on a fully connected layer and a softmax function connected in sequence.

[0121] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working process and related descriptions of the above-described system can refer to the corresponding process in the foregoing method embodiments, and will not be elaborated herein.

[0122] It should be noted that the above-described heterogeneous database construction system based on a knowledge graph only takes the division of the above functional modules as an example. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be combined into one module, or further split into multiple sub-modules to complete all or part of the functions described above. For the names of the modules and steps involved in the embodiments of the present invention, they are only used to distinguish each module or step, and are not regarded as an improper limitation of the present invention.

[0123] An electronic device according to a third embodiment of the present invention includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned method for constructing a heterogeneous database based on a knowledge graph.

[0124] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned method for constructing a heterogeneous database based on a knowledge graph.

[0125] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes and related descriptions of the above-described electronic device and readable storage medium can refer to the corresponding processes in the foregoing method examples, and will not be elaborated herein.

[0126] Those skilled in the art should be able to realize that the modules and method steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. The programs corresponding to the software modules and method steps can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in the form of electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0127] The terms "first", "second", "third", etc. are used to distinguish similar objects, rather than to describe or represent a specific order or sequence.

[0128] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims

1. A method for constructing a heterogeneous database based on a knowledge graph, characterized in that: The method includes: S10, obtaining multimodal data of a heterogeneous database to be constructed as input data; the input data includes text data, image data, and table data; S20, preprocessing the input data to obtain preprocessed data; extracting feature vectors of the preprocessed data and performing multimodal feature fusion to obtain fused features; S30, inputting the fusion features into a pre-built context-aware multimodal neural network to automatically extract the relationships between entities; S40, constructing a knowledge graph based on the entities and the relationships between the entities; S50, mapping the knowledge graph into the table structure of the heterogeneous database to complete the construction of the heterogeneous database; The context-aware multimodal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence; The three context-aware modules are all constructed based on a multi-head attention mechanism and a Transformer encoder connected in sequence, and are all used to obtain context-aware feature vectors; the multi-scale feature extraction layer is used to extract features of different scales; the graph construction layer is used to construct a graph structure; the hierarchical neural network layer is used to extract node features; the feature fusion layer is used to fuse input features through an attention mechanism; the adaptive gating mechanism layer is used to adjust features through one or more GRU units; the feature reconstruction layer is used for feature reconstruction and fusion; the output layer is constructed based on a fully connected layer and a softmax function connected in sequence.

2. The method for constructing a heterogeneous database based on a knowledge graph according to claim 1, characterized in that: The preprocessing includes format conversion, data cleaning, and noise removal.

3. The method for constructing a heterogeneous database based on a knowledge graph according to claim 2, characterized in that: The fusion features are input into a pre-built context-aware multimodal neural network to automatically extract the relationship between entities, and the method is as follows: Processing the fused features by the first context-aware module to obtain a context-aware feature vector as a first vector; Performing feature extraction on the first vector using convolution kernels of different scales of the multi-scale feature extraction layer to obtain a multi-scale feature vector as a second vector; Based on the second vector, construct a graph structure through the graph construction layer, and output the graph structure and node features as first node features; Based on the graph structure, message passing is performed through a graph attention network to output a second node feature; the first node feature and the second node feature are added to obtain a third node feature; Based on the graph structure, message passing is performed through a graph convolutional network to output a fourth node feature; the third node feature is added to the fourth node feature to obtain a fifth node feature; based on the graph structure, message passing is performed through GraphSAGE to output a sixth node feature; Adding the fifth node feature and the sixth node feature to obtain a final node feature; The hierarchical neural network layers include a graph attention network, a graph convolutional network, and GraphSAGE; Processing the final node feature by the second context-aware module to obtain a context-aware feature vector as a third vector; The weight of each relationship type in the third vector is calculated by the attention mechanism of the feature fusion layer, and the third vector is weighted to obtain a fourth vector; the fourth vector is feature-fused with the first vector, and processed by a fully connected layer to obtain a fifth vector; Based on the fifth vector, dynamically adjusting the importance of the feature according to context information through the GRU unit of the adaptive gating mechanism layer, and outputting a sixth vector; Processing the sixth vector by the third context-aware module to obtain a context-aware feature vector as a seventh vector; Reconstructing the seventh vector through the autoencoder of the feature reconstruction layer, and fusing the reconstructed vector with the fusion feature for a second time to obtain an eighth vector; the feature reconstruction layer includes an autoencoder; The eighth vector is classified and predicted through the output layer to generate a prediction result of the relationship between entities.

4. The method for constructing a heterogeneous database based on a knowledge graph according to claim 3 is characterized in that: The loss function of the context-aware multimodal neural network during training is: Among them, L represents the loss function, N represents the number of entity samples, C represents the number of entity categories, and y ij Represents the true value label corresponding to the entity prediction result, p ij represents the entity prediction result, M represents the number of samples of the relationship between entities, R represents the number of relationship categories, and Z ij Represents the true value label corresponding to the predicted result of the relationship between entities, q ij Represents the relationship prediction results between entities, represents the feature vector of text data, represents the image data feature vector, represents the feature vector of the tabular data, ⊙ represents element-by-element multiplication, T represents transposition, A represents the set interaction matrix, ω i represents the context weight, represents the context feature vector of the i-th sample, represents the true value label corresponding to the context feature vector, W k represents the kth parameter matrix, λ, η, θ represent regularization coefficients, and α1, α2, α3, α4, α5 represent weight coefficients.

5. The method for constructing a heterogeneous database based on a knowledge graph according to claim 4, characterized in that: The weight coefficient is obtained by: Calculate the exponential decay function value of the loss function term corresponding to the current weight coefficient to be obtained as the first value; Calculate the sum of the exponential decay function values ​​corresponding to all loss function items as the second value; The ratio of the first value to the second value is used as a weight coefficient.

6. The method for constructing a heterogeneous database based on a knowledge graph according to claim 5, characterized in that: Based on the entities and the relationships between the entities, a knowledge graph is constructed, and the method is as follows: Building a graph based on the entities and the relationships between the entities as an initial knowledge graph; Perform message passing on the initial knowledge graph through a graph neural network to extract feature representations of each entity; Combine the feature representations of each entity, calculate the similarity between each pair of entities, and cluster the entities with similarities higher than the set value through a clustering algorithm; After clustering, the relationship between entities is extracted through a graph traversal algorithm, and relationship conflict detection is performed through a constraint propagation algorithm. If there is a conflicting relationship, the relationship between entities in the initial knowledge graph is deleted or retained through a heuristic algorithm, and the knowledge graph processed by the heuristic algorithm is used as the final constructed knowledge graph; the constraint propagation algorithm includes arc consistency and path consistency.

7. The method for constructing a heterogeneous database based on a knowledge graph according to claim 6, characterized in that: The relationship between entities in the initial knowledge graph is deleted or retained by a heuristic algorithm, and the method is as follows: Acquire the context information of the conflict relationship and encode it to obtain a coding vector; weight the coding vector by setting a first learnable parameter, and activate the weighted coding vector to obtain a context feature vector; The feature vectors of the multimodal data corresponding to the conflict relationship are merged to obtain a multimodal feature vector; the multimodal feature vector is weighted by a set second learnable parameter, and the weighted multimodal feature vector is activated to obtain a multimodal information feature vector; Combining the multimodal data corresponding to the conflict relationship and the context information of the conflict relationship, and calculating the dynamic adjustment weight through the attention mechanism; Based on the dynamic adjustment weight, the confidences corresponding to the context feature vector, the multimodal information feature vector, and the conflict relationship are weighted and summed respectively, and the conflict relationship corresponding to the sum of which is less than a set threshold is deleted.

8. A heterogeneous database construction system based on knowledge graph, characterized in that: The system comprises: A data acquisition module is configured to acquire multimodal data of a heterogeneous database to be constructed as input data; the input data includes text data, image data, and table data; A feature fusion module is configured to preprocess the input data to obtain preprocessed data; extract feature vectors of the preprocessed data and perform multimodal feature fusion to obtain fused features; A relationship extraction module is configured to input the fused features into a pre-built context-aware multimodal neural network to automatically extract the relationship between entities; A graph construction module, configured to construct a knowledge graph based on the entities and the relationships between the entities; A database construction module is configured to map the knowledge graph into a table structure of a heterogeneous database to complete the construction of the heterogeneous database; The context-aware multimodal neural network is constructed based on a first context-aware module, a multi-scale feature extraction layer, a graph construction layer, a hierarchical neural network layer, a second context-aware module, a feature fusion layer, an adaptive gating mechanism layer, a third context-aware module, a feature reconstruction layer, and an output layer connected in sequence; The three context-aware modules are all constructed based on a multi-head attention mechanism and a Transformer encoder connected in sequence, and are all used to obtain context-aware feature vectors; the multi-scale feature extraction layer is used to extract features of different scales; the graph construction layer is used to construct a graph structure; the hierarchical neural network layer is used to extract node features; the feature fusion layer is used to fuse input features through an attention mechanism; the adaptive gating mechanism layer is used to adjust features through one or more GRU units; the feature reconstruction layer is used for feature reconstruction and fusion; the output layer is constructed based on a fully connected layer and a softmax function connected in sequence.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor, and a memory communicatively coupled to at least one of the processors; Among them, the memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the method for constructing a heterogeneous database based on a knowledge graph as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by a computer to implement the method for constructing a heterogeneous database based on a knowledge graph as described in any one of claims 1 to 7.

Images