A multi-modal knowledge graph representation learning method based on a graph neural network

Abstract

The application discloses a multi-modal knowledge graph representation learning method based on a graph neural network, which comprises the following steps: obtaining multi-modal text features, visual features and numerical features and converting the features into super points; confirming a target node super point and neighbor node super points from the super points; calculating the relationship attention of the neighbor node super points relative to the target node super points; taking the edges between the target node and the neighbor nodes as adjacent edges, fusing the neighbor nodes and the adjacent edges in a cross-correlation manner to obtain entity relationship fusion information; constructing a relationship aggregation function through the relationship attention and the entity relationship fusion information; updating the super points and the relationship attention based on the relationship aggregation function until a convergent relationship graph neural network model is obtained, and the multi-modal knowledge graph representation learning is completed; the application effectively models the information interaction between different entities with multi-modal information and in the same mode and between different modes; different entities and different numbers of modes are processed in a flexible form; and better graph representation learning capability is realized.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing, and more specifically to a multimodal knowledge graph representation learning method based on graph neural networks. Background Technology

[0002] A knowledge graph is a knowledge base that represents entities in the objective world and the relationships between them in the form of graphs. Entities can be real-world objects or abstract concepts, while relationships represent the connections between entities. Knowledge graphs can represent human knowledge in a structured form. Through knowledge representation and reasoning techniques, they can provide artificial intelligence systems with processable prior knowledge, enabling them to solve complex tasks like humans. Therefore, knowledge graphs, which describe common sense and facts, have become a widely used knowledge representation method in academia and industry. Meanwhile, graph neural networks have also demonstrated excellent performance in information propagation and relational induction bias. Considering that knowledge graphs themselves are a type of graph-structured data, using graphs to build connections between knowledge and data, and applying graph neural network technology, holds promise for combining knowledge and data to achieve better interpretable and trustworthy artificial intelligence technologies. On the one hand, by leveraging the advantages of graph neural networks in learning node and relation representations, we can better learn the embedded representations of entities and relations in knowledge graphs, improve tasks such as relation extraction, help construct knowledge graphs, and enhance tasks such as link prediction, thus helping to complete knowledge graphs. On the other hand, by leveraging the advantages of graph neural networks in information propagation and reasoning, we can more effectively introduce information from knowledge graphs into application tasks, thereby improving the application effects in fields such as text mining and recommendation systems, and providing interpretable models.

[0003] Current knowledge graph representation learning methods based on graph neural networks (GNNs) are mostly focused on unimodal knowledge graphs, neglecting the fact that in real-life scenarios, entities in knowledge graphs are not limited to textual descriptions but may also include images, speech, and other modalities. Existing GNN-based knowledge graph representation learning methods typically only work with static knowledge graphs with textual descriptions and cannot model multimodal knowledge graphs. With the increasing volume of multimodal data, multimodal knowledge graph representation learning has become a research hotspot in knowledge graph representation learning and applications. To handle multimodal information, researchers have proposed many multimodal knowledge graph embedding methods. For example, the KBLRN (Alberto Garcia-Duran, Mathias Niepert: KBLRN: End-to-End Learning of Knowledge Base Representations with Latent, Relational, and Numerical Features. UAI 2018:372-381) method is the first to learn knowledge graph representations by integrating latent, relational, and numerical features. The MKBE method (Pouya Pezeshkpour, Liyan Chen, Sameer Singh: Embedding Multimodal Relational Data for Knowledge Base Completion. EMNLP 2018:3208-3218) uses different neural encoders for different modal information and combines them with existing relational models to learn the embeddings of multimodal data entities. The MKHAN method (Yingying Zhang, Shengsheng Qian, Quan Fang, Changsheng Xu: Multi-modal Knowledge-aware Hierarchical Attention Network for Explainable Medical Question Answering. ACM Multimedia 2019:1089-1097) proposes a hierarchical attention network that encodes multimodal information into medical knowledge graphs for explainable medical question answering.The MMRFAN (Yingying Zhang, Quan Fang, Shengsheng Qian, Changsheng Xu: Multi-modal Multi-relational Feature Aggregation Network for Medical Knowledge Representation Learning. ACM Multimedia 2020:3956-3965) method proposes a multimodal medical knowledge representation learning model based on graph neural networks.

[0004] However, the above-mentioned multimodal knowledge graph embedding methods still cannot address the following challenges: (1) Different entities with multimodal information have information interactions within the same modality and between different modalities, but traditional simple connection or attention methods cannot handle this attribute. Although MMRFAN uses adversarial feature learning to unify multimodal representations, this method cannot achieve end-to-end training, thus affecting the representation learning effect. (2) The multimodal information of entities is unbalanced, for example, some product entities lack the sound modality. MMRFAN uses a fixed-size sampling mechanism to solve this problem, but this operation cannot fully utilize the graph structure information. (3) Multimodal knowledge graphs are heterogeneous graphs with multiple relationships, and traditional graph neural network methods cannot effectively learn representations for such graphs. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a multimodal knowledge graph representation learning method based on graph neural networks, which solves the problems of imbalanced modal information and poor learning performance in existing technologies.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0007] A multimodal knowledge graph representation learning method based on graph neural networks is provided, which includes the following steps:

[0008] S1. Obtain modal information from the multimodal knowledge graph and learn the modal information through a pre-trained encoder to obtain text features, visual features, and numerical features;

[0009] S2. Convert text features, visual features, and numerical features into superpoints;

[0010] S3. Confirm that the target node is an outlier from the outliers, and confirm that the neighboring nodes of the target node are also outliers.

[0011] S4. Construct an initial relational graph neural network and calculate the relational attention of the neighbor node superpoint relative to the target node superpoint.

[0012] S5. Take the edge between the target node and the neighboring nodes as the adjacent edge, and fuse the neighboring nodes and the adjacent edge through cross correlation to obtain the entity relationship fusion information;

[0013] S6. Construct a relation aggregation function by fusing information from relation attention and entity relations;

[0014] S7. Update the superpoints and relation attention based on the relation aggregation function until a converged relation graph neural network model is obtained, thus completing the multimodal knowledge graph representation learning.

[0015] Furthermore, in step S1:

[0016] Modal information includes textual information, visual information, and numerical information;

[0017] The pre-trained encoder includes a BERT model for acquiring textual and numerical features, and two fully connected layers. The BERT model includes 12 Transformer layers, each containing 768 hidden neurons and 12 multi-head self-attention units. The BERT model converts textual information into initial textual features, the first fully connected layer converts numerical information into numerical information, and the second fully connected layer maps the initial textual features and numerical information to obtain 768-dimensional textual features containing numerical features.

[0018] The pre-trained encoder includes a VGG-16 convolutional neural network model for acquiring visual features. The VGG-16 convolutional neural network model includes a feature module and a classification module. The feature module includes a 13-layer 3*3 convolutional kernel neural network, and the classification module includes two fully connected layers. The input is a 224*224 RGB image, and 4096-dimensional visual features are obtained.

[0019] Furthermore, the expression for the superpoint in step S2 is:

[0020]

[0021] Where e m For the m-th superpoint, x t,m Let x be the m-th text feature. v,m For the m-th visual feature, x n,m For the m-th numerical feature, Let i be the low-rank factor of the m-th text feature. Let i be the i-th low-rank factor of the m-th visual feature. Let i be the i-th low-rank factor of the m-th numerical feature. Let r be the Hadamard product, r be the total number of low-rank factors, m ∈ M, and M be the number of superpoints.

[0022] Furthermore, the expression for relational attention in step S4 is:

[0023]

[0024] b kjr (l) =W a [e k (l) ||r kj (l) ||e j (l) ]

[0025] Where α ijr (l) For the k-th target node superpoint e in the l-th layer relational graph neural network k (l) And the j-th neighbor node exceeds point e j (l) Relational attention, exp(·) is an exponential function with base e, b kjr (l) In the l-th layer relational graph neural network, the attention score of node j linked to node k through relation r is b. knr (l) For the l-th layer relational graph neural network, node j is linked to node n through relation r, where r is the attention score. kj (l) W represents the neighboring edges between the k-th target node and the j-th neighbor node in the l-th layer relational graph neural network. a [·] represents the attention weight matrix. Let be the set of neighboring nodes of the target node. Let M be the set of adjacent edges, || be the concatenation operation, k∈M, j∈M, and M be the number of superpoints.

[0026] Furthermore, the specific process of obtaining entity relationship fusion information in step S5 is as follows:

[0027] According to the formula:

[0028]

[0029] Obtain entity relationship fusion information in For r kj (l) The complex conjugate of , * is convolution.

[0030] Furthermore, the specific process of constructing the relation aggregation function in step S6 is as follows:

[0031] According to the formula:

[0032]

[0033] Obtain the k-th target node superpoint of the (l+1)-th layer relational graph neural network. Where R is the set of relations; N is the number of adjacent edges r that the k-th target node passes through. kj (l) The set of connected neighbor nodes; For regularization terms; Let r be the weight matrix between the k-th target node superpoint and the j-th neighbor node superpoint in the l-th layer relational graph neural network; r be the relational neighbor edges. kj (l) Through the formula:

[0034] r kj (l+1) =W r (l) r kj (l)

[0035] Update; where W r (l) This is the relation weight matrix.

[0036] The beneficial effects of this invention are as follows:

[0037] 1. The information fusion module proposed in this invention can capture the multimodal information of entities in a multimodal knowledge graph and can effectively model the information interaction process between different entities with multimodal information within the same modality and between different modalities. In addition, the operations in this fusion module are differentiable to ensure that the method of this invention can be trained end-to-end through a neural network.

[0038] 2. In low-rank multimodal fusion, this invention uses decomposition factors to handle the modal imbalance characteristics of entities, and handles different entities with different numbers of modes in a flexible manner.

[0039] 3. In order to capture the structural information of the graph, this patent proposes a special relational graph neural network. Compared with traditional graph neural networks, this patent uses a relation-specific attention mechanism and entity relation fusion operation to improve the representation learning of multimodal knowledge graphs, and achieves better graph representation learning capabilities.

[0040] 4. This invention fuses the visual modalities, digital features, and textual descriptions of knowledge graphs with low-rank tensors, which can capture the intramodal and intermodal interaction information of entities; it can capture natural graph structures that capture multimodal information and multimodal knowledge. Attached Figure Description

[0041] Figure 1 This is a flowchart of the present invention;

[0042] Figure 2This is a diagram illustrating the node update process in the embodiment. Detailed Implementation

[0043] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0044] like Figure 1 As shown, this multimodal knowledge graph representation learning method based on graph neural networks includes the following steps:

[0045] S1. Obtain modal information from the multimodal knowledge graph and learn the modal information through a pre-trained encoder to obtain text features, visual features, and numerical features;

[0046] S2. Convert text features, visual features, and numerical features into superpoints;

[0047] S3. Confirm that the target node is an outlier from the outliers, and confirm that the neighboring nodes of the target node are also outliers.

[0048] S4. Construct an initial relational graph neural network and calculate the relational attention of the neighbor node superpoint relative to the target node superpoint.

[0049] S5. Take the edge between the target node and the neighboring nodes as the adjacent edge, and fuse the neighboring nodes and the adjacent edge through cross correlation to obtain the entity relationship fusion information;

[0050] S6. Construct a relation aggregation function by fusing information from relation attention and entity relations;

[0051] S7. Update the superpoints and relation attention based on the relation aggregation function until a converged relation graph neural network model is obtained, thus completing the multimodal knowledge graph representation learning.

[0052] In step S1:

[0053] Modal information includes textual information, visual information, and numerical information;

[0054] The pre-trained encoder includes a BERT model for acquiring textual and numerical features, and two fully connected layers. The BERT model includes 12 Transformer layers, each containing 768 hidden neurons and 12 multi-head self-attention units. The BERT model converts textual information into initial textual features, the first fully connected layer converts numerical information into numerical information, and the second fully connected layer maps the initial textual features and numerical information to obtain 768-dimensional textual features containing numerical features.

[0055] The pre-trained encoder includes a VGG-16 convolutional neural network model for acquiring visual features. The VGG-16 convolutional neural network model includes a feature module and a classification module. The feature module includes a 13-layer 3*3 convolutional kernel neural network, and the classification module includes two fully connected layers. The input is a 224*224 RGB image, and 4096-dimensional visual features are obtained.

[0056] The expression for the superpoint in step S2 is:

[0057]

[0058] Where e m For the m-th superpoint, x t,m Let x be the m-th text feature. v,m For the m-th visual feature, x n,m For the m-th numerical feature, Let i be the low-rank factor of the m-th text feature. Let i be the i-th low-rank factor of the m-th visual feature. Let i be the i-th low-rank factor of the m-th numerical feature. Let r be the Hadamard product, r be the total number of low-rank factors, m ∈ M, and M be the number of superpoints.

[0059] The expression for relational attention in step S4 is:

[0060]

[0061] b kjr (l) =W a [e k (l) ||r kj (l) ||e j (l) ]

[0062] Where α ijr (l) For the k-th target node superpoint e in the l-th layer relational graph neural network k (l)And the j-th neighbor node exceeds point e j (l) Relational attention, exp(·) is an exponential function with base e, b kjr (l) In the l-th layer relational graph neural network, the attention score of node j linked to node k through relation r is b. knr (l) For the l-th layer relational graph neural network, node j is linked to node n through relation r, where r is the attention score. kj (l) W represents the neighboring edges between the k-th target node and the j-th neighbor node in the l-th layer relational graph neural network. a [·] represents the attention weight matrix. Let be the set of neighboring nodes of the target node. Let M be the set of adjacent edges, || be the concatenation operation, k∈M, j∈M, and M be the number of superpoints.

[0063] The specific process of obtaining entity relationship fusion information in step S5 is as follows:

[0064] According to the formula:

[0065]

[0066] Obtain entity relationship fusion information in For r kj (l) The complex conjugate of , * is convolution.

[0067] The specific process of constructing the relation aggregation function in step S6 is as follows:

[0068] According to the formula:

[0069]

[0070] Obtain the k-th target node superpoint of the (l+1)-th layer relational graph neural network. Where R is the set of relations; N is the number of adjacent edges r that the k-th target node passes through. kj (l) The set of connected neighbor nodes; For regularization terms; Let r be the weight matrix between the k-th target node superpoint and the j-th neighbor node superpoint in the l-th layer relational graph neural network; r be the relational neighbor edges. kj (l) Through the formula:

[0071] r kj (l+1) =W r (l) r kj(l)

[0072] Update; where W r (l) This is the relation weight matrix.

[0073] In one embodiment of the present invention, assuming there are a total of 7 nodes, the update diagram of target node 4 (corresponding to step S6) is as follows: Figure 2 As shown: This method adds self-edges to each node to avoid over-smoothing caused by sharing information from neighboring nodes. The relational graph neural network module aggregates the relational information of neighboring nodes 1, 2, 6, and 7 and their corresponding edges to the target node 4. Simultaneously, during the backpropagation of the graph neural network, the relational aggregation function of the target node is adaptively updated through the relational weight matrix (corresponding to step S6) and the entity relation fusion operation (corresponding to step S5).

[0074] Performance verification on knowledge graph completion tasks:

[0075] Knowledge graph completion is based on existing knowledge graphs. It uses knowledge graph representation learning to reason about unknown knowledge and complete the knowledge graph. The general result is to return an entity ranking (finding the correct answer in the entity set). The effectiveness of the representation learning is measured by comparing the ranking position of the correct entity (the first entity in the returned ranking is the correct answer).

[0076] This invention was experimented on two datasets, FB15k-237 and WN18RR. FB15k-237, from Freebase (a content-sharing website similar to Wikipedia, where entries are presented in structured data format), contains 14,541 entities and 237 relations, primarily related to sports, film, and other knowledge information. For multimodal information, each entity in FB15k-237 has sentence-level text information, 13,444 entities have image information, and 29,395 numerical information (some entities contain multiple numerical information, such as height and weight). WN18RR, from WordNet (an English dictionary based on cognitive linguistics, jointly designed by psychologists, linguists, and computer engineers at Princeton University; it doesn't simply arrange words alphabetically but forms a "network of words" according to their meanings), contains 40,943 entities and 11 relations. Because WordNet is a semantic web based on an English dictionary, its entities only contain text information. The evaluation metrics used in this patent are as follows: Hits@s, Mean Rank (MR), and Mean Reciprocal Rank (MRR). Hits@s describes the probability that the correct entity answer appears in the first s items; the higher the score, the better the effect. MR describes the average rank of the correct entity answer; the lower the score, the better the effect. MRR describes the average of the reciprocals of the ranks of the correct entities; the higher the score, the better the effect. The experimental results are shown in Table 1:

[0077] Table 1. Results of the Superpoint Relationship Graph Attention Network in the Knowledge Graph Completion Task

[0078]

[0079] This invention (hyper-node relational graph attention, HRGAT) achieved the best results in most metrics across both datasets.

[0080] TransE embeds entities and relationships from a knowledge graph into a low-dimensional linear space and uses a scoring function to determine the probability that two entities have a certain relationship.

[0081] RotatE: A derivative method that processes TransE using different mapping and transformation methods.

[0082] TorusE is an improvement based on the limitations of TransE, replacing the mapping space from a regular vector space with a Lie group.

[0083] DistMult: A semantic matching model that obtains entity embedding representations through a linear neural network and characterizes entity-to-entity relationships through a diagonal matrix, where each element of the diagonal matrix represents a different relationship.

[0084] ConvE: Uses 2D convolution to reshape the head entity and relation into a 2D matrix to simulate the interaction between the entity and relation, and then calculates the score of the triple with the tail entity based on similarity.

[0085] KMAE is an improved version of ConvE, which uses different convolution kernels to extract features for entity attributes and relation attributes.

[0086] R-GCN: The first to apply graph neural networks to knowledge graph representation tasks, extending graph convolution operations on traditional undirected graphs to directed weighted graphs.

[0087] WGCN: Weighted Graph Neural Network that models the connection strength between two adjacent nodes of the same relation type.

[0088] KBLRN: A method for learning knowledge graph representations by integrating latent, relational, and numerical features.

[0089] MMRFAN: A multimodal medical knowledge representation learning model based on graph neural networks.

[0090] Meanwhile, the effects of different modules were tested on the FB15k-237 dataset, as shown in Table 2:

[0091] Table 2. Experimental Results on the Influence of Different Modules on Model Performance

[0092]

[0093] HRGAT-IF: By removing the information fusion module in this patent, and starting from the original knowledge graph without multimodal information, the impact of multimodal information on the original knowledge graph was verified.

[0094] HRGAT-LMF: The effectiveness of low-rank multimodal fusion operation is verified by replacing the low-rank multimodal fusion operation of information fusion block with vector concatenation operation.

[0095] HRGAT-RA: Removes relation-specific attention mechanisms from relation graph neural networks in order to verify the effectiveness of relation attention mechanisms on relation graph neural networks.

[0096] HRGAT-ERF: Removed the entity-relation fusion operation in the relational graph neural network in order to verify the impact of strong correlation between entities and relations and the effectiveness of the entity-relation fusion operation.

[0097] The removal of different modules weakens the experimental results, demonstrating the effectiveness of different modules in multimodal knowledge graph representation learning. HRGAT represents the experimental results of the complete model. HRGAT-IF verifies the information advantage of the multimodal knowledge graph compared to the original knowledge graph. HRGAT-LMF verifies that the effect is weakened after replacing the low-rank multimodal modules with concatenation operations, indicating the effectiveness of low-rank multimodal fusion operations. HRGAT-RA verifies the effectiveness of the attention mechanism. In particular, HRGAT-ERF reflects that the experimental results decrease the most after removing the entity-relationship fusion operation in the relational graph neural network, verifying that the entity-relationship fusion operation adopted in this patent has the greatest impact on multimodal knowledge graph representation learning.

[0098] This invention addresses the multimodal characteristics of knowledge graphs by proposing an information fusion module. In this module, the superpoint relation graph attention network (the entire invention process) first obtains the pre-trained embedding vectors for each modality of an entity. Then, it uses low-rank multimodal fusion to obtain the multimodal representation of each entity, modeling the information interaction process between different entities within the same modality and between different modalities. The nodes from which multimodal information is obtained are called superpoints. Furthermore, all operations in the low-rank multimodal fusion are differentiable to ensure that the method of this patent can be trained end-to-end through a neural network.

[0099] This invention employs a decomposition factor to address the modal imbalance characteristics of entities in low-rank multimodal fusion, thereby flexibly handling different entities with varying numbers of modes.

[0100] In order to capture the structural information of graphs, this patent proposes a special relational graph neural network. Compared with traditional graph neural networks, this patent uses a relation-specific attention mechanism and entity-relation fusion operation to improve the representation learning of multimodal knowledge graphs, achieving better graph representation learning capabilities.

[0101] This invention fuses the visual modalities, digital features, and textual descriptions of knowledge graphs with low-rank tensors, enabling the capture of intramodal and intermodal interaction information of entities; it is also capable of capturing natural graph structures that capture multimodal information and multimodal knowledge.

Claims

1. A multi-modal knowledge graph representation learning method based on a graph neural network, characterized in that, The method comprises the following steps: S1, acquiring modality information of a multi-modal knowledge graph, and learning the modality information through a pre-training encoder to obtain text features, visual features and numerical features; The modality information comprises text information, visual information and numerical information; The pre-training encoder comprises a BERT model and two fully connected layers for acquiring the text features and the numerical features, the BERT model comprises 12 Transformer layers, each of which comprises 768 hidden neurons and 12 multi-head self-attention units; the text information is converted into initial text features through the BERT model, the numerical information is converted into digital information through the first fully connected layer, and the initial text features and the digital information are mapped through the second fully connected layer to obtain 768-dimensional text features comprising the numerical features; The pre-training encoder comprises a VGG-16 convolutional neural network model for acquiring the visual features, the VGG-16 convolutional neural network model comprises a feature module and a classification module, the feature module comprises a 13-layer 3*3 convolution kernel neural network, and the classification module comprises two fully connected layers; an input is an RGB image of 224*224 to obtain 4096-dimensional visual features; S2, converting the text features, the visual features and the numerical features into hyperpoints; The expression of the hyperpoint is: wherein is the m th superpoint, is the m th text feature, is the m th visual feature, is the m th numerical feature, is the m th low-rank factor of the i th text feature, is the m th low-rank factor of the i th visual feature, is the m th low-rank factor of the i th numerical feature, is the Hadamard product, r is the total number of low-rank factors, , M is the number of superpoints; S3, confirming a target node hyperpoint from the hyperpoints, and confirming neighbor node hyperpoints of the target node hyperpoint; S4, constructing an initial relation graph neural network, and calculating relation attention of the neighbor node hyperpoints relative to the target node hyperpoint; The expression of the relation attention is: wherein is the l layer relational graph neural network k is the target node super node j is the neighbor node super node is the relation attention, e is an exponential function with a natural constant is the l layer relational graph neural network j is the attention score of the node r linked by the relation k to the node is the l layer relational graph neural network j is the attention score of the node r linked by the relation n to the node is the l layer relational graph neural network k is the relation edge between the j target node and the neighbor node is the attention weight matrix, is the neighbor node set of the target node, is the edge set, is the concatenation operation, , , M is the number of super nodes; S5, taking edges between the target node and the neighbor nodes as neighbor edges, fusing the neighbor nodes and the neighbor edges through cross correlation to obtain entity relation fusion information, and the specific process is: According to the formula: Obtaining entity relation fusion information ; wherein is a complex conjugate, is a convolution; S6, constructing a relation aggregation function through the relation attention and the entity relation fusion information; S7, updating the hyperpoint and the relation attention based on the relation aggregation function until a convergent relation graph neural network model is obtained, and multi-modal knowledge graph representation learning is completed.

2. The multi-modal knowledge graph representation learning method based on a graph neural network according to claim 1, characterized in that, The specific process of constructing the relation aggregation function in step S6 is: According to the formula: A first l layer relational graph neural network is obtained k The target node supernode ; wherein R is a relationship set; is a target node supernode k The neighbor node set connected by the relationship edge ; wherein is a regularization term; is a weight matrix between the target node supernode l of the first k layer relational graph neural network and the neighbor node supernode j ; the relationship edge is connected by the formula: updating; wherein is a relation weight matrix.

Images