Knowledge graph completion method fusing multi-modal content
By using a dual-track fusion method to perform feature fusion on the multimodal content and structure of knowledge graphs, the problems of incompleteness and inflexibility of knowledge graphs are solved, and more efficient knowledge graph completion and entity relationship interactivity are achieved, which can adapt to the fusion of more types of content.
Patent Information
- Application Number
- CN202111278020.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-30
- Publication Date
- 2025-12-09
- Estimated Expiration
- 2041-10-30
AI Technical Summary
Existing knowledge graphs suffer from incompleteness and inflexibility during the completion process due to the underutilization of multimodal content. Furthermore, existing models lack effectiveness in integrating different types of features and neglect the interactions between entities and relationships.
A dual-track fusion method is adopted to fuse features of the multimodal content and structure of the knowledge graph separately. Feature vectors are extracted using a pre-trained model, and feature selection and combination are performed through bilinear methods, attention mechanisms and gating units. Finally, link prediction is performed in conjunction with a decoder to complete the knowledge graph.
It achieves full utilization of multimodal content, improves the completion effect of knowledge graph, enhances the interactivity of entities and relationships, and has a flexible structure that is easy to expand and select appropriate decoders to adapt to the fusion of more types of content.
Smart Images
Figure CN114239675B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of data processing, and particularly relates to a knowledge graph completion method fusing multi-modal content. BACKGROUND
[0002] Existing large-scale knowledge graphs (KGs) have shown effectiveness in downstream applications, such as question answering and dialogue systems. In various fields, knowledge graphs effectively reflect the correlation between entities, which helps to improve the efficiency of the overall downstream task. However, due to confidentiality and other conditions, many knowledge graphs are far from complete and cannot fully express real and complete information. Therefore, knowledge graph embedding (KGE) models are proposed to solve the incompleteness of knowledge graphs. The model based on knowledge graph embedding represents entities and relationships in KGs as dense low-dimensional vectors (embeddings), and then uses the embeddings to predict new relationships.
[0003] Most knowledge graph embedding models only use triples contained in KGs, i.e. the relationship between the head entity and the tail entity, which is usually defined as structural information. However, different forms of content contained in entities, such as text and images associated with entities, are also very valuable for completing KGs. As Figure 1 , knowledge graphs not only contain triples, but also contain related text descriptions and images. These multi-modal content plays an important role in the reasoning and completion of knowledge graphs. This valuable data has not been effectively explored in most KGE models.
[0004] Recent research attempts to further improve the performance of knowledge graph embedding models by utilizing multiple types of features. These models are effective, but have limitations. First, different types of features contained in KGs are in different representation spaces, and it is not appropriate to directly integrate features in different representation spaces. Second, most models cannot deeply explore the interaction between entities and relationships in triples. However, a large number of studies have shown that the interaction between entities and relationships in KGE models should not be ignored. Finally, some models are not flexible and are not convenient for expansion, whether it is to merge more types of content or to expand to other expressive models. SUMMARY
[0005] In order to solve the above problems, the present application provides a knowledge graph completion method fusing multi-modal content, comprising the following steps:
[0006] Content fusion, the entity structure of the knowledge graph, the text associated with the entity, and the image associated with the entity are fused;
[0007] Structural fusion, the relationship structure of the knowledge graph, and the entity structure specific to the relationship are fused;
[0008] Link prediction task, the features after content fusion and structural fusion are input into the decoder for link prediction;
[0009] Graph completion, the output of the decoder is used to complete the knowledge graph or exclude incorrect knowledge.
[0010] Further, the content fusion specifically includes the following steps:
[0011] A feature fusion network based on a bilinear method, a feature selection network based on attention, and a feature combination network based on a gating unit are constructed.
[0012] Feature vectors of text, feature vectors of images, and structure vectors of entities and relationships are obtained.
[0013] According to the feature fusion network, cross-modal features at an element level are obtained.
[0014] According to the feature selection network, cross-modal features at a modal level are obtained.
[0015] According to the cross-modal features at the element level and the cross-modal features at the modal level, cross-modal features of entities are obtained through the feature combination network.
[0016] Further, the feature vectors of the text are obtained by a pre-training model BERT, the feature vectors of the images are obtained by a pre-training model ResNet18, and the structure vectors of the entities and relationships are obtained by a pre-training decoder.
[0017] Further, in the feature fusion stage based on the bilinear method in content fusion, the features in different representation spaces and different dimensions are projected into a unified representation space using a weight matrix corresponding to each modality, and the features of multiple modalities after projection are point multiplied, thereby obtaining cross-modal features at an element level.
[0018] In the feature selection stage based on attention in content fusion, the features in different representation spaces and different dimensions are projected into a unified representation space using a weight matrix corresponding to each modality, and the modal features are selected and fused by a multi-head attention mechanism, thereby obtaining cross-modal features at a modal level.
[0019] In the feature combination stage based on the gating unit in content fusion, the two groups of outputs of the bilinear network and the attention network are selected and combined by a gating mechanism, forming the final output as the cross-modal features of the entities.
[0020] Furthermore, the matrix weights in the bilinear network and the attention network are orthogonal to extract features from different perspectives, which are then used as one of the final loss terms during the training process.
[0021] Furthermore, the structural fusion process is similar to the content fusion process, including a feature fusion network based on the bilinear method, a feature selection network based on attention, and a feature combination network based on gating units, to form the final output as cross-modal features of the relationship.
[0022] Furthermore, if the triples output by the decoder are valid triples, the knowledge graph is completed; if the triples output by the decoder are invalid triples, erroneous knowledge is eliminated.
[0023] Furthermore, a link prediction task decoder is specified, which takes the cross-modal features of entities and the cross-modal features of relations as input, and outputs the scores of triples. The output scores indicate whether the triples are valid—if the score is higher than a threshold, the triples are judged as valid triples, and if the score is lower than or equal to the threshold, the triples are judged as invalid triples.
[0024] Furthermore, this application adopts a dual-track simultaneous approach, using two parallel tracks to perform content fusion and structural fusion of the knowledge graph multimodalities respectively, and then outputting them to the decoder respectively.
[0025] The beneficial effects of this invention are as follows: This invention provides a knowledge graph completion method that integrates multimodal content, making full use of entity-related text and images, and promoting the interactivity of entities and relationships; its structure is flexible and easy to expand. This embodiment is not limited to the integration of structure, text, and image content, but can also integrate other types of content without creative effort; the selection of decoder is also flexible, and the decoder for the knowledge graph completion task can be specified according to actual needs. Attached Figure Description
[0026] Figure 1 Knowledge graphs containing both textual and image information;
[0027] Figure 2 A schematic diagram of the process of this invention. Detailed Implementation
[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are only used to explain the invention and are not intended to limit the scope of protection of the present invention.
[0029] like Figure 1 The image shown is an example of a knowledge graph, which includes both textual and image information.
[0030] In combination Figure 2 , the embodiment adopts two parallel processing tracks, each of which contains three network components, and the implementation process is as follows:
[0031] Step 1, obtaining the feature vector of the multi-modal content
[0032] The first track performs fusion of multi-modal content, selects and aggregates key features of different types of multi-modal content from different angles. For a knowledge graph contains a set of constructed triples, that is, a set of (head entity, relationship, tail entity) represented as (h, r, t), such that h, t ∈ ε and where ε represents the entity set, represents the relationship set. The present application is implemented for contains three types of modal features, that is, a KG with multi-modal content includes features (h s ,t s ,h t ,t t ,h v ,t v ,r s ). h s and t s are the structural features of the head entity and the tail entity, h s ∈R x , t s ∈R x . h t and t t are the text features of the head entity and the tail entity, h t ∈R y , t t ∈R y . h v and t v are the visual (image) features of the head entity and the tail entity, h v ∈R z , t v ∈R z . The structural feature of the relationship is defined as r s , where r s ∈R x . By inputting the text content into the pre-trained model BERT, a 768-dimensional text embedding is obtained; by inputting the image content into ResNet18, a 512-dimensional image feature is obtained, and for multiple images, feature splicing is performed; the structural feature of the triple is obtained by the corresponding decoder pre-training.
[0033] Step 2, feature fusion network based on bilinear method
[0034] The bilinear network is used to fuse multimodal features in the same representation space. Given two different types of features, such as e s and e t where e can be h or t, the bilinear function is defined as
[0035]
[0036] where e f is the fused feature. To compress the dimension of matrix W and make the bilinear function suitable for fusing more types of features, W is split into two smaller dimension matrices U s and V t , then we have:
[0037]
[0038] where is the element-wise multiplication operation. Thus, the fusion process of e s , e t , e v related to the entity is as follows
[0039]
[0040] where W s , W t , W v are the weight matrices corresponding to the structural features, textual features, and visual features, respectively. σ is a nonlinear activation function (in this embodiment, the tanh activation function). b e is a bias.
[0041] This sub-step projects each type of feature into the same vector space and further performs the dot product operation on the elements to be fused. The nonlinear activation function not only maintains the interval consistency of the representation space but also increases the capacity of the model. The fusion method of this network can be regarded as element-level fusion.
[0042] Step 3, attention-based feature selection network
[0043] To select and fuse the key information in different multimodal content, an attention-based network is designed to calculate the task-specific weights of each type of feature. Then the weighted sum calculated using the weights is used as the fused feature.
[0044]
[0045]
[0046] where A e is an attention matrix. d m is the dimension of the feature after projection, e′i is the projected feature. Each feature is projected into the same vector space before computing the attention value. It can be found that this attention-based network is specifically designed to compute the weights at the modality level.
[0047] In addition, to encourage W e and W′ e in step 3 to learn from different perspectives, a constraint is made for W e and W′ e . That is, the bilinear-based network and the attention-based network can learn from different perspectives:
[0048]
[0049] where λ is a hyper-parameter with a small value, and ||…||2 returns the L2 norm of a matrix. Additionally, multi-head attention is used to stabilize the training process. Specifically, a single-head attention is repeated K times:
[0050]
[0051] The attention-based network can be understood as computing the weights of each type of feature according to the task, and then performing feature selection and fusion.
[0052] Step 4, Feature combination network based on gating unit
[0053] This step aims to further merge the features fused in step 2 and step 3. Specifically, a network based on gating unit is designed. Unlike the traditional gating unit using nonlinear activation function Sigmoid, Softmax is used here to perform feature selection and fusion from the modality level rather than the element level:
[0054]
[0055]
[0056] where e m is the feature vector after the fusion of the multi-modal content related to entity e (e can be h or t). Note that and Bilin(…) and Attm(…) can be controlled by β φ and In special cases, β φ or can be trained to 0, and the model only obtains features from e s . Therefore, the performance of the model is at least not lower than that of the extended model (i.e., the decoder).
[0057] Step 5, Feature fusion between structures
[0058] This step is the second track for this embodiment, which performs the fusion between the structures of the knowledge graph, i.e., the structural information of entities is aggregated into relations. Since the text and image matching the relations are usually missing in the knowledge graph, in order to effectively model the relations and improve the fusion between the structures, the structural features of entities are taken as the external modal information of the relations. This operation aims to enhance the interaction between entities and relations, and further improve the performance of the model. The relations are processed using steps similar to the entity feature fusion process:
[0059]
[0060] where h s and t s represent the structural information contained in h and t, respectively. Note that h s and t s share the same matrix W ε . This processing method is equivalent to fusing the structural information in the triplets. The relation r will supplement the necessary information from h s and t s , and thus enhance the interaction between entities and relations. The subsequent fusion is similar to steps 3 and 4, using the attention-based feature fusion network and the gate unit-based feature combination network, which will not be described here. Finally, the various types of features related to r (i.e., the information of the triplets) are fused into r m as the cross-modal features of the relation.
[0061] Step 6, Link Prediction Task Implementation
[0062] The features h m , t m , and r m obtained after steps 1-5, which fuse the structural, textual, and visual features, are taken as the cross-modal embeddings of entities and relations, and are input into the decoder for link prediction to achieve knowledge graph completion. Note that the decoder in this embodiment is chosen to be flexible and is not limited to a specific decoder. This embodiment chooses MuRE or InteractE as the decoder. MuRE is a translation-based model, and its scoring function is
[0063] ψ(h, r, t) = -d(Rh m , t m + r m ) 2 + b h + b t
[0064] where R is a matrix specific to the relation, d(·) is a Euclidean distance function, b h and bt are bias terms for h and t, respectively. InteractE is a neural model. Its scoring function is
[0065]
[0066] where denotes a circular convolution operation, φ(·) is a reshaping operation for h m and r m , f and g are activation functions ReLU and sigmoid, respectively. The fused features of the triplets are input to the decoder, and the output is the score for the triplet, indicating whether the triplet is valid.
[0067] In summary, only the preferred embodiments of the present application, not to limit the scope of protection of the present application, any equivalent changes and modifications made in accordance with the scope of the patent and the content of the specification, are within the scope of the present application.
Claims
1. A knowledge graph completion method integrating multimodal content, characterized in that, Includes the following steps: Content fusion involves fusing features from the entity structure of the knowledge graph, the text associated with the entity, and the images associated with the entity. Structural fusion involves fusing features from the relational structure of the knowledge graph and the relation-specific entity structure. The link prediction task inputs the features obtained from content fusion and structural fusion into the decoder to predict links. Knowledge graph completion, based on the decoder output, completes the knowledge graph or eliminates erroneous knowledge; The content integration specifically includes the following steps: Construct feature fusion networks based on bilinear methods, feature selection networks based on attention, and feature combination networks based on gating units; Obtain feature vectors from text, image, and structural vectors from entities and relationships; Based on the feature fusion network, element-level fused cross-modal features are obtained; Based on the aforementioned features, a network is selected to obtain cross-modal features fused at the modality level; Based on the cross-modal features fused at the element level and the cross-modal features fused at the modality level, the cross-modal features of the entity are obtained through the feature combination network.
2. The knowledge graph completion method integrating multimodal content according to claim 1, characterized in that: The feature vectors of the text are obtained by the pre-trained model BERT, the feature vectors of the images are obtained by the pre-trained model ResNet18, and the structure vectors of the entities and relations are obtained by the pre-trained decoder.
3. The knowledge graph completion method integrating multimodal content according to claim 2, characterized in that: In the feature fusion stage based on the bilinear method in content fusion, the weight matrix corresponding to each modality is used to project features from different representation spaces and different dimensions onto a unified representation space, and the projected features from multiple modalities are multiplied to obtain cross-modal features fused at the element level. In the attention-based feature selection stage of content fusion, weight matrices corresponding to each modality are used to project features from different representation spaces and different dimensions onto a unified representation space. Multi-head attention mechanism selects and fuses modal features to obtain cross-modal features fused at the modality level. In the feature combination stage based on gating units in content fusion, the modalities are selected and combined through gating mechanism based on the two sets of outputs of the bilinear network and the attention network to form the final output, which serves as the cross-modal feature of the entity.
4. The knowledge graph completion method integrating multimodal content according to claim 3, characterized in that: The matrix weights in the bilinear network and the attention network are orthogonal to extract features from different perspectives, which are then used as one of the final loss terms during the training process.
5. The knowledge graph completion method integrating multimodal content according to claim 1, characterized in that: The structural fusion process is similar to the content fusion process, and includes a feature fusion network based on the bilinear method, a feature selection network based on attention, and a feature combination network based on gating units to form the final output as cross-modal features of the relationship.
6. The knowledge graph completion method integrating multimodal content according to claim 5, characterized in that: If the triples output by the decoder are valid triples, the knowledge graph is completed; if the triples output by the decoder are invalid triples, erroneous knowledge is eliminated.
7. The knowledge graph completion method integrating multimodal content according to claim 6, characterized in that: The specified link prediction task decoder takes the cross-modal features of entities and the cross-modal features of relations as input, and outputs the scores of triples. The output scores indicate whether the triples are valid - if the score is higher than the threshold, the triples are judged as valid triples, and if the score is lower than or equal to the threshold, the triples are judged as invalid triples.
8. The knowledge graph completion method integrating multimodal content according to claim 1, characterized in that: This application adopts a dual-track approach, using two parallel tracks to perform content fusion and structural fusion of the knowledge graph multimodalities, and then outputting them to the decoder respectively.