A multimodal knowledge graph completion method based on neighborhood semantic enhancement

Abstract

A multimodal knowledge graph completion method based on neighborhood semantic enhancement belongs to the field of multimodal knowledge graphs. Specifically, a neighborhood semantic enhancement module is first designed to fuse different neighborhood information into the representation of the central entity to enhance the semantic expression ability of the model. Then, the cross-modal attention mechanism is used to capture the interaction and common semantic features of information between different modalities to achieve the learning of neighborhood-guided features. Then, based on the contrastive learning method, the similarities and differences between positive and negative samples are compared to learn richer entity representations, and finally improve the multimodal knowledge graph completion results.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal knowledge graphs and relates to a multimodal knowledge graph completion method based on neighborhood semantic enhancement. Background Art

[0002] Knowledge graphs represent entities and their relationships in the objective world in the form of graphs, and can be widely used in information retrieval systems, recommendation systems, question-answering systems, etc. Multimodal knowledge graphs are an extension of knowledge graphs. By integrating multimodal data such as images and texts, they can more comprehensively understand and express things in the objective world. However, due to the frequent emergence of new entities and relationships or insufficient accumulation of multimodal corpora, multimodal knowledge graphs are also incomplete. In order to mine missing facts from existing knowledge graphs, multimodal knowledge graph completion can effectively solve this problem.

[0003] The task of multimodal knowledge graph completion mainly focuses on entity prediction (i.e. link prediction). The completion methods mainly follow two paths: embedding-based methods and fine-tuning-based methods. The embedding-based method is developed from the knowledge graph embedding technology and realizes entity embedding by fusing multimodal information. RSME (Relation Sensitive Multi-modal Embedding) is a method based on adaptive modal information selection to realize multimodal entity embedding. MMRNS (MultiModal Relation-enhanced Negative Sampling), KBGAN (Adversarial Learning for Knowledge Graph Embeddings), and NS-KGE (Efficient Non-Sampling Knowledge Graph Embedding) methods use the relationship-enhanced negative sampling strategy to adaptively select high-quality negative samples to alleviate the problem of gradient vanishing during multimodal knowledge graph completion. The fine-tuning-based method can be divided into the discriminator strategy model and the generative model method. The discriminator strategy model regards the knowledge graph completion task as a classification problem and judges the rationality of knowledge based on the positive probability of the model. For example, MKGformer (Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion) and SGMPT (Structure Guided Pre-trained Transformer for Multi-modal Knowledge Graph Reasoning) methods use hybrid transformers to complete multi-level multimodal fusion and combine entity descriptions, relations and images to predict masked entities. KGLLaMA (Knowledge Graph based on LLaMA) and KoPA (Knowledge Prefix Adapter) are generative model methods that convert the knowledge graph completion task into a sequence-sequence task to predict missing entities. Although existing multimodal knowledge graph completion methods can supplement entity representation by fusing interactive information from different modalities, it is difficult to fully explore the semantic correlation between cross-modal neighborhood information.Therefore, there is an urgent need to explore the relationship between neighborhood information to guide the interaction between different modal information and improve the expressiveness of information in multimodal knowledge graphs. Summary of the invention

[0004] To solve the above problems, a multimodal knowledge graph completion method based on neighborhood semantic enhancement (NE-MKGC) is proposed. First, a neighborhood semantic enhancement module based on attention mechanism is designed to fuse the relationship between the central entity and different neighborhoods with the tail entity representation, and enhance the semantic expression ability of the model by fusing with neighborhood semantic information. Then, a cross-modal attention mechanism based on neighborhood guidance is used to capture the potential correlation between different modalities, model the interaction mechanism between text and vision, calculate the cross-modal attention weights of vision and text through neighborhood embedding, and extract the common semantic features of different modalities to realize neighborhood-guided feature learning. Finally, a more discriminative semantic information representation is achieved by using a contrastive learning method, and the similarities and differences between visual text pairs are compared by using self-adversarial technology. The trainable parameters of the contrastive semantic sampler are optimized by Gumbel-Softmax reparameterization technology, and the effectiveness of gradient back propagation in the training process of multimodal knowledge graph completion is guaranteed. The method consists of three parts: semantic enhancement based on knowledge graph neighborhood information, construction of multimodal attention mechanism based on neighborhood guidance, and learning of multimodal neighborhood representation based on contrastive learning. Based on the semantic enhancement of neighborhood information in the knowledge graph, the neighborhood dimensions are aligned, and the multimodal neighborhood semantic relations are captured through neighborhood representation enhancement to help the model better understand the neighborhood relations and entity associations. Multimodal data are modeled based on the neighborhood-guided multimodal attention mechanism to calculate the correlation between multimodal modalities. Multimodal representations are integrated based on contrastive learning to shorten the distance between positive samples and increase the distance between negative samples.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A multimodal knowledge graph completion method based on neighborhood semantic enhancement includes the following steps:

[0007] Step 1: Semantic enhancement based on neighborhood information in the knowledge graph: Use the neighborhood information in the knowledge graph to mine the correlation between entities across modalities, and capture the interaction between different modalities (such as images and text) based on the contextual information and relevant clues involved in the neighborhood information;

[0008] Step 2: Construct a multimodal attention mechanism based on neighborhood guidance, including: modeling the interaction between text features and visual features based on a fully connected network, and guiding the calculation of visual and text cross-modal attention weights based on neighborhood embedding;

[0009] Step 3: Use contrastive learning to compare the similarities and differences between positive and negative samples, and learn neighborhood representation based on cross-modal semantics of contrastive learning; use perturbation Gumbel-Softma technology to ensure effective gradient back propagation.

[0010] Furthermore, step 1 is as follows:

[0011] 1.1 Constructing the neighborhood representation of the head entity:

[0012] Based on the attention mechanism, the neighborhood relationship and the tail entity are used to construct the neighborhood representation of the head entity:

[0013]

[0014]

[0015]

[0016] in, and A central entity Neighborhood relations and tail entity representation; Indicates a serial operation; yes and splicing; Representation and A collection of connected relations; is the attention weight; is the neighborhood representation of the head entity, For the relationship, is the attention weight calculation function, and its detailed calculation formula is as follows:

[0017]

[0018] in , , and is a trainable parameter vector; represents the activation function; is a temperature mechanism hyperparameter to avoid dominance by a single vector; Represents the tail entity, Representing a set of relations the relationship between

[0019] 1.2 Fully connected entity representation fusion mapping:

[0020] The constructed neighborhood representation of the head entity is passed to the fully connected network for nonlinear mapping; the result after nonlinear mapping is added to the central entity to obtain the fused entity representation :

[0021]

[0022]

[0023] in, is the leaky rectified linear unit activation function; and are the weights and bias parameters of the fully connected layer network;

[0024] 1.3 Neighborhood Representation Enhancement:

[0025] The splicing method is used to splice the fused entity representation and relationship representation, and the final neighborhood representation as follows:

[0026]

[0027] Furthermore, the specific steps of step 2 are:

[0028] 2.1 Modeling the interaction between text features and visual features based on a fully connected network:

[0029] The text features and visual features are fed into the fully connected network for nonlinear mapping to unify the encoding dimensions of the text features and visual features. The process is as follows:

[0030]

[0031]

[0032] in, , Respectively represent the trainable weights and biases of text features, , Respectively represent the trainable weights and biases of visual features; Represents text features; Represents visual features; and Represents text features and video features that are unified by encoding dimensions;

[0033] 2.2 Guiding the calculation of visual and textual cross-modal attention weights based on neighborhood embedding:

[0034] By guiding the cross-modal attention weights of vision and text through neighborhood embedding, we can learn the cross-modal data relationship and extract effective features from multimodal data. When cross-modal information interacts, captures and highlights the same semantic features between vision and text, we can obtain the cross-modal matrix in the following way: :

[0035]

[0036] in, Indicates a missing connection; represents the cross-modal matrix;

[0037] Based on the cross-modal matrix , generate the attention weight of text features to visual features and multiply it by Images, obtaining relation-independent visual representations ,Relation-independent visual representation This applies to any relationship:

[0038]

[0039] Neighborhood information is used to guide the normalized attention weights, which are also related to i Multiply by , to obtain a visual representation of the participating neighborhood guide :

[0040]

[0041] Similarly, neighborhood-guided text representation for:

[0042]

[0043] The above It is obtained by inputting the neighborhood embedding into two different fully connected vectors, which are used to guide the bidirectional generation of visual and textual attention respectively. The calculation formula is as follows:

[0044]

[0045]

[0046] in, , Respectively represent the weight and bias of entity feature generation visual attention; , Respectively represent the weight and bias of text attention generated by entity features;

[0047] Relation-independent visual representation and visual representation of the participating neighborhood guide All are normalized at the input layer and added together to obtain a knowledge-guided visual representation :

[0048]

[0049] in, is the normalization function;

[0050] Similarly, knowledge-guided text representation The calculation process is:

[0051]

[0052] in, Representation of text that is relation-independent:

[0053]

[0054] Concatenate the entity and neighborhood information and input them into the fully connected network to obtain the entity embedding itself:

[0055]

[0056] Among them, represents the tail entity, , They represent the weight and bias of entity feature dimension mapping respectively;

[0057] The above mentioned , and There are three entity representations, which are used to replace the sample representation below. , which is further input into the three loss functions below.

[0058] Step 3 The specific steps are:

[0059] 3.1 Cross-modal semantic representation based on contrastive learning:

[0060] Use contrastive learning to compare the similarities and differences between positive and negative samples, learn high-quality neighborhood representation; reduce the distance between positive samples and expand the gap between positive and negative samples; use cosine similarity to measure the similarity between two entities and The similarity between:

[0061]

[0062] in, is a very small number used to ensure that the denominator is not 0;

[0063] Since there is a one-to-many relationship in the knowledge graph, there will be multiple positive samples, so the similarity is used as input to construct the contrast loss function; triplets, loss weight Calculated by the knowledge graph completion model:

[0064] where S is the set of sampled triplets, α is the sampled temperature, Represents the KGC model;

[0065] The final contrast loss function for visual feature similarity is as follows:

[0066]

[0067] Where P is the positive sample set, N is the negative sample set, Represents the visual positive sample loss weight; represents the visual negative sample loss weight, That is , Represents the positive sample representation of vision, Negative sample representation for vision; express and The visual similarity of express and Visual similarity of ; contrast loss function for text feature similarity Contrastive loss function for similarity with entity features Computation and same:

[0068]

[0069]

[0070] Among them, when calculating hour, That is , represents the loss weight of the text positive sample, Represents the text negative sample loss weight; represents the positive sample representation of the text, Represents text negative sample representation; express and The text representation similarity of express and The text representation similarity of

[0071] When calculating hour, That is ; Represents the loss weight of the entity positive sample; represents the entity negative sample loss weight, Represents the positive sample representation of the entity, Negative representations of entities; express and Entity representation similarity, express and Entity representation similarity.

[0072] The contrastive learning loss is the average of the three:

[0073]

[0074] 3.2 Reparameterize the loss function based on Gumbel-Softmax:

[0075] The perturbation Gumbel-Softma technique is used to ensure effective gradient backpropagation and generate continuous distribution to approximate discrete probability distribution. The formula for the sample is as follows:

[0076]

[0077] Among them, the softmax function is used as a differentiable approximation of argmax; Each element g in i are all samples drawn from the standard Gumbel distribution; among them, It is a probability distribution used to increase the semantic similarity difference between positive and negative samples in vision, text, and structure. Convert the similarity into sampling probability, and the calculation process is as follows:

[0078]

[0079] in, Is a factor that balances exploration and utilization; use a non-differentiable mask vector, the value of the negative position of the mask vector is set to 1.0, and the value of the positive position is set to a zero-approaching number; use the mask vector to perform element-by-element multiplication on the probability distribution p. The following is the perturbed Gumbel-Softmax process:

[0080]

[0081] in, is a non-differentiable mask vector with negative values ​​set to 1.0;

[0082] Total loss By adding knowledge graph to complete the model loss and contrastive learning loss What we get:

[0083]

[0084] in, It's balance and The weight parameters of the two losses are, Obtained through the loss function of the KGC model.

[0085] The beneficial effects of the present invention are as follows: the present invention proposes a multimodal knowledge graph completion method based on neighborhood semantic enhancement. Specifically, a neighborhood semantic enhancement module is first designed to fuse different neighborhood information into the representation of the central entity to enhance the semantic expression ability of the model. Then, the cross-modal attention mechanism is used to capture the interaction and common semantic features of information between different modalities to achieve the learning of neighborhood-guided features. Based on the comparison of the similarities and differences between positive and negative samples based on the contrastive learning method, richer entity representations are learned, and finally the multimodal knowledge graph completion results are improved. Through experiments, it is verified that the present invention has excellent performance in the task of knowledge graph completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a framework diagram of the multimodal knowledge graph completion method based on neighborhood semantic enhancement. DETAILED DESCRIPTION

[0087] The embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

[0088] like Figure 1 In this paper, a multimodal knowledge graph completion method with neighborhood semantic enhancement (NE-MKGC) is proposed. First, the neighborhood information in the knowledge graph is used to mine the correlation between entities across modalities, and the interaction between different modalities (such as images and texts) is captured based on the contextual information and relevant clues involved in the neighborhood information; then, a multimodal neighborhood representation is constructed and the similarities and differences between different samples are compared through contrastive learning methods; finally, a more meaningful representation of the model is obtained.

[0089] The present invention is summarized into three steps: In step 1, the present invention proposes a neighborhood semantic enhancement module. The neighborhood semantic information is aggregated into the central entity through the attention mechanism, and the neighborhood information and background information are injected into the central entity to enhance the model's expression ability. In step 2, a multimodal attention mechanism based on neighborhood guidance is introduced. On the basis of neighborhood information, a cross-modal attention mechanism is learned to capture cross-modal interactions, so that multimodal data can be deeply integrated and aligned with common semantics. In step 3, in order to ensure that the multimodal knowledge graph model based on neighborhood guidance learns high-quality neighborhood representations, the present invention introduces a contrastive learning method to increase the representation quality, and uses Gumbel-Softmax to ensure the back propagation of the neighborhood parameter model.

[0090] The specific implementation of each step is as follows:

[0091] Step 1: Semantic enhancement based on knowledge graph neighborhood information;

[0092] The neighborhood information in the knowledge graph is used to mine the cross-modal correlation between entities, and the interaction between different modalities (such as images and text) is captured based on the contextual information and relevant clues involved in the neighborhood information; the details are as follows:

[0093] 1.1 Calculate the neighborhood representation of the head entity:

[0094] Based on the attention mechanism, the neighborhood relationship and tail entity are used to construct the neighborhood representation of the head entity. Specifically, through the fusion of neighborhood relationships, the semantic information of neighborhood relationships and the interaction of central entities are integrated; the semantic expression of central entities is further enriched through the fusion of tail entity representations; the neighborhood representation of the head entity for:

[0095]

[0096]

[0097]

[0098] in, and A central entity Neighborhood relations and tail entity representation; Indicates a serial operation; yes and splicing; Representation and A collection of connected relations; is the attention weight; is the neighborhood representation of the head entity, For the relationship, is the attention weight calculation function, and its detailed calculation formula is as follows:

[0099]

[0100] in , , and is a trainable parameter vector; represents the activation function; is a temperature mechanism hyperparameter to avoid dominance by a single vector; Represents the tail entity, Representing a set of relations the relationship between

[0101] 1.2 Fully connected entity representation fusion mapping:

[0102] In order to unify the dimensions of the neighborhood representation of the head entity, the constructed neighborhood representation of the head entity is passed to the fully connected network for nonlinear mapping; the result after nonlinear mapping is added to the central entity to obtain the fused entity representation :

[0103]

[0104]

[0105] in, is the leaky rectified linear unit activation function; and are the weights and bias parameters of the fully connected layer network;

[0106] 1.3 Neighborhood Representation Enhancement:

[0107] In some cases: even if the entity representation that integrates neighborhood information is achieved, it is still not enough to fully capture the semantic information of the neighborhood. Therefore, in order to further enrich the neighborhood representation, this paper adopts a splicing method to splice the fused entity representation and relationship representation. Splicing entities and relationships can help the model better understand the semantic association of the neighborhood and capture the interaction between entities and relationships. The final neighborhood representation as follows:

[0108]

[0109] Step 2: Construct a multimodal attention mechanism based on neighborhood guidance;

[0110] Based on a fully connected network, the interaction between text features and visual features is modeled, and the cross-modal attention weights of images and texts are calculated based on neighborhood embedding. The neighborhood-guided cross-modal attention module aims to capture the potential correlation between different modalities and emphasize common semantic features. The specific steps are:

[0111] 2.1 Modeling the interaction between text and visual features based on a fully connected network:

[0112] The text features and visual features are fed into the fully connected network for nonlinear mapping to unify the encoding dimensions of the text features and visual features. The process is as follows:

[0113]

[0114]

[0115] in, , Respectively represent the trainable weights and biases of text features, , Respectively represent the trainable weights and biases of visual features; Represents text features; Represents visual features; and Represents text features and video features that are unified by encoding dimensions;

[0116] 2.2 Calculation of cross-modal attention weights of images and texts guided by neighborhood embedding:

[0117] By guiding the cross-modal attention weights of vision and text through neighborhood embedding, we can learn the cross-modal data relationship and extract effective features from multimodal data. When cross-modal information interacts, captures and highlights the same semantic features between vision and text, we can obtain the cross-modal matrix in the following way: :

[0118]

[0119] in, Indicates a missing connection; represents the cross-modal matrix;

[0120] Based on the cross-modal matrix , generate the attention weight of text features to visual features and multiply it by Images, obtaining relation-independent visual representations ,Relation-independent visual representation This applies to any relationship:

[0121]

[0122] At the same time, the present invention uses neighborhood information to guide the normalized attention weights, and the attention weights are also related to i Multiply by , to obtain a visual representation of the participating neighborhood guide :

[0123]

[0124] Similarly, neighborhood-guided text representation for:

[0125]

[0126] The above It is obtained by inputting the neighborhood embedding into two different fully connected vectors, which are used to guide the bidirectional generation of visual and textual attention respectively. The calculation formula is as follows:

[0127]

[0128]

[0129] in, , Respectively represent the weight and bias of entity feature generation visual attention; , Respectively represent the weight and bias of text attention generated by entity features;

[0130] Relation-independent visual representation and visual representation of the participating neighborhood guide All are normalized at the input layer and added together to obtain a knowledge-guided visual representation :

[0131]

[0132] in, is the normalization function;

[0133] Similarly, knowledge-guided text representation The calculation process is:

[0134]

[0135] in, Representation of text that is relation-independent:

[0136]

[0137] Concatenate the entity and neighborhood information and input them into the fully connected network to obtain the entity embedding itself:

[0138]

[0139] Among them, represents the tail entity, , They represent the weight and bias of entity feature dimension mapping respectively;

[0140] The above mentioned , and There are three entity representations, which are used to replace the sample representation below. , which is further input into the three loss functions below.

[0141] Step 3: Multimodal neighborhood representation learning based on contrastive learning;

[0142] Contrastive learning is used to compare the similarities and differences between positive and negative samples to learn high-quality neighborhood representations; the perturbation Gumbel-Softmax technique is used to ensure effective gradient backpropagation.

[0143] 3.1 High-quality cross-modal semantic representation based on contrastive learning:

[0144] Use contrastive learning to compare the similarities and differences between positive and negative samples, learn high-quality neighborhood representation; reduce the distance between positive samples and expand the gap between positive and negative samples; use cosine similarity to measure the similarity between two entities and The similarity between:

[0145]

[0146] in, is a very small number used to ensure that the denominator is not 0;

[0147] Since there is a one-to-many relationship in the knowledge graph, there will be multiple positive samples, so the similarity is used as input to construct the contrast loss function; triplets, loss weight Calculated by the knowledge graph completion model:

[0148] where S is the set of sampled triplets, α is the sampled temperature, Represents the KGC model;

[0149] The final contrast loss function for visual feature similarity is as follows:

[0150]

[0151] Where P is the positive sample set, N is the negative sample set, Represents the visual positive sample loss weight; represents the visual negative sample loss weight, That is , Represents the positive sample representation of vision, Negative sample representation for vision; express and The visual similarity of express and Visual similarity of ; contrast loss function for text feature similarity Contrastive loss function for similarity with entity features Computation and same:

[0152]

[0153]

[0154] Among them, when calculating hour, That is , represents the loss weight of the text positive sample, Represents the text negative sample loss weight; represents the positive sample representation of the text, Represents text negative sample representation; express and The text representation similarity of express and The text representation similarity of

[0155] When calculating hour, That is ; Represents the loss weight of the entity positive sample; represents the entity negative sample loss weight, Represents the positive sample representation of the entity, Negative representations of entities; express and Entity representation similarity, express and Entity representation similarity.

[0156] The contrastive learning loss is the average of the three:

[0157]

[0158] 3.2 Reparameterize the loss function based on Gumbel-Softmax:

[0159] The perturbation Gumbel-Softma technique is used to ensure effective gradient backpropagation and generate continuous distribution to approximate discrete probability distribution. The formula for the sample is as follows:

[0160]

[0161] Among them, the softmax function is used as a differentiable approximation of argmax; Each element g in i are all samples drawn from the standard Gumbel distribution; among them, It is a probability distribution used to increase the semantic similarity difference between positive and negative samples in vision, text, and structure. Convert the similarity into sampling probability, and the calculation process is as follows:

[0162]

[0163] in, Is a factor that balances exploration and utilization; use a non-differentiable mask vector, the value of the negative position of the mask vector is set to 1.0, and the value of the positive position is set to a zero-approaching number; use the mask vector to perform element-by-element multiplication on the probability distribution p. The following is the perturbed Gumbel-Softmax process:

[0164]

[0165] in, is a non-differentiable mask vector with negative values ​​set to 1.0;

[0166] Total loss By adding knowledge graph to complete the model loss and contrastive learning loss What we get:

[0167]

[0168] in, It's balance and The weight parameters of the two losses are, Obtained through the loss function of the KGC model.

[0169] Verification results:

[0170] The present invention adopts multiple groups of experiments to verify the effectiveness of the proposed method.

[0171] Dataset introduction:

[0172] To verify the effectiveness of the invented algorithm, extensive experiments were conducted on the following three benchmark datasets: MKG-W, MKG-Y, and MMKB-DB15K. In order to effectively evaluate the performance and generalization ability of the model, each dataset is divided into a training set, a valid set, and a test set. The statistical information of the datasets used is shown in Table 1, and the remaining details are as follows:

[0173] (1) MKG-W: The full name is Multimodal KG-Wikipedia, which is structured knowledge extracted from Wikipedia and further expanded by a web search engine. The text description information is obtained from DBpedia and aligned with the corresponding entity through an additional sameAs connection. The dataset is randomly divided into training, validation, and test sets in a ratio of 8:1:1. This division method ensures the adequacy and representativeness of the dataset.

[0174] (2) MKG-Y: The full name is Multimodal KG-YAGO, which is the structured knowledge extracted from YAGO. MKG-Y and MKG-W use the same extraction steps to ensure that the datasets have consistent data structure and quality, allowing them to be effectively compared and evaluated in experiments.

[0175] (3) MMKB-DB15K: structured knowledge extracted from DBpedia, image information is crawled through image search engines, and text information of entities is supplemented from the DBpedia database.

[0176] Table 1 Dataset statistics

[0177]

[0178] Baseline model introduction:

[0179] The experiments are conducted on several state-of-the-art knowledge graph completion models, including TransE, DistMult, and RotatE. To evaluate the performance of the NSE-MKGC framework, we compare our method with the following methods, including KBGAN, NS-KGE, and MMRNS. The detailed description of each method is as follows:

[0180] (1) KBGAN: The key idea is to use the triplets (relationships between entities) in the knowledge graph as prior knowledge for generative adversarial networks. In KBGAN, the game process between the generator and the discriminator is extended to a triple-level game, where the goal of the generator is to generate data samples that match the given triples, and the discriminator aims to distinguish between real triplets and generated triplets.

[0181] (2) NS-KGE: A knowledge graph embedding method based on negative sampling, which learns low-dimensional vector representations of entities and relations by maximizing the similarity of positive samples and minimizing the similarity of negative samples. It provides an effective way for knowledge graph analysis and application and can be used for a variety of knowledge graph-related tasks.

[0182] (3) MMRNS: Using a novel knowledge-guided attention mechanism, enhanced by contrastive semantic samplers, cross-modal semantic learning is performed under the guidance of complex knowledge graph relationships. The above baseline methods will be used in link prediction experiments.

[0183] At the same time, in order to verify the effectiveness of the contrastive learning sampling method in this paper, we will compare it with the following methods, including Uniform, Bernoulli, NSCaching and SANS. The detailed description is as follows:

[0184] (1) Uniform: Sampling based on uniform distribution. By sampling from a uniform distribution, we can obtain random variable values ​​with equal probability, thereby achieving uniform sampling of data.

[0185] (2) Bernoulli: A Bernoulli distribution is defined to replace entities, whose parameters depend on the mapping properties of the relationship.

[0186] (3) NSCaching: Use additional memory to cache negative samples with larger scores and samples calculated by weight.

[0187] (4) SANS: This method takes into account the structure and semantic association of the knowledge graph. Based on the association between entities and relations in the knowledge graph, negative examples that are more meaningful than positive example triplets are selected for sampling. This method can better maintain the consistency of the knowledge graph and improve the performance of the representation learning model. The above baseline methods will be used in the sampling strategy comparison experiment.

[0188] Experimental results:

[0189] Table 2 Performance of link prediction task evaluated on MKG-W dataset

[0190]

[0191] Table 3 Performance of link prediction task evaluated on MKG-Y dataset

[0192]

[0193] Table 4. Performance of link prediction task evaluated on MMKB-DB15K dataset.

[0194]

[0195] It can be seen from the table that: (1) the method of the present invention can achieve better results under different data conditions, which shows that it is very effective in extracting implicit category information from unlabeled data; (2) the method proposed in the present invention is superior to the comparative method in experimental results under all fault levels, which proves the effectiveness of the present invention.

Claims

1. A multimodal knowledge graph completion method based on neighborhood semantic enhancement, characterized in that: The following steps are involved: Step 1: Semantic enhancement based on neighborhood information in the knowledge graph: Use the neighborhood information in the knowledge graph to mine the correlation between entities across modalities, and capture the interaction between different modalities based on the contextual information and relevant clues involved in the neighborhood information; 1.1 Constructing the neighborhood representation of the head entity: Based on the attention mechanism, the neighborhood relationship and the tail entity are used to construct the neighborhood representation of the head entity: n rt =[r n ||t n ] (1) a rt =Attention(n rt ,R h ) (2) Among them, r n and t n is a central entity h n Neighborhood relations and tail entity representation; [r n ||t n ] indicates r n and t n Series operation; n rt Yes n and t n splicing; R h Indicates that n A set of connected relations; α rt is the attention weight; is the neighborhood representation of the head entity, r is the relationship representation, Attention(·) is the attention weight calculation function, and its detailed calculation formula is as follows: Among them, u, Wn rt 、Wn r′t and b are trainable parameter vectors; σ(·) represents the activation function; τ is a temperature mechanism hyperparameter to avoid dominance by a single vector; t represents the tail entity, and r′ represents the relation set R h the relationship between 1.2 Fully connected entity representation fusion mapping: The constructed neighborhood representation of the head entity is passed to the fully connected network for nonlinear mapping; the result after nonlinear mapping is added to the central entity to obtain the fused entity representation Where LeakyRELU(·) is the leaky rectified linear unit activation function; W n and b n are the weights and bias parameters of the fully connected layer network; 1.3 Neighborhood Representation Enhancement: The splicing method is used to splice the fused entity representation and relationship representation, and the final neighborhood representation n is as follows: Step 2: Construct a multimodal attention mechanism based on neighborhood guidance, including: modeling the interaction between text features and visual features based on a fully connected network, and guiding the calculation of visual and text cross-modal attention weights based on neighborhood embedding; 2.1 Modeling the interaction between text features and visual features based on a fully connected network: The text features and visual features are fed into the fully connected network for nonlinear mapping to unify the encoding dimensions of the text features and visual features. The process is as follows: Among them, W t 、b t Respectively represent the trainable weights and biases of text features, W i 、b i Respectively represent the trainable weights and biases of visual features; e t Represents text features; e i Represents visual features; and Represents text features and video features unified by encoding dimensions; 2.2 Guiding the calculation of visual and textual cross-modal attention weights based on neighborhood embedding: By guiding the cross-modal attention weights of vision and text through neighborhood embedding, we learn the cross-modal data relationship and extract effective features from multimodal data. When cross-modal information interacts, captures and highlights the same semantic features between vision and text, we obtain the cross-modal matrix M in the following way: Where T represents the missing connection; M represents the cross-modal matrix; Based on the cross-modal matrix M, generate the attention weights of text features to visual features and multiply them by Images, obtaining relation-independent visual representations Relation-independent visual representation This applies to any relationship: Neighborhood information is used to guide the normalized attention weights, which are also related to Multiply by , to obtain a visual representation of the participating neighborhood guide Similarly, neighborhood-guided text representation for: The above and It is obtained by inputting the neighborhood embedding into two different fully connected vectors, which are used to guide the bidirectional generation of visual and textual attention respectively. and The calculation formula is as follows: in, Respectively represent the weight and bias of entity feature generation visual attention; Respectively represent the weight and bias of text attention generated by entity features; Relation-independent visual representation and visual representation of the participating neighborhood guide All are normalized at the input layer and added together to obtain a knowledge-guided visual representation Where, Norm(·) is the normalization function; Similarly, knowledge-guided text representation The calculation process is: in, Representation of text that is relation-independent: Concatenate the entity and neighborhood information and input them into the fully connected network to obtain the entity embedding itself: e s =t·σ(conact(n,t)·W s +b s )(19) Among them, represents the tail entity, W s 、b s They represent the weight and bias of entity feature dimension mapping respectively; The above mentioned and e s They are three entity representations, which are used to replace the sample representation z below and are further input into the three loss functions below. Step 3: Use contrastive learning to compare the similarities and differences between positive and negative samples, and learn neighborhood representations based on cross-modal semantics of contrastive learning. Use the perturbation Gumbel-Softma technique to ensure effective gradient back propagation. 3.1 Cross-modal semantic representation based on contrastive learning: Use contrastive learning to compare the similarities and differences between positive and negative samples, learn high-quality neighborhood representation; reduce the distance between positive samples and expand the gap between positive and negative samples; use cosine similarity to measure the similarity between two entities z u and z v The similarity between: in, δ is a very small number used to ensure that the denominator is not zero; Since there is a one-to-many relationship in the knowledge graph, there will be multiple positive samples, so the similarity is used as input to construct the contrast loss function; for the qth triplet, the loss weight p(h q ,r,t q ) is calculated by the knowledge graph completion model: Where S is the set of sampling triplets, α is the sampling temperature, KGC(h q ,r,t q ) represents the KGC model; The final contrast loss function for visual feature similarity is as follows: Where P is the positive sample set, N is the negative sample set, Represents the visual positive sample loss weight; represents the visual negative sample loss weight, z i That is Represents the positive sample representation of vision, Negative sample representation for vision; Indicates z i and The visual similarity of Indicates z i and Visual similarity of ; contrast loss function for text feature similarity Contrastive loss function for similarity with entity features Computation and same: Among them, when calculating When t That is Represents the loss weight of the text positive sample Represents the text negative sample loss weight; represents the positive sample representation of the text, Represents text negative sample representation; Indicates z t and The text representation similarity of Indicates z t and The text representation similarity of When calculating When s That is e s ; Represents the loss weight of the entity positive sample; represents the entity negative sample loss weight, Represents the positive sample representation of the entity, Negative representations of entities; Indicates z s and Entity representation similarity, Indicates z s and Entity representation similarity; The contrastive learning loss is the average of the three: 3.2 Reparameterize the loss function based on Gumbel-Softmax: The perturbation Gumbel-Softma technique is used to ensure effective gradient back propagation and generate a continuous distribution to approximate the samples in the discrete probability distribution p. The formula is as follows: Here, the softmax function is used as a differentiable approximation of argmax; each element g in g i All samples are drawn from the standard Gumbel distribution; where p is a probability distribution, which is used to increase the semantic similarity difference between positive and negative samples in vision, text, and structure. Softmax is used to convert similarity into sampling probability. The calculation process is as follows: Among them, ξ is a factor that balances exploration and exploitation; a non-differentiable mask vector is used, the value of the negative position of the mask vector is set to 1.0, and the value of the positive position is set to a zero-approaching number; the probability distribution p is multiplied element by element using the mask vector. The following is the perturbed Gumbel-Softmax process: Where mask is a non-differentiable mask vector with the value of negative position set to 1.0; The total loss L is obtained by adding the knowledge graph completion model loss L khc and contrastive learning loss L con What we get: L=L kgc +βL con (29) Where β is the equilibrium L kgc and L con The weight parameter of the two losses, L kgc Obtained through the loss function of the KGC model.

Images