Multimodal knowledge completion method based on optimal transmission and multi-head self-attention network

Abstract

The application belongs to the technical field of knowledge graph embedding, and discloses a multimodal knowledge completion method based on optimal transmission and a multi-head self-attention network, which comprises the following steps: using a pre-training model of a corresponding mode to extract structural, visual image and text embedding of data of different modes; inputting visual image vector representation and text vector representation into a multi-head self-attention network layer to assign weights to different modal information of entities, so as to obtain embedding of auxiliary modal information; taking structural embedding as a benchmark, aligning and keeping consistent other two modes with the structural embedding by minimizing the Wasserstein distance through an optimal transmission strategy; and fusing the aligned multimodal embedding and the structural embedding to obtain a unified representation. The application makes full use of multimodal auxiliary information, effectively solves the modal space heterogeneity, and realizes better prediction in link prediction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of knowledge graph embedding, and particularly relates to a multi-modal knowledge completion method based on optimal transport and multi-head self-attention network. BACKGROUND

[0002] The formation of a rich and complete knowledge graph (KG) has undergone several stages, among which knowledge graph completion is an indispensable stage. Due to the existence of a large amount of real-world knowledge, it is difficult for a KG to cover all of this knowledge; therefore, the incompleteness of a KG is inevitable. In the process of knowledge graph completion, knowledge reasoning is the main method, and link prediction is a typical reasoning task.

[0003] Existing knowledge graph completion pays more attention to the embedding of a single-modal knowledge graph, and traditional completion methods include a translation model, a semantic matching model and a neural network model; however, these methods mostly focus on learning structured triple information, and ignore multi-modal information outside the triples, such as text and pictures, resulting in the fact that the auxiliary information of these entities is not effectively utilized. With the increasing complexity and diversity of application scenarios, single-modal embedding has been difficult to meet the current various needs, and therefore multi-modal knowledge graph completion is a popular research direction. Multi-modal knowledge graph completion methods are roughly divided into two categories: a neural network model and a translation model. The neural network method uses a neural network to learn the embedding of structure and multi-modal knowledge, and the translation method usually measures the rationality of a multi-modal triple by calculating the distance between two entities after translation. However, the performance of such research is limited, because they project all modalities into a unified space with the same relationship to capture commonality, which may not be able to preserve specific information in each modality, and inevitably leaves the problem of heterogeneity. Therefore, they cannot effectively model the complex interactions between modalities to capture complementarity. SUMMARY

[0004] In view of the technical problems existing in the prior art, the application provides a multi-modal knowledge completion method based on optimal transport and a multi-head self-attention network, which can effectively integrate information of different modalities by the multi-head self-attention network for link prediction, and in order to prevent direct fusion from destroying the inherent spatial structure of different modality embeddings, the application uses an optimal transport strategy to move different modality embeddings to a unified space, which not only fully uses the information of auxiliary modalities, but also can ensure that the final result remains consistent and comprehensive, and the performance is obviously improved on a selected data set.

[0005] In order to achieve the above technical purposes, the application is implemented by the following technical scheme:

[0006] The application is a multimodal knowledge completion method based on optimal transmission and multi-head self-attention network, and the multimodal knowledge completion method specifically comprises the following steps:

[0007] Step 1, selecting a data set, using a pre-training model of a corresponding mode to extract structural, visual image and text embedding for data of different modes, outputting structural vector representation, visual image vector representation and text vector representation, and in order to achieve good results, different pre-training models are used for different data sets in the application;

[0008] Step 2, the visual image vector representation and the text vector representation obtained after step 1 are input into a multi-head self-attention network to calculate an attention score, the calculated attention score is used to assign weights to a text multimodal message and a visual image multimodal information, and an auxiliary text multimodal message embedding and an auxiliary visual image multimodal information embedding are obtained;

[0009] Step 3, the structural vector representation extracted in step 1, the auxiliary text multimodal message embedding obtained in step 2 and the auxiliary visual image multimodal information embedding are taken as a reference, the Wasserstein distance is calculated to minimize the Wasserstein distance, the modal alignment process is modeled as a process of transferring modal embeddings of different dimensions to an alignment space, i.e., a structural vector representation embedding space, the auxiliary text multimodal message embedding and the auxiliary visual image multimodal information embedding are aligned to keep consistent with the structural vector representation embedding, and a text alignment embedding and a visual image alignment embedding are formed;

[0010] Step 4, the text alignment embedding, the visual image alignment embedding and the structural vector representation embedding obtained in step 3 are located in the same space, the text alignment embedding, the visual image alignment embedding and the structural vector representation embedding are fused, and a final unified representation is obtained.

[0011] The further improvement of the application is that the data set in step 1 comprises structural information of entity-relation-entity triplets, text multimodal messages and visual image multimodal information.

[0012] The further improvement of the application is that the data set in step 1 is WN9-IMG and FB-IMG, a TransE single-modal KGE method is used to learn structural information embedding of subject entity-relation-object entity, for WN9-IMG, entity text multimodal message embedding is learned by adopting GloVe technology, and entity visual image multimodal information embedding is learned by adopting a VGG19 model, for FB-IMG, entity text multimodal message embedding is learned by adopting word2vec technology, and entity visual image multimodal information embedding is learned by adopting a VGG-m-128 CNN model.

[0013] Further improvement of the present application is that in step 2, the text vector is input to the multi-head self-attention network to calculate the attention score, and the formula is:

[0014]

[0015] The calculated attention score is weighted for the text multi-modal message of the entity, and the formula is:

[0016]

[0017] Wherein, X I is the text multi-modal feature matrix obtained after text multi-modal feature encoding of the entity, is the query matrix of the text multi-modal, is the key matrix of the text multi-modal, is the value matrix of the text multi-modal, V represents the feature dimension, V I is the text multi-modal feature matrix after multi-head self-attention processing;

[0018] The visual image vector representation is input to the multi-head self-attention network to calculate the attention score, and the formula is:

[0019]

[0020] The calculated attention score is weighted for the visual image multi-modal information of the entity, and the formula is:

[0021]

[0022] Wherein, X V is the visual image multi-modal feature matrix obtained after visual image multi-modal feature encoding of the entity, is the query matrix of the visual image multi-modal, is the key matrix of the visual image multi-modal, is the value matrix of the visual image multi-modal, V represents the feature dimension, V V is the visual image multi-modal feature matrix after multi-head self-attention processing.

[0023] Further improvement of the present application is that in step 3, the auxiliary text multi-modal message embedding I is aligned with the structure vector representation embedding S, which specifically comprises the following steps:

[0024] Step 3.1.1, estimate the distribution u I of the auxiliary text multi-modal message embedding E I and the distribution v of the structure vector representation embedding E S ;

[0025] Step 3.1.2, find the I The transmission coupling T to v, where T ij Given the multimodal message embedding E from auxiliary text I The i-th feature dimension is transferred to the structure vector representation embedding E S The probability of the j-th dimension;

[0026] Step 3.1.3. Calculate the center of gravity using the transmission coupling T, and then embed the auxiliary text multimodal message into E I Mapped to the unified space S to form a new text alignment embedding

[0027] A further improvement of the present invention is that in step 3.1.2, the measurement u is calculated I The transmission coupling T between and v is given by:

[0028]

[0029] Among them, L ij It is an assessment and The cost function of the distance between I ,v) is the distribution u I and the Wasserstein distance between the distribution v.

[0030] A further improvement of the present invention is that: in step 3.1.3, after obtaining the transmission coupling T, the auxiliary text multimodal message is embedded in E I Convert to target aligned embedding The formula is as follows:

[0031]

[0032] Among them, Δ θ is an adjustable transmission parameter, T T is the transpose of the transmission coupling T.

[0033] A further improvement of the present invention is that in step 3, aligning the visual image alignment embedding V with the structural vector representation embedding S specifically includes the following steps:

[0034] Step 3.2.1. Estimate the auxiliary visual image multimodal information embedding E V The distribution of u V and the structure vector representation embedding E S The distribution of v;

[0035] Step 3.2.2, find the V The transmission coupling T to v, where T ij Given the multimodal information embedding E from the auxiliary visual image Vthe i-th feature dimension of u is transmitted to E S the probability of the j-th dimension of u;

[0036] Step 3.2.3, calculate the barycenter with the transmission coupling T, and then embed the auxiliary visual image multi-modal information into E I mapped to the unified space S, forming a new visual image alignment embedding

[0037] A further improvement of the present application is that in step 3.2.2, the transmission coupling T between the measurements u V and v is calculated, and the formula is as follows:

[0038]

[0039] where L ij is the cost function for evaluating the distance between and , and W(u V , v) is the Wasserstein distance between the distribution u V and the distribution v.

[0040] A further improvement of the present application is that in step 3.2.3, after obtaining the transmission coupling T, the auxiliary visual image multi-modal information embedding E V is converted into a visual image alignment embedding , and the formula is as follows:

[0041]

[0042] where Δ θ is an adjustable transmission parameter, and T T is the transpose of the transmission coupling T.

[0043] The beneficial effects of the present application are:

[0044] The present application combines multi-head self-attention with optimal transport embedding, not only fully utilizes multi-modal auxiliary information, but also effectively solves the problem of destroying the inherent distribution caused by the fusion of multi-modal information in traditional completion methods.

[0045] The present application is not limited to a certain data set, but is a general embedding method.

[0046] The link prediction aspect in the data sets WN9-IMG and FB-IMG of the present application can achieve better prediction than other baselines. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a process diagram of the multi-modal knowledge completion method of the present application based on optimal transport and multi-head self-attention network. DETAILED DESCRIPTION

[0048] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms. One will recognize, however, that the various embodiments can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures have not been described in order to avoid obscuring the concepts of the various embodiments.

[0049] The application is a multimodal knowledge completion method based on optimal transmission and multi-head attention network. Different modal data are embedded respectively, and then the text embedding and visual embedding are input into the multi-head self-attention network. The features of different modalities can influence each other, and the multimodal information can be fused to obtain multimodal embedding. In order to keep the multimodal embedding consistent with the structure embedding, the spatial heterogeneity is overcome by minimizing the Wasserstein distance between the multimodal embedding. Finally, the multimodal embedding and the structure embedding are fused to achieve unified representation. The specific steps include:

[0050] Step 1, select the data set, for different modal data, use the pre-training model of the corresponding modal to extract the structure, visual image and text embedding, output the structure vector representation, visual image vector representation and text vector representation, and in order to achieve good effect, different pre-training models are used for different data sets.

[0051] The data set in step 1 includes the structure information of the entity-relation-entity triple, the text multimodal message and the visual image multimodal information. The data set in the embodiment of the application is WN9-IMG and FB-IMG. The structure information embedding of the subject entity-relation-object entity is learned by using the TransE single modal KGE method. For WN9-IMG, the text multimodal message embedding of the entity is learned by using GloVe technology, and the visual image multimodal information embedding of the entity is learned by using VGG19 model. For FB-IMG, the text multimodal message embedding of the entity is learned by using word2vec technology, and the visual image multimodal information embedding of the entity is learned by using VGG-m-128CNN model.

[0052] Step 2, the visual image vector representation and the text vector representation obtained after step 1 are input into the multi-head self-attention network to calculate the attention score. The calculated attention score is used as the weight of the text multimodal message and the visual image multimodal information to obtain the auxiliary text multimodal message embedding and the auxiliary visual image multimodal information embedding.

[0053] The text vector is input into the multi-head self-attention network to calculate the attention score, and the formula is:

[0054]

[0055] The calculated attention score is weighted for the text multimodal message of the entity, and the formula is:

[0056]

[0057] Wherein, X I is the text multimodal feature matrix obtained after text multimodal feature encoding of the entity, is the query matrix of the text multimodal, is the key matrix of the text multimodal, is the value matrix of the text multimodal, denotes the feature dimension, V I is the text multimodal feature matrix after multi-head self-attention processing;

[0058] Wherein, the visual image vector represents input to the multi-head self-attention network to calculate the attention score, and the formula is:

[0059]

[0060] The calculated attention score is weighted for the visual image multimodal information of the entity, and the formula is:

[0061]

[0062] Wherein, X V is the visual image multimodal feature matrix obtained after visual image multimodal feature encoding of the entity, is the query matrix of the visual image multimodal, is the key matrix of the visual image multimodal, is the value matrix of the visual image multimodal, denotes the feature dimension, V V is the visual image multimodal feature matrix after multi-head self-attention processing.

[0063] Step 3, the structure vector representation extracted in step 1, the embedding of the auxiliary text multimodal message obtained in step 2, and the auxiliary visual image multimodal information embedding are taken as the benchmark, and the Wasserstein distance is calculated to minimize the Wasserstein distance, and the modal alignment process is modeled as a process of transferring modal embeddings of different dimensions to an alignment space, i.e. structure vector representation embedding space, the auxiliary text multimodal message embedding and the auxiliary visual image multimodal information embedding are aligned with the structure vector representation embedding to keep consistent, forming a text alignment embedding and a visual image alignment embedding.

[0064] The auxiliary text multimodal message embedding I and the structure vector representation embedding S are aligned, which specifically includes the following steps:

[0065] Step 3.1.1, estimating the distribution u I of the auxiliary text multimodal message embedding E I and the distribution v of the structural vector representation embedding E S ;

[0066] Step 3.1.2, finding the transport coupling T from u I to v, where T ij gives the probability of transporting the i-th feature dimension of the auxiliary text multimodal message embedding E I to the j-th dimension of the structural vector representation embedding E S .

[0067] In this step, the transport coupling T between u I and v is computed, which is formulated as follows:

[0068]

[0069] where L ij is a cost function that evaluates the distance between and , and W(u I , v) is the Wasserstein distance between the distribution u I and the distribution v.

[0070] Step 3.1.3, calculating the barycenter with the transport coupling T, and then mapping the auxiliary text multimodal message embedding E I to the unified space S to form a new text-aligned embedding

[0071] In this step, after obtaining the transport coupling T, the auxiliary text multimodal message embedding E I is converted to the target-aligned embedding , which is formulated as follows:

[0072]

[0073] where Δ θ is an adjustable transport parameter, and T T is the transpose of the transport coupling T.

[0074] Aligning the visual image-aligned embedding V with the structural vector representation embedding S specifically includes the following steps:

[0075] Step 3.2.1, estimating the distribution u V of the auxiliary visual image multimodal information embedding E V and the distribution v of the structural vector representation embedding E S ;

[0076] Step 3.2.2, find the V The transmission coupling T to v, where T ij Given the multimodal information embedding E from the auxiliary visual image V The i-th feature dimension is transferred to E S The probability of the j-th dimension of .

[0077] Calculate measurement u V The transmission coupling T between and v is given by:

[0078]

[0079] Among them, L ij It is an assessment and The cost function of the distance between V ,v) is the distribution u V and the Wasserstein distance between the distribution v.

[0080] Step 3.2.3, calculate the center of gravity using transmission coupling T, and then embed the auxiliary visual image multimodal information into E I Mapped to a unified space S, forming a new visual image aligned embedding

[0081] In step 3.2.3, after obtaining the transmission coupling T, the auxiliary visual image multimodal information is embedded in E V Converting to visual image aligned embeddings The formula is as follows:

[0082]

[0083] Among them, Δ θ is an adjustable transmission parameter, T T is the transpose of the transmission coupling T.

[0084] Step 4: After step 3, the text alignment embedding, visual image alignment embedding and structural vector representation embedding are obtained in the same space. The text alignment embedding, visual image alignment embedding and structural vector representation embedding are fused to obtain the final unified representation.

[0085] The formula is as follows:

[0086]

[0087] Among them E A is a unified representation; i Represents weight;

[0088] W(E i ,E) is E iWasserstein distance between the distribution and the E distribution.

[0089] The unified representation of the entity is obtained on the basis of the above process. Subsequently, the application transforms the head entity h in the triple (h, r, t) by means of r

[0090]

[0091] where h is the embedding of the head entity h, r is the embedding of the relation r, h relation is the embedding of the transformed head entity. Here is the transformation imposed by r on h, and the application adopts the TransE method.

[0092] After obtaining the transformed head entity, the scoring function can be calculated, and the model is trained by minimizing the loss.

[0093] The performance indicators of the application are compared with those of several benchmark methods, and the experimental results on the dataset WN9-IMG are shown in Table 1, and the experimental results on the dataset WN9-IMG are shown in Table 2. Table 1

[0094] Model / Indicator MRR Hits@1 Hits@3 Hits@10 TransE 0.865 0.765 0.816 0.871 DistMult 0.901 0.895 0.905 0.925 ComplEx 0.908 0.903 0.907 0.928 IKRL 0.898 0.894 0.908 0.922 TransAE 0.901 0.900 0.912 0.928 The invention 0.912 0.905 0.916 0.935

[0095] Table 2

[0096] Model / Indicator MRR Hits@1 Hits@3 Hits@10 TransE 0.712 0.618 0.781 0.859 DistMult 0.706 0.606 0.742 0.808 ComplEx 0.808 0.757 0.845 0.892 IKRL 0.742 0.691 0.785 0.844 TransAE 0.755 0.698 0.794 0.857 The invention 0.812 0.783 0.859 0.902

[0097] According to the above Tables 1 and 2, the visible performance of the application is improved compared with other models. TransE, DistMult and ComplEx are single-modal completion methods, and it can be seen that the method of the present application embodies the advantage of multi-modal. IKRL and TransAE are multi-modal completion methods, but they simply fuse the multi-modal auxiliary information of the entity and embed it into different spaces, which destroys the inherent distribution. Therefore, the application proves the effectiveness of feature fusion in a unified space.

[0098] The above only describes the embodiments of the application and is not intended to limit the application. The application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the scope of the claims of the application.

Claims

1. A multimodal knowledge completion method based on optimal transmission and multi-head self-attention network, characterized by: The multimodal knowledge completion method specifically includes the following steps: Step 1: Select a dataset and use the pre-trained model of the corresponding modality to extract the structure, visual image and text embeddings of the data of different modalities, and output the structure vector representation, visual image vector representation and text vector representation; Step 2: The visual image vector representation and text vector representation obtained after step 1 are input into the multi-head self-attention network to calculate the attention score. The calculated attention score is assigned a weight to the text multimodal message and the visual image multimodal information to obtain the embedding of the auxiliary text multimodal message and the embedding of the auxiliary visual image multimodal information. Step 3: For the structural vector representation extracted in step 1, the embedding of the auxiliary text multimodal message and the auxiliary visual image multimodal information obtained in step 2, the Wasserstein distance is calculated to minimize the Wasserstein distance, taking the structural vector representation embedding as a benchmark. The modal alignment process is modeled as the process of transferring modal embeddings of different dimensions to the alignment space, i.e., the structural vector representation embedding space. The auxiliary text multimodal message embedding and the auxiliary visual image multimodal information embedding are aligned with the structural vector representation embedding to form text alignment embedding and visual image alignment embedding. Step 4: After step 3, the text alignment embedding, visual image alignment embedding and structural vector representation embedding are located in the same space, and the text alignment embedding, visual image alignment embedding and structural vector representation embedding are fused to obtain the final unified representation; wherein, In step 3, the auxiliary text multimodal message embedding I and the structural vector representation embedding S are aligned, specifically comprising the following steps: Step 3.1.

1. Estimate the auxiliary text multimodal message embedding E I The distribution of u I and the structure vector representation embedding E S The distribution of v; Step 3.1.2, find the I The transmission coupling T to v, where T ij Given the multimodal message embedding E from auxiliary text I The i-th feature dimension is transferred to the structure vector representation embedding E S The probability of the j-th dimension; Step 3.1.

3. Calculate the center of gravity using the transmission coupling T, and then embed the auxiliary text multimodal message into E I Mapped to the unified space S to form a new text alignment embedding In step 3.1.2, calculate the measurement u I The transmission coupling T between and v is given by: Among them, L ij It is an assessment and The cost function of the distance between I ,v) is the distribution u I and the Wasserstein distance between the distribution v; In step 3.1.3, after obtaining the transmission coupling T, the auxiliary text multimodal message embedding E I Convert to target aligned embedding The formula is as follows: Among them, Δ θ is an adjustable transmission parameter, T T is the transpose of the transmission coupling T; In step 3, aligning the visual image alignment embedding V with the structural vector representation embedding S specifically includes the following steps: Step 3.2.

1. Estimate the auxiliary visual image multimodal information embedding E V The distribution of u V and the structure vector representation embedding E S The distribution of v; Step 3.2.2, find the V The transmission coupling T to v, where T ij Given the multimodal information embedding E from the auxiliary visual image V The i-th feature dimension is transferred to E S The probability of the j-th dimension; Step 3.2.3, calculate the center of gravity using transmission coupling T, and then embed the auxiliary visual image multimodal information into E I Mapped to a unified space S, forming a new visual image aligned embedding In step 3.2.2, calculate the measurement u V The transmission coupling T between and v is given by: Among them, L ij It is an assessment and The cost function of the distance between V ,v) is the distribution u V and the Wasserstein distance between the distribution v; In step 3.2.3, after obtaining the transmission coupling T, the auxiliary visual image multimodal information is embedded in E V Converting to visual image aligned embeddings The formula is as follows: Among them, Δ θ is an adjustable transmission parameter, T T is the transpose of the transmission coupling T; In step 2, the text vector is input into the multi-head self-attention network to calculate the attention score. The formula is: The calculated attention score is assigned to the text multimodal message of the entity, and the formula is: Among them, X I It is the text multimodal feature matrix obtained after encoding the text multimodal features of the entity. is the query matrix of text multimodality, is the key matrix of text multimodality, is the value matrix of text multimodality, Represents the feature dimension, V I It is the multimodal feature matrix of the text after multi-head self-attention processing; The visual image vector representation is input into the multi-head self-attention network to calculate the attention score, the formula is: The calculated attention score is weighted as the multimodal information of the entity’s visual image. The formula is: Among them, X V It is the visual image multimodal feature matrix obtained after encoding the visual image multimodal features of the entity. is the query matrix of visual image multimodality, is the multimodal bond matrix of visual images, is the value matrix of the multimodal visual image, Represents the feature dimension, V V It is the multimodal feature matrix of the visual image after multi-head self-attention processing.

2. The multimodal knowledge completion method based on optimal transmission and multi-head self-attention network according to claim 1 is characterized by: The dataset in step 1 includes: structural information of entity-relationship-entity triples, textual multimodal messages, and visual image multimodal information.

3. The multimodal knowledge completion method based on optimal transmission and multi-head self-attention network according to claim 2 is characterized by: The datasets in step 1 are WN9-IMG and FB-IMG. The TransE single-modal KGE method is used to learn the structural information embedding of the subject entity-relationship-object entity. For WN9-IMG, the text multimodal message embedding of the entity is learned by adopting the GloVe technology, and the visual image multimodal information embedding of the entity is learned by the VGG19 model. For FB-IMG, the text multimodal message embedding of the entity is learned by adopting the word2vec technology, and the visual image multimodal information embedding of the entity is learned by the VGG-m-128CNN model.

Images