An optimal transmission-based knowledge graph processing method

Abstract

The application provides a knowledge graph processing method based on optimal transmission, which comprises the following steps: S1, obtaining multi-modal information vectors and structure information vectors corresponding to all entities in a knowledge graph; S2, calculating an optimal transmission matrix between multi-modal information and structure information of each entity based on the multi-modal information vectors and the structure information vectors of all entities; S3, taking a space where a structure information vector of each entity is located as a reference space, and performing feature alignment on the multi-modal information of each entity based on the optimal transmission matrix to form a unified feature representation; and S4, predicting and completing a triple of each missing head entity or tail entity based on the unified feature representation of all entities obtained in step S3, wherein all entities are used as candidate entities, a score of each candidate entity as a predicted entity is calculated according to a preset rule, and the predicted entity with the highest score is finally supplemented into the triple as an entity.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of knowledge graph, in particular to the field of knowledge graph embedding containing multiple modal information, more particularly to a knowledge graph processing method based on optimal transmission. BACKGROUND

[0002] Knowledge graph is a structured semantic knowledge base, which is used to describe the concepts and their mutual relations in the real world in symbolic form. Through the knowledge graph, the information, data and link relations obtained from the real world can be aggregated as knowledge, so that the information resources are more easily calculated, understood and evaluated, and the knowledge can be quickly responded and reasoned.

[0003] In the past few decades, with the emergence and development of a large number of knowledge graphs, a series of downstream tasks related to knowledge graphs, such as knowledge question answering, semantic search and recommendation system, have been widely and successfully applied. However, with the continuous growth of data, the existing knowledge graphs usually have the problem of missing links, which limits the application of knowledge graphs in related downstream tasks (knowledge question answering, semantic search, etc.). In order to solve this problem, researchers propose the task of knowledge graph completion, which aims to infer new facts from the existing facts in the knowledge graph, so as to make the knowledge graph more complete. Further, in order to achieve the task of knowledge graph completion, researchers propose to use knowledge graph embedding to predict the missing links in the knowledge graph. Knowledge graph embedding is to embed entities and relations in the knowledge graph into a continuous vector space to learn the low-dimensional representation of entities and relations, so as to facilitate calculation while preserving the structural information in the knowledge graph.

[0004] According to the different utilization of modal information, the knowledge graph embedding method can be divided into single-modal embedding method and multi-modal embedding method. The single-modal embedding method in the prior art mainly utilizes the structural information of the knowledge graph, models the relationship as different geometric transformations (translation, rotation) in space to mine the potential relationship between entities, and the representative methods include single-modal embedding method based on translation and single-modal embedding method based on rotation. Although the single-modal embedding method can mine the potential relationship between entities, researchers find that the single-modal embedding method mainly focuses on the single-modal knowledge graph represented by pure symbols or concepts (also known as structural information), and a large amount of multi-modal (text, visual) information related to entities is not utilized. However, researchers find that the multi-modal information related to entities can significantly improve the quality of representation learning in the embedding method, thereby promoting the application of downstream tasks based on the knowledge graph. Based on this, researchers propose a multi-modal information-based knowledge graph embedding method, that is, by embedding the multi-modal information (for example, text, picture) of the entity into different spaces, then performing connection or mean value operation on the multi-modal information vectors to realize multi-modal information fusion, to obtain a unified representation of the multi-modal entity. The multi-modal embedding method in the prior art mainly includes two categories, the first category is a deep learning-based multi-modal embedding method, which utilizes a neural network model to learn the structured information representation and multi-modal information representation of the knowledge graph, and then embeds the learned structured information representation and multi-modal information representation into a unified space to obtain a unified representation of the multi-modal entity. This scheme adopts a neural network model to learn the text information and visual information of the entity, and then embeds the text information and visual information of the entity into a unified space to perform multi-modal knowledge graph embedding to obtain a unified representation of the text information and visual information of the entity. Although the neural network model can learn the structured information representation and multi-modal information representation, the neural network model therein is designed for a specific medical knowledge graph and is not applicable to other types of knowledge graphs. The second category is a translation-based multi-modal embedding method, which models the relationship as a transformation between multi-modal entities, and then fuses the structured information and multi-modal information through vector connection or mean value operation to obtain a unified representation of the multi-modal entity. This scheme follows the modeling principle of the TransE model, models the relationship as a translation transformation between multi-modal entities, and then integrates the structured information, text information and visual information in a unified space to obtain a unified representation of the multi-modal entity, and takes the sum of the score functions of the text information, visual information and structural information in the knowledge graph as the total score function of the triple.

[0005] Although the multi-modal embedding method can improve the quality of its representation learning, the existing multi-modal embedding method has a problem when fusing modal information: the embedding of different modal information is usually in various heterogeneous spaces, and direct fusion may destroy the internal distribution of different spaces, resulting in poor embedding effect, which is not conducive to the interaction between multi-modal information, and thus is not conducive to improving the effect of knowledge graph embedding. SUMMARY

[0006] Therefore, the purpose of the present application is to overcome the defects of the prior art, and to provide a knowledge graph processing method based on optimal transmission.

[0007] The present application provides a knowledge graph processing method based on optimal transmission, wherein the knowledge graph comprises a plurality of triples, each triple comprising a head entity, a relationship and a tail entity, and the head entity and the tail entity are connected to each other through the relationship, and some triples are missing the head or tail entity, the method comprising the following steps: S1, obtaining the multi-modal information vector and the structure information vector corresponding to all entities in the knowledge graph; S2, calculating the optimal transmission matrix between the multi-modal information and the structure information of each entity based on the multi-modal information vector and the structure information vector of all entities obtained in step S1; S3, taking the space where the structure information vector of each entity is located as the reference space, and performing feature alignment on the multi-modal information of each entity based on its optimal transmission matrix to form a unified feature representation; S4, predicting and completing each triple missing the head entity or the tail entity based on the unified feature representation of all entities obtained in step S3, wherein all entities are used as candidate entities, the score of each candidate entity as a predicted entity is calculated according to a predetermined rule, and the predicted entity with the highest score is finally supplemented into the triple as the entity.

[0008] According to some embodiments of the present application, the multi-modal information comprises text information and picture information.

[0009] According to some embodiments of the present application, in the step S1, different pre-training models are used to process the knowledge graph to obtain the multi-modal information vector and the structure information vector corresponding to all entities.

[0010] According to some embodiments of the present application, the step S1 comprises: S11, using a pre-trained knowledge representation model to process the knowledge graph to obtain the structure information vector of each entity; S12, using a pre-trained text processing model to process the knowledge graph to obtain the text information vector of each entity; S13, using a pre-trained image processing model to process the knowledge graph to obtain the picture information vector of each entity.

[0011] According to some embodiments of the present application, the pre-trained knowledge representation model is a TransE model or a TransH model.

[0012] According to some embodiments of the present invention, the pre-trained text processing model is a word2vec model, a GloVe model or a fastText model.

[0013] According to some embodiments of the present invention, the pre-trained image processing model is a VGG model or a ResNet model.

[0014] According to some embodiments of the present invention, the optimal transfer matrix between the multimodal information and the structural information of each entity includes a first optimal transfer matrix between the entity text information and the entity structural information, and a second optimal transfer matrix between the entity image information and the entity structural information, wherein:

[0015] The first optimal transmission matrix between the entity text information and the entity structure information is calculated in the following manner:

[0016]

[0017] in, A set of first transmission matrices representing the transmission between entity text information and entity structure information, express The first optimal transmission matrix with the minimum transmission cost, Represents the joint probability matrix of entity text information and structural information, The spatial dimension of the text information vector representing the entity, Represents the spatial dimension of the entity's structural information vector, The first vector representing the entity text information The dimension is transferred to the entity structure information vector The probability of dimension, The first vector representing the entity text information The dimension is transferred to the entity structure information vector The cost of dimension.

[0018] The second optimal transmission matrix between the entity image information and the entity structure information is calculated in the following manner:

[0019]

[0020] in, A set of second transmission matrices representing the transmission between entity image information and entity structure information, express The second optimal transmission matrix with the minimum transmission cost is Represents the joint probability matrix of entity image information and structure information, Indicates the spatial dimension of the entity image information vector, The first vector representing the entity image information The dimension is transferred to the entity structure information vector The probability of dimension, The first vector representing the entity image information The dimension is transferred to the entity structure information vector The cost of dimension.

[0021] According to some embodiments of the present invention, the joint probability matrix of entity text information and structure information is calculated in the following manner:

[0022]

[0023] in, Indicates a dimensional matrix and the elements in the matrix are greater than or equal to 0, express The transposed matrix of Indicates the dimension A vector of all 1s, Indicates the dimension A vector of all 1s, Represents the distribution characteristics of entity text information, Represents the distribution characteristics of entity structure information, Represents the probabilistic features of entity text information, Probabilistic features of entity structure information.

[0024] The joint probability matrix of entity image information and structure information is calculated as follows:

[0025]

[0026] in, Indicates a dimensional matrix and the elements in the matrix are greater than or equal to 0, express The transposed matrix of Indicates the dimension A vector of all 1s, Represents the distribution characteristics of entity image information, Probabilistic features representing entity image information.

[0027] According to some embodiments of the present invention, the distribution characteristics of entity structure information are calculated in the following manner:

[0028]

[0029] in, Represents the spatial dimension of the entity's structural information vector, Represents entity structure information The probability of dimension, represents the Dirac function, The first vector representing the entity structure information dimensional embedding, when When it is 0, is 0; when When it is not 0, is 1.

[0030] The distribution characteristics of entity text information are calculated as follows:

[0031]

[0032] in, The dimension of the space where the text information vector representing the entity is located, Represents entity text information The probability of dimension, represents the Dirac function, The first vector representing the entity text information dimensional embedding, when When it is 0, is 0; when When it is not 0, is 1.

[0033] The distribution characteristics of entity image information are calculated in the following way:

[0034]

[0035] in, The spatial dimension of the entity's picture information vector, Indicates entity image information The probability of dimension, represents the Dirac function, The first vector representing the entity image information dimensional embedding, when When it is 0, is 0; when When it is not 0, is 1.

[0036] According to some embodiments of the present invention, in step S3: a first optimal transfer matrix is ​​used to perform feature alignment on the text information vector and the structure information vector corresponding to the same entity:

[0037]

[0038] in, Represents the entity text information vector after feature alignment, represents a diagonal matrix, Probabilistic features that represent entity structure information, represents the first optimal transmission matrix The transposed matrix of Represents the entity text information vector before feature alignment.

[0039] The second optimal transfer matrix is ​​used to align the image information vector and the structure information vector corresponding to the same entity:

[0040] in, Represents the entity image information vector after feature alignment, represents a diagonal matrix, Probabilistic features that represent entity structure information, represents the second optimal transmission matrix The transposed matrix of Represents the entity image information vector before feature alignment.

[0041] The structural information vector of the same entity, the entity text information vector after feature alignment, and the entity image information vector after feature alignment are fused into a unified feature representation:

[0042]

[0043] in, Represents the unified feature representation of the entity, N is 3, Represents the entity multimodal information vector and entity structure information vector after feature alignment, A set of unified feature representations representing the entity to be sought, For any one in the set, Represents the Wassertein distance function.

[0044] According to some embodiments of the present invention, in step S4: for each triplet with a missing tail entity, all entities are used as candidate tail entities, and the distance between each candidate tail entity represented by the unified feature and the transformed entity corresponding to the head entity is calculated as the score as follows:

[0045]

[0046] in, express Head entity as unified feature representation The score of the alternative tail entity, Indicates relationship, Head entity representing unified feature representation The corresponding transformed entity, represents the alternative tail entity of the unified feature representation, represents the distance function.

[0047] For each triplet with a missing head entity, with all entities as candidate head entities, the distance between the transformed entity and the tail entity corresponding to each candidate head entity represented by the unified feature is calculated as follows and used as the score:

[0048]

[0049] in, express Tail Entity as a Unified Feature Representation The score of the candidate head entity, Alternative head entity representing unified feature representation The corresponding transformed entity, represents the tail entity of the unified feature representation, Indicates relationship, represents the distance function.

[0050] According to some embodiments of the present invention, the entity represented by the unified feature is transformed as follows:

[0051]

[0052] in, Entity representing a unified feature representation The corresponding transformed entity, Indicates relationship, represents the entity of the unified feature representation before transformation, Relationship-based For entities transformation.

[0053] According to some embodiments of the present invention, the relationship is a translation relationship, a reflection relationship, or a rotation relationship.

[0054] According to some embodiments of the present invention, the distance function refers to a Euclidean distance function or an inner product distance function.

[0055] Compared with the existing technology, the advantages of the present invention are: taking the space where the entity structure information vector in the knowledge graph is located as the alignment space, first using the optimal transfer matrix to align the multimodal information and the structural information, so that the multimodal information and the structural information obey the same embedding distribution, and then overcoming the spatial heterogeneity through the Wasserstein distance to obtain a unified feature representation after the fusion of multimodal information and structural information, and then taking into account the distribution characteristics of different modal information, so as to better perform the knowledge graph completion task. BRIEF DESCRIPTION OF DRAWINGS

[0056] The embodiments of the present application will be further described with reference to the drawings, wherein:

[0057] Figure 1 A flowchart of a knowledge graph processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0059] As mentioned in the background, the multimodal embedding method in the prior art has a problem when fusing modal information: the embedding of different modal information is usually in various heterogeneous spaces, and direct fusion may destroy the inherent distribution of different spaces, resulting in poor embedding effect, which is not conducive to the interaction between multimodal information, and thus is not conducive to improving the effect of knowledge graph embedding.

[0060] To overcome this defect, the present application provides a knowledge graph processing method based on optimal transport, which first acquires a structural information vector and a multimodal information vector of an entity, and takes the space where the structural information vector of the entity is located as an alignment space (reference space), and then uses an optimal transport matrix to perform feature alignment on the multimodal information, and at the same time, overcomes the spatial heterogeneity between different distributions by minimizing the Wasserstein distance to obtain a unified feature representation of the entity. In this way, the information interaction between the multimodal information embedding can be promoted, and the spatial heterogeneity of the distribution of different modal information is overcome, so that the unified feature representation of different modal information can be better learned.

[0061] It should be noted that the knowledge graph includes a plurality of triples, each of the triples includes a head entity, a relation and a tail entity, and the head entity and the tail entity are connected to each other through the relation, and part of the triples lack the head entity or the tail entity.

[0062] In summary, as Figure 1As shown, the knowledge graph processing method based on optimal transmission provided by the present application comprises: S1, obtaining the multi-modal information vector and the structure information vector corresponding to all entities in the knowledge graph; S2, calculating the optimal transmission matrix between the multi-modal information and the structure information of each entity based on the multi-modal information vector and the structure information vector of all entities obtained in step S1; S3, taking the space where the structure information vector of each entity is located as the reference space, and performing feature alignment on the multi-modal information of each entity based on its optimal transmission matrix to form a unified feature representation; S4, predicting and completing the triplets of each missing head entity or tail entity based on the unified feature representation of all entities obtained in step S3, wherein all entities are used as candidate entities, the score of each candidate entity as a predicted entity is calculated according to a preset rule, and the predicted entity with the highest score is finally supplemented into the triplet as the entity.

[0063] In order to better understand the present application, each step is described in detail below.

[0064] In step S1, the multi-modal information vector and the structure information vector corresponding to all entities in the knowledge graph are obtained. The purpose of obtaining the multi-modal information vector and the structure information vector corresponding to the entity is to improve the calculation efficiency when applying. The reason why the calculation efficiency can be improved is that after the multi-modal information and the structure information of the entity are converted into vector representation, the relationship between entities can be described through simple mathematical calculation. Among them, the multi-modal information includes text information, picture information, audio information, etc. For the sake of description, the multi-modal information in the present application is taken as an example to be described in the form of text information and picture information, and other multi-modal information is similar to them and will not be described here. According to an embodiment of the present application, different pre-trained models are used to process the knowledge graph to obtain the multi-modal information vector and the structure information vector corresponding to all entities. For example, for the structure information of the entity, a pre-selected pre-trained knowledge representation model can be used to process the knowledge graph to obtain the structure information vector of each entity. The knowledge representation model can use TransE model or TransH model, etc. For the text information of the entity, a pre-trained text processing model can be used to process the knowledge graph to obtain the text information vector of each entity. The text processing model can use word2vec, GloVe, or fastText model. For the picture information of the entity, a pre-trained image processing model can be used to process the knowledge graph to obtain the picture information vector of each entity. The image processing model can use VGG model or ResNet model.

[0065] In step S2, the optimal transmission matrix between the multimodal information and the structural information of each entity is calculated based on the multimodal information vectors and the structural information vectors of all entities obtained in step S1. The purpose of calculating the optimal transmission matrix is ​​to find a transmission matrix that minimizes the transmission cost of transmitting the entity text information and the entity image information to the space where the entity structural information is located. According to one embodiment of the present invention, the optimal transmission matrix between the multimodal information and the structural information of each entity includes a first optimal transmission matrix between the entity text information and the entity structural information, and a second optimal transmission matrix between the entity image information and the entity structural information.

[0066] The first optimal transmission matrix between the entity text information and the entity structure information is calculated in the following manner:

[0067]

[0068] in, A set of first transmission matrices representing the transmission between entity text information and entity structure information, express The first optimal transmission matrix with the minimum transmission cost, Represents the joint probability matrix of entity text information and structural information, The spatial dimension of the text information vector representing the entity, Represents the spatial dimension of the entity's structural information vector, The first vector representing the entity text information The dimension is transferred to the entity structure information vector The probability of dimension, The first vector representing the entity text information The dimension is transferred to the entity structure information vector According to one embodiment of the present invention, the joint probability matrix of entity text information and structure information is calculated in the following manner:

[0069]

[0070] in, Indicates a dimensional matrix and the elements in the matrix are greater than or equal to 0, express The transposed matrix of Indicates the dimension A vector of all 1s, Indicates the dimension A vector of all 1s, Represents the distribution characteristics of entity text information, Represents the distribution characteristics of entity structure information, probability feature of entity structure information, According to one embodiment of the present application, the distribution feature of entity structure information is calculated in the following way:

[0071]

[0072] wherein, denotes the dimension of the space where the structure information vector of the entity is located, denotes the probability of the entity structure information in the i-th dimension, denotes the Dirac function, denotes the embedding of the i-th dimension of the structure information vector of the entity, when is 0, is 0; when is not 0, is 1. According to one embodiment of the present application, the probability feature of entity structure information can be set as , denotes the probability simplex with dimension denotes is a dimensional vector, at this time the probability feature of structure information includes , and the probability of the entity structure information in the i-th dimension . According to one embodiment of the present application, the distribution feature of entity text information is calculated in the following way: wherein, denotes the dimension of the space where the text information vector of the entity is located, denotes the probability of the entity text information in the i-th dimension,

[0073] denotes the Dirac function, denotes the embedding of the i-th dimension of the text information vector of the entity, when is 0,

[0074] is 0; when is not 0, is 1. According to one embodiment of the present application, the probability feature of entity text information can be set as , denotes the probability simplex with dimension denotes is a dimensional vector, at this time the probability feature of text information includes , and the probability of the entity text information in the i-th dimension . According to one embodiment of the present application, the distribution feature of entity text information is calculated in the following way: is a ​​​​​dimension of the entity text information vector , including one dimension of the entity structure information dimension of the entity text information vector According to one embodiment of the present application, the dimension of the entity text information vector is transferred to the dimension of the entity structure information in the following way:

[0075]

[0076] wherein, represents the dimension of the entity text information vector represents the dimension of the entity structure information vector According to the content of the foregoing embodiment, the first optimal transmission matrix

[0077] The second optimal transmission matrix between the entity picture information and the entity structure information is calculated in the following way:

[0078]

[0079] wherein, represents a set of the second transmission matrixes between the entity picture information and the entity structure information, represents the second optimal transmission matrix with the minimum transmission cost in the set represents the joint probability matrix of the entity picture information and the structure information, represents the dimension of the entity picture information vector represents the dimension of the entity picture information vector dimension of the entity structure information vector dimension of the entity picture information vector dimension of the entity structure information vector dimension of the entity picture information vector dimension of the entity structure information vector According to one embodiment of the present application, the joint probability matrix of the entity picture information and the structure information is calculated in the following way:

[0080]

[0081] wherein, represents a matrix with one dimension and the elements in the matrix are greater than or equal to 0, represents the transpose matrix of ​​​​​Indicates the dimension A vector of all 1s, Represents the distribution characteristics of entity image information, Represents the probability characteristics of entity image information. According to one embodiment of the present invention, the distribution characteristics of entity image information are calculated in the following manner:

[0082]

[0083] in, The spatial dimension of the entity's picture information vector, Indicates entity image information The probability of dimension, represents the Dirac function, The first vector representing the entity image information dimensional embedding, when When it is 0, is 0; when When it is not 0, According to an example of the present invention, the probability feature of the image information can be set to , Indicates the dimension The probability simplex of express is a dimensional vector, the probability feature of the text information ,include indivual , and the entity structure information dimensional probability According to one embodiment of the present invention, the first entity image information vector is calculated as follows: The dimension is transferred to the entity structure information vector The cost of dimension:

[0084]

[0085] in, The first vector representing the entity image information dimensional embedding, The first vector representing the entity structure information According to the content of the above embodiment, combined with formulas (3), (6)-(9), the second optimal transmission matrix can be obtained .

[0086] By obtaining the optimal transmission matrix between different modal information of an entity and the entity structure information, the original structural features of the multi-modal information of the entity are not destroyed when the multi-modal information in different feature spaces is aligned to the space where the entity structure information is located, so that the unified feature representation of the entity can be better learned.

[0087] In step S3, the multi-modal information of each entity is feature-aligned based on the optimal transmission matrix to form a unified feature representation, taking the space where the structure information vector of each entity is located as the reference space.

[0088] According to an embodiment of the present application, the text information vector and the structure information vector corresponding to the same entity are feature-aligned using the first optimal transmission matrix:

[0089]

[0090] wherein, represents the entity text information vector after feature alignment, represents a diagonal matrix, represents the probability features of the entity structure information, represents the transpose matrix of the first optimal transmission matrix , and represents the entity text information vector before feature alignment.

[0091] The picture information vector and the structure information vector corresponding to the same entity are feature-aligned using the second optimal transmission matrix:

[0092]

[0093] wherein, represents the entity picture information vector after feature alignment, represents a diagonal matrix, represents the probability features of the entity structure information, represents the transpose matrix of the second optimal transmission matrix , and represents the entity picture information vector before feature alignment. The structure information vector, the entity text information vector after feature alignment, and the entity picture information vector after feature alignment of the same entity are fused into a unified feature representation:

[0094]

[0095] wherein, represents the unified feature representation of the entity, and N is 3, represents the entity multi-modal information vector after feature alignment and the entity structure information vector, represents the set of the unified feature representation of the entity to be solved, for any one of the set, denotes the Wassertein distance function. According to the content of the foregoing embodiments, the unified feature representation of the fused entity can be obtained in combination with formulas (1), (6), (10)-(12). It should be noted that the value of N is related to the embedding information of the entity. When the entity has three kinds of embedding information, for example, the embedding information includes text information of the entity, picture information of the entity, and structure information of the entity, at this time, the number of N is 3, and comprises , and , wherein, denotes the aligned entity text information vector, denotes the aligned entity picture information vector, denotes the entity structure information vector; when the entity has four kinds of embedding information, for example, the embedding information includes text information of the entity, picture information of the entity, visual information of the entity, and structure information of the entity, at this time, the number of N is 4, and comprises , , and , wherein, denotes the aligned entity text information vector, denotes the aligned entity picture information vector, denotes the entity visual information vector, denotes the entity structure information vector; the others are the same, and will not be described here. The entity structure information is the most common information, which usually represents the concept of the entity. The concept of some entities may not have multi-modal information, and therefore, the entity multi-modal information is aligned with the entity structure information in the space where the entity structure information is located, so as to maintain the universality of the entity information representation.

[0096] According to an embodiment of the present application, the pseudo code for multi-modal information representation fusion of each entity is shown in Table 1.

[0097] Table 1 Pseudo code for multi-modal information representation fusion of entity

[0098]

[0099] The first row represents the input, which includes the multi-modal information representation of the entity and the embedding distribution thereof , the structure information representation of the entity and the embedding distribution thereof , wherein the multi-modal information representation of the entity In order to use the pre-trained model to process the knowledge graph to obtain the multimodal information vector corresponding to the entity, the embedding distribution of the entity's multimodal information Probabilistic Simplex and Representation of Entity Multimodal Information The specific formulas can be referred to above (4) and (8) and will not be repeated here; the entity structure information representation The pre-trained model is used to process the knowledge graph to obtain the structural information vector corresponding to the entity. The embedding distribution of the entity's structural information uses the probability simplex of the entity's structural information and the representation of the entity's structural information. For details, please refer to the above formula (3) and will not be repeated here.

[0100] The second line represents the output of the pseudocode, which is a multimodal information representation of the entity. and entity structure information representation Unified feature representation after fusion .

[0101] The third line indicates the execution of initialization operations.

[0102] The fourth line represents the probability simplex of the initialized entity structure information. The probability simplex is a mathematical space where each point represents the probability distribution between a finite number of mutually exclusive events. A point on the probability simplex can be represented by W non-negative numbers whose sum is 1. At this time, the probability simplex of the entity structure information represents a dimensional vector, and the entity structure information probability feature on each dimension is .

[0103] The fifth row shows the representation of the probability simplex and entity structure information using entity structure information To solve the embedding distribution of entity structure information, refer to the above formula (3).

[0104] The sixth line indicates setting the number of multimodal information of the entity. When k=1, the number of multimodal information of the entity is 1.

[0105] The seventh line represents the probability simplex of the initial entity multimodal information. At this time, the probability simplex of the entity multimodal information represents a dimensional vector, and the entity multimodal information probability feature in each dimension is .

[0106] The eighth row shows the representation of the probability simplex and entity multimodal information using entity multimodal information To solve the embedding distribution of entity multimodal information, refer to the above formulas (4) and (8).

[0107] The ninth row represents representing the entity multi-modal information with multi-modal information and the entity structure information Solving the cost function between the entity multi-modal and the entity structure embedding, refer to the foregoing formula (5), (9).

[0108] The tenth row represents calculating the optimal transmission matrix between the entity multi-modal information and the entity structure information, refer to the foregoing formula (1), (6).

[0109] The eleventh row represents calculating the entity multi-modal representation after aligning the entity multi-modal information and the entity structure information, refer to the foregoing formula (10), (11).

[0110] The twelfth row represents calculating the entity multi-modal information after aligning the features and the entity structure information The unified feature representation of the fused entity, refer to the foregoing formula (12).

[0111] After representing the entity multi-modal information and the entity structure information as the unified feature representation, each entity has a unique unified feature representation at this time, and the relationship transformation based on this can better capture the potential relationship between entities, thereby better performing the completion task in the knowledge graph and the downstream task related to the knowledge graph.

[0112] In step S4, for each triple of missing tail entity, taking all entities as candidate tail entities, the distance between the transformed entity corresponding to each unified feature representation of candidate tail entity and the head entity is calculated as a score in the following manner:

[0113]

[0114] wherein, represents the score when the candidate tail entity is the unified feature representation of the head entity , represents the relationship, represents the transformed entity corresponding to the unified feature representation of the head entity , represents the unified feature representation of the candidate tail entity, represents the distance function. For each triple of missing head entity, taking all entities as candidate head entities, the distance between the transformed entity corresponding to each unified feature representation of candidate head entity and the tail entity is calculated as a score in the following manner:

[0115] wherein, represents

[0116] the score when the candidate head entity is the unified feature representation of the tail entity , represents the relationship, Tail entity of unified feature representation score of the alternative head entity, Alternative head entity of unified feature representation Corresponding transformed entity, Tail entity of unified feature representation Relationship Distance function. According to an embodiment of the present application, the distance function refers to an Euclidean distance function or an inner product distance function. According to an embodiment of the present application, the entity of unified feature representation is transformed in the following manner:

[0117]

[0118] wherein, Entity of unified feature representation Corresponding transformed entity, Relationship Entity of unified feature representation before transformation Based on relationship Transformation of entity According to an embodiment of the present application, the relationship is a translation relationship, a reflection relationship or a rotation relationship.

[0119] It should be noted that, for the triple with missing tail entity, the shorter the distance between the alternative tail entity and the transformed entity corresponding to the head entity, the higher the score; for the triple with missing head entity, the shorter the distance between the transformed entity corresponding to the alternative head entity and the tail entity, the higher the score. The reason why the aforementioned manner can be used to predict and complete the triple with missing head entity or tail entity is as follows: after the information of the entity is converted into vector representation, the relationship between entities can be obtained through operation, for example, the vector representation of entity A can be translated to obtain the vector representation of entity B, assuming that entity A is the head entity and entity B is the tail entity, at this time, the relationship between the head entity and the tail entity is a translation relationship; for another example, the vector representation of entity C can be rotated by 90° to obtain the vector representation of entity D, assuming that entity C is the head entity and entity D is the tail entity, at this time, the relationship between entity C and entity D is a translation relationship. Based on this, it can be known that the head entity can obtain the tail entity after relationship transformation, or in other words, the head entity can be close to the tail entity after relationship transformation, therefore, when completing the tail entity, the distance between the alternative tail entity and the relationship-transformed head entity can be calculated to determine whether the alternative tail entity can be used as the final tail entity to be filled into the triple, and the same applies to completing the head entity, which will not be described herein again.

[0120] The beneficial effect of the present application is that: taking the space where the entity structure information vector in the knowledge graph is as the alignment space, first aligning the multi-modal information and the structure information by using the optimal transmission matrix, making the multi-modal information and the structure information subject to the same embedding distribution, then overcoming the space heterogeneity by using the Wasserstein distance to obtain the unified feature representation after the fusion of the multi-modal information and the structure information, and then giving consideration to the distribution characteristics of different modal information, so as to better perform the knowledge graph completion task.

[0121] It should be noted that although the above describes each step in a specific order, it does not mean that each step must be performed in the above specific order, in fact, some of these steps can be performed concurrently, or even in a changed order, as long as the required function can be realized.

[0122] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0123] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, and any suitable combination of the foregoing. A non-transitory, computer-readable storage medium, as used herein, is expressly intended not to encompass transitory propagating signals per se.

[0124] The above has described the embodiments of the present application, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements in the technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A knowledge graph processing method based on optimal transmission, wherein the knowledge graph includes multiple triples, each triple includes a head entity, a relationship, and a tail entity, and the head entity and the tail entity are interconnected through a relationship, and some triples lack a head entity or a tail entity, characterized in that: The method comprises the following steps: S1. Obtain the multimodal information vectors and structural information vectors corresponding to all entities in the knowledge graph; S2. Calculate the optimal transfer matrix between the multimodal information and the structural information of each entity based on the multimodal information vectors and the structural information vectors of all entities obtained in step S1, wherein the multimodal information includes text information and image information, and the optimal transfer matrix between the multimodal information and the structural information of each entity includes a first optimal transfer matrix between the text information of the entity and the structural information of the entity, and a second optimal transfer matrix between the image information of the entity and the structural information of the entity, wherein: The first optimal transmission matrix between the entity text information and the entity structure information is calculated in the following manner: in, A set of first transmission matrices representing the transmission between entity text information and entity structure information, express The first optimal transmission matrix with the minimum transmission cost, Represents the joint probability matrix of entity text information and structural information, The spatial dimension of the text information vector representing the entity, Represents the spatial dimension of the entity's structural information vector, The first vector representing the entity text information The dimension is transferred to the entity structure information vector The probability of dimension, The first vector representing the entity text information The dimension is transferred to the entity structure information vector The cost of dimensionality; The second optimal transmission matrix between the entity image information and the entity structure information is calculated in the following manner: in, A set of second transmission matrices representing the transmission between entity image information and entity structure information, express The second optimal transmission matrix with the minimum transmission cost is Represents the joint probability matrix of entity image information and structure information, Indicates the spatial dimension of the entity image information vector, The first vector representing the entity image information The dimension is transferred to the entity structure information vector The probability of dimension, The first vector representing the entity image information The dimension is transferred to the entity structure information vector The cost of dimensionality; S3. Using the space where the structural information vector of each entity is located as the reference space, the multimodal information of each entity is feature-aligned based on its optimal transfer matrix to form a unified feature representation; S4. Based on the unified feature representation of all entities obtained in step S3, predict and complete each triple that is missing a head entity or a tail entity, wherein all entities are taken as candidate entities, and the score of each candidate entity as a predicted entity is calculated according to preset rules, and the predicted entity with the highest score is added to the triple as the final entity.

2. The method according to claim 1, characterized in that In step S1, different pre-trained models are used to process the knowledge graph to obtain multimodal information vectors and structural information vectors corresponding to all entities.

3. The method according to claim 2, characterized in that The step S1 comprises: S11. Processing the knowledge graph using a pre-trained knowledge representation model to obtain a structural information vector for each entity; S12. Processing the knowledge graph using a pre-trained text processing model to obtain a text information vector for each entity; S13. Use a pre-trained image processing model to process the knowledge graph to obtain an image information vector for each entity.

4. The method according to claim 3, characterized in that The pre-trained knowledge representation model is a TransE model or a TransH model.

5. The method according to claim 4, characterized in that The pre-trained text processing model is a word2vec model, a GloVe model or a fastText model.

6. The method according to claim 5, characterized in that The pre-trained image processing model is a VGG model or a ResNet model.

7. The method according to claim 6, characterized in that The joint probability matrix of entity text information and structure information is calculated as follows: in, Indicates a dimensional matrix and the elements in the matrix are greater than or equal to 0, express The transposed matrix of Indicates the dimension A vector of all 1s, Indicates the dimension A vector of all 1s, Represents the distribution characteristics of entity text information, Represents the distribution characteristics of entity structure information, Represents the probabilistic features of entity text information, Probabilistic features of entity structure information; The joint probability matrix of entity image information and structure information is calculated as follows: in, Indicates a dimensional matrix and the elements in the matrix are greater than or equal to 0, express The transposed matrix of Indicates the dimension A vector of all 1s, Represents the distribution characteristics of entity image information, Probabilistic features representing entity image information.

8. The method according to claim 7, characterized in that The distribution characteristics of entity structure information are calculated as follows: in, Represents the spatial dimension of the entity's structural information vector, Represents entity structure information The probability of dimension, represents the Dirac function, The first vector representing the entity structure information dimensional embedding, when When it is 0, is 0; when When it is not 0, is 1; The distribution characteristics of entity text information are calculated as follows: in, The dimension of the space where the text information vector representing the entity is located, Represents entity text information The probability of dimension, represents the Dirac function, The first vector representing the entity text information dimensional embedding, when When it is 0, is 0; when When it is not 0, is 1; The distribution characteristics of entity image information are calculated in the following way: in, The spatial dimension of the entity's picture information vector, Indicates entity image information The probability of dimension, represents the Dirac function, The first vector representing the entity image information dimensional embedding, when When it is 0, is 0; when When it is not 0, is 1.

9. The method according to claim 8, characterized in that In step S3: The first optimal transfer matrix is ​​used to align the text information vector and the structure information vector corresponding to the same entity: in, Represents the entity text information vector after feature alignment, represents a diagonal matrix, Probabilistic features that represent entity structure information, represents the first optimal transmission matrix The transposed matrix of Represents the entity text information vector before feature alignment; The second optimal transfer matrix is ​​used to align the image information vector and the structure information vector corresponding to the same entity: in, Represents the entity image information vector after feature alignment, represents a diagonal matrix, Probabilistic features that represent entity structure information, represents the second optimal transmission matrix The transposed matrix of Represents the entity image information vector before feature alignment; The structural information vector of the same entity, the entity text information vector after feature alignment, and the entity image information vector after feature alignment are fused into a unified feature representation: in, Represents the unified feature representation of the entity, N is 3, Represents the entity multimodal information vector and entity structure information vector after feature alignment, A set of unified feature representations representing the entity to be sought, For any one in the set, Represents the Wassertein distance function.

10. The method according to claim 9, characterized in that In step S4: For each triplet with a missing tail entity, all entities are used as candidate tail entities. The distance between each candidate tail entity represented by the unified feature and the transformed entity corresponding to the head entity is calculated as the score as follows: in, express Head entity as unified feature representation The score of the alternative tail entity, Indicates relationship, Head entity representing unified feature representation The corresponding transformed entity, represents the alternative tail entity of the unified feature representation, represents the distance function; For each triplet with a missing head entity, with all entities as candidate head entities, the distance between the transformed entity and the tail entity corresponding to each candidate head entity represented by the unified feature is calculated as follows and used as the score: in, express Tail Entity as a Unified Feature Representation The score of the candidate head entity, Alternative head entity representing unified feature representation The corresponding transformed entity, represents the tail entity of the unified feature representation, Indicates relationship, represents the distance function.

11. The method according to claim 10, characterized in that The entities represented by the unified feature are transformed as follows: in, Entity representing a unified feature representation The corresponding transformed entity, Indicates relationship, represents the entity of the unified feature representation before transformation, Relationship-based For entities transformation.

12. The method according to claim 11, characterized in that The relationship is a translation relationship, a reflection relationship or a rotation relationship.

13. The method according to claim 12, characterized in that The distance function refers to a Euclidean distance function or an inner product distance function.

14. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any one of the methods of claims 1 to 13.

15. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the method according to any one of claims 1 to 13.

Images