Multimodal knowledge graph completion method based on compound operation

Abstract

The present application relates to the field of electronic digital processing technology, and specifically, to a multimodal knowledge graph completion method based on compound operations. The method includes obtaining a multimodal knowledge graph, converting the multimodal knowledge graph into a text knowledge graph including plain text quadruplets by generating text descriptions from pictures; learning the first embedding representation of the plain text quadruplets based on a knowledge graph embedding model; learning the second embedding representation of the multimodal quadruplets based on a feedforward neural network model; inputting the current head entity, current time and current relationship into the knowledge graph embedding model and the feedforward neural network model to obtain a predicted tail entity; and completing the multimodal knowledge graph according to the predicted tail entity. The present application performs high-precision feature expression in the multimodal knowledge graph, thereby completing the multimodal knowledge graph.

Description

Technical Field

[0001] The present application relates to the field of electronic digital processing technology, and more specifically, to a multimodal knowledge graph completion method based on composite operations. Background Art

[0002] In the era of big language models, all aspects have begun to explore and mine big models more deeply. As we all know, knowledge graphs, as training sets for big language models, provide rich corpus knowledge for big language models, so that big models have professional data support when processing tasks in many professional fields, so that they can better adapt to changes in fields and backgrounds. With the development of big models, knowledge graphs have become more and more diverse; from a single picture or a single text knowledge graph at the beginning, it has transformed into a multimodal knowledge graph that mixes pictures and texts, and it is even more likely that there will be a multimodal knowledge graph that mixes pictures, texts and voices. The emergence of multimodal knowledge graphs has also given rise to different multimodal tasks, including image classification, visual question answering, multimodal knowledge graph completion and entity alignment, etc.

[0003] For various tasks in the multimodal knowledge graph, the multimodal knowledge graph field has also derived models and methods suitable for its own specific fields, integrating multimodal data for training. This approach maintains a high degree of consistency between text and images to a certain extent; but on the other hand, it also ignores the unique characteristics contained in different data of text and images. These two data expressions have their own advantages and disadvantages. For example, the text expression: "There is a rose seed buried in the flowerpot", or a picture of "a classmate is giving a speech"; from these two examples, it can be imagined that it is difficult for a picture to show that there is a seed buried in the flowerpot, let alone a rose seed. However, a picture of a classmate giving a speech should show more results than a text description, such as the appearance, clothing, hair color, etc. of the speaker, and these features can be further enriched by the information of the image.

[0004] In a multimodal knowledge graph, images and texts have their own advantages and disadvantages. Therefore, in the process of completing the knowledge graph, how to use these features to predict the best missing entities is the main technical problem to be solved in this application. Summary of the invention

[0005] The purpose of this application is to provide a multimodal knowledge graph completion method based on compound operations, so as to perform high-precision feature expression in the multimodal knowledge graph and thus complete the completion of the multimodal knowledge graph.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] This application provides a multimodal knowledge graph completion method based on compound operations, including:

[0008] Obtain a multimodal knowledge graph, wherein the multimodal knowledge graph includes a multimodal quadruple consisting of a head entity, a time, a relationship, and a tail entity, wherein the head entity, the tail entity, and the relationship are images or texts;

[0009] By generating text descriptions from pictures, converting the multimodal knowledge graph into a textual knowledge graph including plain text quadruples;

[0010] Based on the knowledge graph embedding model, learning a first embedding representation of the plain text quadruple;

[0011] Based on a feedforward neural network model, learning a second embedding representation of the multimodal quadruple;

[0012] The current head entity, the current time and the current relationship are input into the knowledge graph embedding model and the feedforward neural network model to obtain the predicted tail entity; and the multimodal knowledge graph is completed according to the predicted tail entity.

[0013] Compared with the prior art, the beneficial effects of this application are:

[0014] This application first extracts a text knowledge graph from a multimodal knowledge graph, thereby learning the text information separately; through the knowledge graph embedding module and the feedforward neural network model, the first embedding representation of the plain text and the second embedding representation of the multimodal are learned respectively. Through the training process of the knowledge graph embedding module and the feedforward neural network model, the unique characteristics of the text information content can be better represented. Later, in the process of tail entity prediction, the text information content and the multimodal information can be fused to better predict the tail entity, thereby achieving the purpose of knowledge graph completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 It is a flowchart of a multimodal knowledge graph completion method based on composite operation provided in an embodiment of the present application;

[0017] Figure 2 It is a flowchart of a multimodal knowledge graph completion method based on composite operation provided in an embodiment of the present application;

[0018] Figure 3 It is a schematic diagram of the prediction result provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0020] The embodiment of the present application provides a multimodal knowledge graph completion method based on compound operations, which is suitable for scenarios of feature learning and tail entity completion of multimodal, temporal knowledge graphs (TKG). Knowledge graphs are usually manually annotated data sets. To a certain extent, knowledge graphs cannot contain all knowledge. In order to make the knowledge graph cover more knowledge, knowledge graph completion uses the existing knowledge in the current knowledge graph as a basis to infer the implicit relationships covered in the knowledge graph. After completion, the knowledge graph can become more complete and can better adapt to downstream tasks such as question-answering and recommendation systems.

[0021] See also Figure 1 and Figure 2 , the method provided in this embodiment includes:

[0022] S110. Obtain a multimodal knowledge graph, wherein the multimodal knowledge graph includes a multimodal quadruple consisting of a head entity, a time, a relationship, and a tail entity, wherein the head entity, the tail entity, and the relationship are images or texts.

[0023] TKG not only contains entities, relations, and attributes, but also adds timestamps or time intervals to indicate changes in entities and relations over time. Its basic unit is a quadruple (s, r, o, t), where s and o represent the head entity and tail entity, respectively, r represents the relationship connecting the head entity and the tail entity, and t represents time. Some of the head entities, tail entities, and relations in the multimodal knowledge graph are represented by text, and some are represented by pictures. Optionally, it can also be represented in other forms such as audio. This embodiment only takes text and pictures as examples for illustration.

[0024] S120, by generating text descriptions from images, converting the multimodal knowledge graph into a text knowledge graph including plain text quadruples. S120 includes the following five steps:

[0025] The first step is to generate a first text description from a quadruple including a picture, and to generate a second text description from a plain text quadruple.

[0026] There are multiple knowledge representation methods such as text and pictures in the multimodal knowledge graph, and it is necessary to filter out the quadruple containing image information in the multimodal knowledge graph. By traversing all quadruple in the multimodal knowledge graph, the quadruple that does not contain image information is stored again, and the quadruple containing image information is used to generate a brief text description through the image classification module in the CLIP large model. Among them, the CLIP (Contrastive Language-Image Pre-training) large model is a multimodal pre-trained neural network that aims to represent images and text in a unified semantic space through cross-modal learning. The first text description is generated after the format of the brief text description and the existing knowledge graph content is unified.

[0027] The process of generating a second text description from a plain text quadruple is a process of information extraction and reorganization, and also a process of converting structured data into a natural language description.

[0028] Step 2: Obtain a target second text description similar to the target first text description by calculating text similarity.

[0029] BERT (Bidirectional Encoder Representations from Transformers) similarity calculation is performed on each first text description and each second text description. If the cosine similarity is greater than a threshold, such as 0.9, it is considered that a second text description similar to the first text description is found. For the convenience of description and distinction, the two similar text descriptions are called target second text descriptions similar to the target first text description.

[0030] Step 3: Entity merge the multimodal quadruple of the target first text description source with the plain text quadruple of the target second text description source.

[0031] If the target first text description of the multimodal quadruple is similar to the target second text description of the plain text quadruple, it means that the two quadruple express the same event and can be merged into one quadruple. For example, the merged quadruple retains the head entity and tail entity of the multimodal quadruple or the plain text quadruple, retains the relationship between the multimodal quadruple and the plain text quadruple, and retains the time of the multimodal quadruple or the plain text quadruple.

[0032] Step 4: Generate a plain text quadruple based on the first text description that is not similar to any second text description.

[0033] If the first text description of a multimodal quadruple is not similar to any second text description, it means that the multimodal quadruple is not similar to any plain text quadruple and expresses a new event. Then, the image in the multimodal quadruple is converted into text to form a plain text quadruple.

[0034] Step 5: Merge the generated plain text quadruple with the plain text quadruple after the entity to form a text knowledge graph.

[0035] By storing the plain text quadruple generated in the third and fourth steps, we get the text knowledge graph.

[0036] S130. Based on the knowledge graph embedding model, learn the first embedding representation of the plain text quadruple.

[0037] Knowledge graph completion technology usually uses a knowledge graph embedding model and a neural network architecture model, and the knowledge graph embedding model is superior to the neural network architecture model in performance. Therefore, this embodiment uses a knowledge graph embedding model, improves the structure of the knowledge graph embedding model, and designs a unique scoring function and loss function. Specifically, S130 includes S131~S133.

[0038] S131. Obtain a vector representation of the head entity.

[0039] Use word embedding technology (such as Word2Vec, GloVe, etc.) or sentence embedding technology (such as BERT, GPT, etc.) to convert the text of the head entity into a vector representation.

[0040] S132. Perform a mixed operation of translation and scaling of time and relationship on the vector representation of the head entity to obtain a vector representation of the predicted tail entity.

[0041] Traditional knowledge graph embedding methods usually use a single operation to represent the embedding method of relations and time, which will lead to certain defects in the model, and the model cannot cover all relational patterns. This embodiment applies a combination of translation and scaling operations in the two dimensions of relations and time to explore an embedding representation method suitable for text quadruple knowledge graphs. The following first briefly introduces the mathematical expression of translation and scaling operations.

[0042] Translation and scaling transformations are basic operations in graphics and are often used in various engineering applications. The translation operation is represented in 2D as follows:

[0043] ;

[0044] in, represents the displacement in the x direction of the 2D space, Indicates the displacement in the y direction of 2D space. The x direction is perpendicular to the y direction.

[0045] The matrix representation of the scaling operation in 2D space is:

[0046] ;

[0047] in, Indicates the scaling factor in the x direction of the 2D space. Indicates the scaling factor in the y direction of 2D space. The x direction is perpendicular to the y direction.

[0048] The embedding of relation r and the embedding of time t are expressed as the combination of scaling and translation in geometric operations. In order to facilitate a comprehensive introduction of the knowledge graph embedding model, it is divided into two different parts: time-specific operations and relation-specific operations. Among them, the time-specific operations illustrate the combined use of time-specific operations and relation-specific operations in the quadruple. The relation-specific operations elaborate on the impact of relation-specific operations on the head entity embedding.

[0049] Time-specific operations. Time is translated and scaled before being injected into the scaling operation of the relationship. It must be emphasized that the order of operations, which first translates time and then scales it, is the best order proven by experimental results. Changing the order of operations will affect the results. For the translation operation of the relationship, in order to avoid integrating time information, it is convenient to capture the characteristics of the relationship that remain unchanged over time.

[0050] The vector representation of the predicted tail entity is obtained according to the following formula:

[0051] ；

[0052] ；

[0053] in, represents the scaling of time, represents the time shift, represents the scaling of the relationship, represents the scaling of the relationship after incorporating time information, represents the translation of the relationship, The representation captures the features of a relationship that remain constant over time, i.e., a translation operation on the relationship.

[0054] In order to capture the temporal information in TKG, we avoid applying the temporal translation / scaling operations directly to the head entity embedding, but instead perform relation-specific operations after the temporal translation / scaling operations. and Perform a temporal and relational composite operation on the vector representation of the head entity. This operation is formally expressed as:

[0055] ;

[0056] is the feature vector of the head entity, is the feature vector for predicting the tail entity.

[0057] S133: construct the first loss function of the knowledge graph embedding model according to the semantic similarity between the predicted tail entity and the real tail entity. S133 includes the following four steps:

[0058] Step 1: Construct a scoring function based on the semantic similarity between the predicted tail entity and the true tail entity.

[0059] The scoring function is composed of the predicted tail entity and the real tail entity The scoring function is determined by the semantic similarity between It is expressed as:

[0060] ;

[0061] Step 2: Based on the scoring function and regularization constraints on relations and entities, the first formula is obtained.

[0062] This embodiment uses negative sampling contrastive learning to calculate the first formula, and uses N3 regularization to regularize the relationship and entity constraints. The first formula can be obtained by the following formula:

[0063]

[0064] in is the weight of N3 regularization, Represents the inverse relationship of r, by transforming the quaternary Reverse to get the quaternion To expand the knowledge graph, Indicates a different head entity from s. Indicates a tail entity different from o. and It represents the use of all entities in the entity set to form a quadruple. is the value range of the entity. i represents the i-th quadruple, k is the number of quadruple, is the feature vector of the head entity of the i-th quadruple, is the feature vector of the tail entity of the i-th quadruple, is the translation operation of the relation in the i-th quadruple, is the scaling operation on the relation after incorporating the temporal information in the i-th quadruple.

[0065] Step 3: Regularize the time constraint to obtain the second equation.

[0066] This embodiment applies time linear regularization to constrain the time embedding representation. This method is mainly used to constrain two time points that are close in time. This regularization method mainly simulates the distance in time by using the time point embedding distance. This constraint allows the model to understand the degree of time distance, so that it can more fully understand the time information and mine the evolutionary pattern of the relationship in time. The regularization method is expressed as:

[0067] ;

[0068] in represents the feature vector at time i+1, represents the feature vector at time i, Represents the total number of timestamps.

[0069] Step 4: Construct the first loss function based on the first and second formulas and regularization weights , which can be expressed as:

[0070] ;

[0071] in, represents the weight of temporal regularization.

[0072] S140: Based on the feedforward neural network model, learn a second embedding representation of the multimodal quadruple. Specific operations are S141 to S143.

[0073] S141, extracting image feature vectors and text feature vectors from the head entity, time and relationship of the multimodal quadruple. Specifically, an image feature extraction network is used to extract image feature vectors from the head entity, time and relationship; and a word vector model is used to extract text feature vectors from the head entity, time and relationship.

[0074] Exemplarily, the model VGG16Net pre-trained on ImageNet is used to obtain the image feature vector. VGG16Net consists of several convolutional layers, each followed by a rectified linear unit ReLU layer and a maximum pooling layer, and there are two fully connected layers before the final classification layer divides the image into 1000 categories. The vector of the last fully connected layer is used as the image feature vector required for this embodiment. In order to extract the text feature vector, the PV-DM model (one of the Doc2Vec models) is used to learn the paragraph vector in an unsupervised manner. PV-DM can learn fixed-length representations from sentences of variable length, that is, learn the semantics of head entities, time, and relations. The PV-DM model is similar to the Word2Vec model, which maps a paragraph to a vector, then calculates a joint vector by averaging or concatenating the paragraph vector and the word vector, and predicts the next word based on the joint vector.

[0075] S142. Input the image feature vector and the text feature vector into a feedforward neural network model to obtain a vector representation of the predicted tail entity.

[0076] The feedforward neural network model is a multimodal autoencoder (such as the structure of the TransAE model). It consists of an input layer, three hidden layers, and an output layer. The image feature vector and text feature vector need to be input into the input layer for encoding.

[0077] The input layer obtains image feature vectors and text feature vectors from two different modalities. The input layer of each modality is followed by a fully connected hidden layer with much fewer units. The input feature vectors are mapped to these hidden layers respectively and then input into the next fully connected layer to obtain the final result, which is the vector representation of the predicted tail entity.

[0078] S143. Construct a second loss function based on the distance between the feature vector operation structure of the head entity, time and relationship and the predicted tail entity.

[0079] The second loss function of the feedforward neural network model can be expressed using the following equation:

[0080]

[0081] Among them, Q is the marginal hyperparameter, Represents the real set of quadruples in the text knowledge graph. It represents non The set of quadruple in the text knowledge graph (for example, quadruple in other knowledge graphs or quadruple after replacing the tail entity in the text knowledge graph). Indicates a different head entity from s. indicates a tail entity different from o, indicates a different relationship from r, represents a time different from t. The feature vector representing the head entity, The feature vector representing the relationship, The feature vector representing time, The feature vector representing the tail entity. The Euclidean distance between the head entity feature vector plus the relationship feature vector and the time feature vector and the tail entity feature vector is used as the scoring function. .

[0082] Based on the above steps, the knowledge graph embedding module and the feedforward neural network model are obtained, and the first loss function and the second loss function are obtained. Then, according to the first loss function and the second loss function, the final loss function is constructed. , see the following formula:

[0083] ;

[0084] in, Represents a hyperparameter.

[0085] By minimizing the final loss function, the knowledge graph embedding model and the feedforward neural network model are trained to obtain the displacement parameters and scaling parameters of time and relationship in the knowledge graph embedding model, and the neuron parameters and weight parameters of each layer in the feedforward neural network model.

[0086] Before training, we must first collect training samples. The complete plain text quadruple in the text knowledge graph is used as the positive sample, and the plain text quadruple after the tail entity is replaced is used as the negative sample, or a negative sample is constructed by ourselves. The knowledge graph embedding module and the feedforward neural network model are trained by using the comparative learning training method of positive and negative samples, and the parameters in the two models are iterated to minimize the final loss function.

[0087] This embodiment combines the first loss function with the second loss function to obtain the final loss function, which is beneficial for paying attention to more information in the text description when predicting text quadruple, thereby ensuring the diversification of training data in the multimodal knowledge graph.

[0088] S150. Input the current head entity, current time and current relationship into the knowledge graph embedding model and the feedforward neural network model to obtain the predicted tail entity; and complete the multimodal knowledge graph according to the predicted tail entity.

[0089] This embodiment uses the knowledge graph embedding model and the feedforward neural network model to jointly predict the tail entity to improve the accuracy of the prediction. The current head entity, current time and current relationship lack the tail entity in the current knowledge graph.

[0090] Specifically, the current head entity, the current time, and the current relationship are input into the knowledge graph embedding model to obtain a first score of multiple predicted tail entities; the current head entity, the current time, and the current relationship are input into a feedforward neural network model to obtain a second score of multiple predicted tail entities; based on the first score and the second score, a predicted tail entity for knowledge graph completion is selected from the multiple predicted tail entities.

[0091] In the process of using the model to complete the text knowledge graph, for non-text data, the original data will be input into the multimodal knowledge graph model for prediction, and the text data converted from non-text data will be input into our first module, and the scores of the two scoring functions will be weighted summed through module one and the multimodal module; while for text data, the weighted sum of the scores of the two modules, the multimodal knowledge graph completion model and the text knowledge graph completion model, will be used as the final result. We take the quadruple with the highest score as our prediction object (excluding the quadruple data used in the training data). Specifically, for a quadruple with a missing tail entity, its tail entity needs to be predicted. All entities in the current knowledge graph are input into the scoring functions of the two modules to obtain the first score and the second score. The first score and the second score are weighted summed to obtain the comprehensive score ZH:

[0092] ;

[0093] in, w Represents weight.

[0094] In this way, the comprehensive score ranking of all predicted tail entities is obtained. Because the quadruple in the training set is known, the knowledge graph in the training set needs to be completed, so the quadruple in the training set should be ignored and new quadruple should be predicted. Therefore, the comprehensive score of the known tail entity in the training set is removed from the comprehensive score ranking, and the highest predicted tail entity is selected from the remaining comprehensive scores as the final tail entity.

[0095] Finally, this application conducted experiments in a plain text knowledge graph and mapped high-dimensional space vectors into two-dimensional coordinates through T-SNE dimensionality reduction. T-SNE (t-distributed Stochastic Neighbor Embedding) is a nonlinear dimensionality reduction algorithm that can compress high-dimensional data into a low-dimensional space (such as two-dimensional or three-dimensional), thereby facilitating data visualization and interpretation.

[0096] See also Figure 3 (a) ~ Figure 3 (c), the horizontal and vertical axes can be understood as the distance between entities embedded in the low-dimensional space. The circles and triangles in the figure represent the feature vectors of the head entity and the tail entity respectively, and the same color means that they come from the same quadruple. Figure 3 (a) shows the initial distribution of the head entity and the tail entity. Figure 3 (b) represents the distribution of the feature vector of the head entity and the feature vector of the tail entity after the relational translation operation. Figure 3 (c) shows the distribution of the feature vector of the head entity and the feature vector of the tail entity after the mixed operation of translation and scaling. Figure 3 From the position of the feature vector in (c), it can be found that in the same quadruple, the head entity is in a close position to the tail entity after the mixing operation, which shows the effectiveness of the method of the present application.

[0097] In addition, the representation of the knowledge graph learned by the model in this application has certain reasoning capabilities and will perform better in multimodal knowledge graph reasoning. In addition to the two modalities of images and text, the model in this application can also support knowledge graphs of other modalities. It only needs to convert other modal knowledge graphs into text knowledge graphs; the multimodal encoder supports the encoding of other modal data.

[0098] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0099] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A multimodal knowledge graph completion method based on composite operation, characterized in that: include: Obtain a multimodal knowledge graph, wherein the multimodal knowledge graph includes a multimodal quadruple consisting of a head entity, a time, a relationship, and a tail entity, wherein the head entity, the tail entity, and the relationship are images or texts; By generating text descriptions from pictures, converting the multimodal knowledge graph into a textual knowledge graph including plain text quadruples; Based on the knowledge graph embedding model, learning a first embedding representation of the plain text quadruple; Based on a feedforward neural network model, learning a second embedding representation of the multimodal quadruple; Inputting the current head entity, the current time and the current relationship into the knowledge graph embedding model and the feedforward neural network model to obtain a predicted tail entity; And completing the multimodal knowledge graph according to the predicted tail entity; Based on the knowledge graph embedding model, the first embedding representation of the plain text quadruple is learned, including: obtaining a vector representation of the head entity; applying a mixed operation of translation and scaling of time and relationship to the vector representation of the head entity, injecting the translation and scaling operations of time into the scaling operation of the relationship to obtain a vector representation of the predicted tail entity; constructing a first loss function of the knowledge graph embedding model according to the semantic similarity between the predicted tail entity and the real tail entity L S : ； ； in, represents the weight of temporal regularization, is the weight of N3 regularization, Represents the inverse relationship of r, by transforming the quaternary Reverse to get the quaternion To expand the knowledge graph, Indicates a different head entity from s. indicates a tail entity different from o, and It represents the use of all entities in the entity set to form a quadruple. is the value range of the entity, i represents the i-th quadruple, k is the number of quadruple, is the feature vector of the head entity of the i-th quadruple, is the feature vector of the tail entity of the i-th quadruple, is the translation operation of the relation in the i-th quadruple, is the scaling operation on the relation after incorporating the time information in the i-th quadruple, represents the feature vector at time i+1, represents the feature vector at time i, Represents the total number of timestamps; Applying a time and relationship translation and scaling hybrid operation on the vector representation of the head entity to obtain a vector representation of the predicted tail entity includes: obtaining a vector representation of the predicted tail entity according to the following formula: ； ； ； in, represents the scaling of time, represents the time shift, represents the scaling of the relationship, represents the scaling of the relationship after incorporating time information, represents the translation of the relationship, It represents the feature that captures the relationship and remains unchanged over time, that is, the translation operation of the relationship. is the vector representation of the head entity, is the vector representation of the predicted tail entity.

2. The method according to claim 1, characterized in that By generating text descriptions from images, the multimodal knowledge graph is converted into a text knowledge graph including plain text quadruples, including: Generate a first text description from the quadruple including the picture, and generate a second text description from the first plain text quadruple; By calculating the text similarity, a second target text description similar to the first target text description is obtained; Entity merging the multimodal quadruple of the target first text description source and the plain text quadruple of the target second text description source; Generate a second plain text quadruple according to the first text description that is not similar to any second text description; convert the image in the multimodal quadruple into text to form a second plain text quadruple; The second plain text quadruple generated and the plain text quadruple obtained by merging the entity form a text knowledge graph.

3. The method according to claim 1, characterized in that According to the semantic similarity between the predicted tail entity and the actual tail entity, a first loss function of the knowledge graph embedding model is constructed, including: Construct a scoring function based on the semantic similarity between the predicted tail entity and the true tail entity : ； in, is the feature vector for predicting the tail entity, is the feature vector of the real tail entity; According to the scoring function and regularization constraints on relations and entities, the first formula is obtained ; Regularizing the time constraint, we get the second formula ; According to the first formula, the second formula and the regularization weight , construct the first loss function .

4. The method according to claim 1, characterized in that: Based on a feedforward neural network model, learning a second embedding representation of the multimodal quadruple includes: Extract image feature vectors and text feature vectors from the head entity, time, and relation of the multimodal quadruple; Inputting the image feature vector and the text feature vector into a feedforward neural network model to obtain a vector representation of the predicted tail entity; The second loss function L is constructed based on the distance between the head entity, time and relationship feature vector operation structure and the predicted tail entity. e : Among them, Q is the marginal hyperparameter, Represents the real set of quadruples in the text knowledge graph. It represents non In the set of four tuples, the non The set of quadruple in the middle is the quadruple in other knowledge graphs or the quadruple after replacing the tail entity in the text knowledge graph. Indicates a different head entity from s. indicates a tail entity different from o, indicates a different relationship from r, represents a time different from t, The feature vector representing the head entity, The feature vector representing the relationship, The feature vector representing time, The feature vector representing the tail entity is used to calculate the Euclidean distance between the head entity feature vector plus the relationship feature vector and the time feature vector and the tail entity feature vector as the scoring function. .

5. The method according to claim 4, characterized in that Extract image feature vectors and text feature vectors from the head entity, time, and relationship of the multimodal quadruple, including: Using an image feature extraction network to extract a picture feature vector from the head entity, time and relationship; A word vector model is used to extract text feature vectors from the head entity, time and relationship.

6. The method according to claim 4, characterized in that The feedforward neural network model includes an input layer, three hidden layers and an output layer.

7. The method according to claim 4, characterized in that Before inputting the current head entity, the current time and the current relationship into the knowledge graph embedding model and the feedforward neural network model to obtain the predicted tail entity, the following steps are also included: Constructing a final loss function according to the first loss function and the second loss function; The knowledge graph embedding model and the feedforward neural network model are trained by minimizing the final loss function.

8. The method according to claim 1, characterized in that The current head entity, the current time and the current relationship are input into the knowledge graph embedding model and the feedforward neural network model to obtain the predicted tail entity, including: Inputting the current head entity, the current time and the current relationship into the knowledge graph embedding model to obtain first scores of multiple predicted tail entities; Inputting the current head entity, the current time and the current relationship into the feedforward neural network model to obtain a second score of multiple predicted tail entities; According to the first score and the second score, a predicted tail entity for knowledge graph completion is selected from a plurality of predicted tail entities.

Images