Multi-modal knowledge graph completion method based on modal hierarchical fusion

Abstract

The invention provides a multi-modal knowledge graph completion method based on modal hierarchy fusion, a multi-modal knowledge graph completion model MHF is used to introduce visual features and text features, the model comprises an independent learning layer, a knowledge graph independent learning structure embedding # imgabs0 #, a visual embedding # imgabs1 # and a text embedding # imgabs2 # are embedded from a data set segmented by multiple modals, and the independent learning layer is used to carry out multi-modal hierarchical fusion on the knowledge graph independent learning structure embedding # imgabs0 #, the visual embedding # imgabs1 #, the text embedding # imgabs2 # and the text embedding # imgabs1 # and the text embedding # imgabs2 #; meanwhile, the scale of the external features is adjusted by adopting a scaling factor; the modal fusion layer is used for fusing the structural features with the visual features and the text features respectively to obtain two fusion embedding modes, and the two fusion embedding modes are included in final loss calculation; the semantic constraint layer is used for further processing the five kinds of embedding through a factor interaction regularizer, and five additional regularization loss items generated in the processing process are calculated in final loss; according to the method, the structure data and the external data are integrated, so that the visual data and the text data are more reasonably applied, the expression ability of the model is effectively improved, and a new thought is provided for comprehensiveness and accuracy expression of entities in the knowledge graph.

Description

Technical Field

[0001] The present invention relates to the field of knowledge representation and reasoning technology of knowledge graphs, and in particular to a multimodal knowledge graph completion method based on modal hierarchical fusion. Background Art

[0002] Traditional knowledge graph completion models, including translation-based and neural network methods, face the problem of structural bias between triples. These models are usually modality-specific and optimized for a single modality. Direct application to multimodal scenarios, considering only the structure of the knowledge graph, usually leads to suboptimal performance, mainly due to the lack of a fusion mechanism for additional multimodal information (e.g., text, images, etc.). To address this issue, recent studies have captured multimodal information for link prediction by enriching datasets and proposing new MMKGC models.

[0003] Existing MMKGC methods usually adopt the strategy of sharing relational embeddings, which makes the relations of different modalities tightly coupled. One approach is to fuse the multimodal information of entities into a single embedding, and then learn a unified relational representation based on the entity embedding. Another approach fuses the external modal information of entities (i.e., removing the structural modality) into a single embedding, and then learns a unified relational embedding to represent the relations within and between modalities. However, such studies require all modalities to be projected into a unified relational space and use the commonalities between modalities for prediction. This approach makes it difficult to retain modality-specific knowledge, resulting in the inability to effectively model the complex interactions between modalities to capture complementarity.

[0004] When processing the MMKGC task, it is found that the internal correlation of entity images is usually high, while the connection information with other related entities is relatively small. Existing models either ignore the role of evaluating the visual modality in the reasoning process and directly use the visual modality, which may introduce unnecessary noise; or rely on a separate learning module to process visual information, which does not fully realize its potential value. Summary of the invention

[0005] The present invention proposes a multimodal knowledge graph completion method based on modal hierarchical fusion. By integrating structural data with external data, visual data and text data are more reasonably applied, the expressive ability of the model is effectively improved, and a new idea is provided for the comprehensiveness and accuracy of the representation of entities in the knowledge graph.

[0006] The present invention adopts the following technical solutions.

[0007] A multimodal knowledge graph completion method based on modal hierarchical fusion is used in a question-answering system, a recommendation system, or a knowledge graph for completing or enhancing a multimodal data set. The completion method uses a multimodal knowledge graph completion model MHF based on modal hierarchical fusion to introduce visual features and text features. The multimodal knowledge graph completion model MHF includes:

[0008] Independent learning layer: independently learns structure embedding x from the knowledge graph of datasets segmented from multiple modalities s , visual embedding x v , Text EmbeddingX t , while using a scaling factor to adjust the scale of external features;

[0009] Modality fusion layer: The structural features are fused with the visual features and text features respectively to obtain two fused embeddings, which are included in the final loss calculation;

[0010] Semantic Constraint Layer: The five embeddings mentioned above are further processed by the factor interaction regularizer, and the additional five regularization loss terms generated by the processing process are calculated in the final loss.

[0011] The information fields of the multiple multimodal data sets include geography, people, companies, audiovisual works, and medicines, and the numerical features and image information of the entities contained therein provide data resources for the learning of the multimodal knowledge graph completion model. There are a large number of association relationships between the knowledge graphs of each multimodal data set, in which entities point to the same entity. The multimodal knowledge graph completion model MHF obtains the required prior knowledge information from such association relationships.

[0012] The independent learning layer uses the pre-trained encoder as a fixed feature extractor to obtain modality-specific entity representations and randomly initializes the structural features e s , use random numbers that conform to the normal distribution to initialize the relation embeddings of different modalities;

[0013] The encoder includes an image encoder, which performs preprocessing on the images in the data set, including resizing, converting to tensors, and standardization, to ensure that the image data is normalized to the range of [-1, 1], and then obtains the feature representation f of the image by loading the pre-trained deep learning model. v , and then store the image path and its corresponding feature representation in a dictionary; the feature representation of the image f v After projection, the initial visual embedding of the entity is obtained, which can be expressed as:

[0014] e v ′=W v f v Formula 1;

[0015] Among them, Wv is a transformation matrix specific to visual features. Assume that visual information and textual information are not always related to the factual triples, that is, external information carries uncertainty. In order to alleviate uncertainty, a scaling factor η is used to train visual and textual modalities in a confidence-constrained manner: The formula is:

[0016]

[0017] The image encoder uses the ViT model to randomly extract a picture of an entity for encoding to generate an image embedding of the entity.

[0018] The encoder uses the BERT model to encode the entity text name and text description to capture the text information of the entity, form an independent text embedding, and generate an embedding vector for a given entity description; the text encoder of the encoder reads the description information of the entity from the specified data set file, and then generates the embedding vector of these entity descriptions; if the entity description does not exist or is empty, a randomly initialized embedding vector will be generated for the entity as its representation; the text encoding process adds special tags before and after the text, and then uses the BERT tokenizer to segment the words, and converts the segmentation results into corresponding vocabulary IDs; the text encoder truncates the input of more than 510 tokens to ensure that the total length after including two special tags does not exceed 512 to meet the input length limit of the BERT model, and then sets these vocabulary IDs and paragraph IDs to constants to represent the same sentence, and converts them into tensors and sends them to the pre-trained BERT model for forward propagation to obtain the hidden state of the model; the hidden state of the penultimate layer of the model is used to generate the embedding vector of each token, and the arithmetic average of these hidden state vectors is calculated to generate the text embedding vector representation f t ;

[0019] The text encoder uses the hidden state of the penultimate layer to generate the embedding vector of each word, and generates the text embedding vector representation by calculating the arithmetic mean of these hidden state vectors, providing a dense embedding representation for different entities or text paragraphs. The embedding representation captures the key information and contextual meaning of the text. The text embedding vector f t After projection, the initial text embedding of the entity is obtained, and the formula is as follows:

[0020] e t ′=W t f t Formula 3;

[0021] Where W t is a text feature-specific transformation matrix, which processes the initial text embedding with a scaling factor η, and the formula is:

[0022]

[0023] The confidence of the prediction is limited by the scaling factor operation to soften the output distribution while maintaining the invariance of the prediction results; the multimodal knowledge graph completion model uses independent embedding layers and processing logic for each feature separately to process different feature modalities.

[0024] The multimodal knowledge graph completion model integrates information of different structural modalities, textual modalities, and visual modalities through a modal fusion layer to improve the model's comprehensive understanding and expression capabilities of entities, that is, by fusing the embedding of different modalities, the multimodal knowledge graph completion model can more comprehensively consider diverse information and more accurately represent the characteristics and semantic relationships of entities;

[0025] The core components of the modality fusion layer include the feature fusion network, which obtains two fused embeddings by combining structure embedding and text embedding, as well as structure embedding and image embedding; using x m Represents an embedding x of some modality m =e m ∪r m , x m contains the entity embedding and relation embedding in modality m; when the fusion structure embedding x3 and image embedding x v When , the calculation process of the fusion network is to embed the structure into x3 and the image into x based on the ComplEx model. v Split into real and imaginary parts and fuse them separately; first, define a simple sequential connection module S:

[0026] S(x)=σ(BN(Linear(Dropout(x;p)))) Formula 5;

[0027] In the formula, a random dropout layer with probability p is applied to the input vector x, where p is a hyperparameter and is adjusted according to the model requirements; Linear is a fully connected layer, BN is a batch normalization layer BatchNorm1D, and the activation function σ selects ReLU; the sequential connection module supports multi-layer stacking, and the real or imaginary parts of the structure embedding and the image embedding are fused through element-by-element multiplication after passing through the sequential connection module, and then through the activation function to ensure nonlinearity and positive activation; the process can be repeated l times, and the process of multi-layer module fusion is as follows:

[0028]

[0029] Where Re(·) represents the real part operation, Im(·) represents the imaginary part operation, the subscript re represents the real part of the embedding, the subscript im represents the real part of the embedding, l represents the number of stacked layers of sequentially connected modules, ⊙ represents element-by-element multiplication, and the activation function σ selects ReLU; after concatenating the real and imaginary parts of the fused embedding, the complete fused embedding x is obtained. sv : The formula is:

[0030] x sv =[x sv，re ;xs v，im ] Formula 8;

[0031] The semicolon in the formula represents the concatenation operation;

[0032] If the image embedding and text embedding have been scaled in the independent learning layers before fusion, then the two fusion forms with structure embedding, that is, the embedding methods combined with text embedding and image embedding respectively, do not need additional scaling operations; the two fused embeddings obtained by fusing structure embedding with text embedding and fusing structure embedding with image embedding need to be added to the loss calculation when making predictions.

[0033] In the semantic constraint layer, the factor interaction regularizer utilizes the semantic transfer properties between head entities and tail entities to improve the generalization ability of the multimodal knowledge graph completion model;

[0034] The semantic transfer property between the head entities and the tail entities is specifically: if the semantics of the two head entities in the triple are similar, then the semantics of the two tail entities linked by the same relationship should also be similar; conversely, if the semantics of the two head entities linked by the same relationship are not similar, then the two tail entities are also not similar; let h, r, t be used to represent the head entity, relationship, and tail entity of the triple, and h, r, t represent the corresponding embedding. For a given triple T i =(h i , r i , t i ), and T i Another triple T with the same relation j =(h j , r i , t h ), the loss calculation method of its regularization term is:

[0035] Similarity loss based on the similarity in the semantic transfer properties between the head and tail entities The calculation formula is:

[0036]

[0037] Among them, given T i , T j, It is the L2 norm constraint on the head entity and the tail entity in the triple, hr = h i r i Entity t i Embedding, tr = h j r i Entity t j Embedding of; Computation The similarity between the head and tail entities of each triple It is equivalent to calculating T i , T j The similarity between the two tail entities of ; by inference, we get t i ,t j With similar semantics, Minimize, reduce i ,t j The distance between embeddings, let t i ,t j The embeddings are more likely to be located in similar semantic spaces; the computational complexity of the model is reduced by converting the computation between two triplets into the computation between elements within a specific triplet;

[0038] Based on the difference in the semantic transfer properties between the head and tail entities, the difference loss The calculation formula is:

[0039]

[0040] in, The definition of is the same as above; the semantic space of the entity is bounded to ensure that the semantic space does not diverge; there is also a norm constraint According to the parallelogram law of vectors, the farther the semantic distance between hr and tr is, The smaller it is; when T i =(h i , r i , t i ), T j =(h j , r i , t j ) in the header entity h i 、h j If they are not similar, Minimize and enlarge t i ,t j The distance between embeddings is such that t i ,t j The semantic difference is greater;

[0041] Combining Formula 9 and Formula 10, and considering the similarities and differences in the semantic transfer properties between the head and tail entities, the loss calculation formula of the FI regularizer is as follows

[0042]

[0043] Among them, S is defined as above, α is a hyperparameter used to adjust the weight of the L2 norm regularization term; γ is used to determine whether two entities belong to the same category. When γ = 1, they belong to the same category, and the similarity in the semantic transfer property is used; when γ = 0, they do not belong to the same category, and the difference in the semantic transfer property is used; the cosine is used to calculate the similarity γ of the two triple head entities, and the formula is as follows:

[0044]

[0045] Among them, ∈ j It is a hyperparameter, which is used as the threshold for judging similarity and takes values ​​between [0,1]. i 、h j Taken from Equation 11.

[0046] The multimodal knowledge graph completion model MHF adopts the ComplEx decoder, and its scoring function calculation formula is as follows:

[0047]

[0048] in, Re(·) and Im(·) represent the real and imaginary parts of the complex number, respectively; is the conjugate of the complex vector t; the standard component multilinear dot product<a,b,c> =∑ d a d b d c d , d is the vector dimension;

[0049] When the entity-level triple data is sparse, the multimodal knowledge graph completion model adjusts the modal weights at the relation level to balance the importance of each modality in the final prediction. It first separates the triples into Relation-aware sets Right now The following calculation steps are performed on each In the separate, RankBoost algorithm is applied to model the importance of the modality and generate weight ω for each modality of relation r. m (r), where The final prediction score is obtained by combining the scores of each modality to find the best modality combination; specifically:

[0050] Initialize all modal weights to 0, that is, ω m(r) = 0, The weight of each triple (h, r, e) in the RankBoost algorithm is initialized as ω m (r), D m (e) Updated in the iterative learning of the model; for the query (h, r, ?), the prediction results under each modality are counted to generate the weight ω of each modality of the relation r m (r): If the scores of other candidate tail entities e except the true tail entity are less than the score of the true tail entity t, they are considered to be correctly predicted, indicating function g m (e) is assigned a value of 1, otherwise it is an incorrect prediction, indicating function g m (e) is assigned a value of -1; the formula is as follows:

[0051]

[0052] The result of the statistical indicator function is used to measure the prediction accuracy of the model. The better the ranking prediction result of the triple (h, r, ?) under mode m, the higher the modal weight ω m The larger the value of m Need to go through iterations, the weight of the i-th iteration The calculation formula is as follows:

[0053]

[0054] The value of each iteration is saved, and the maximum value is selected as Used to update the triple weight D m+1 (e) and the modal weight ω of relation r m (r):

[0055]

[0056] ω m (r) = ω m (r)+ω m Formula 17;

[0057] The multimodal knowledge graph completion method uses integrated reasoning to obtain the final prediction, chooses to directly combine scores to retain more original information, and applies different weights and combination strategies to more finely control the integration process. For each query, through independent learning layers, the decoder obtains three scores ψ from the three modalities. s (h, r, t), ψ v (h, r, t), ψ t (h, r, t) to directly reflect the advantages and limitations of the entity prediction modality; the calculation formula for the combined score is as follows:

[0058]

[0059] in, Contains structural modality, visual modality and textual modality, ψ m (h, r, t) represents the score of the triple under mode m, ω m (r) represents the weight of relation r under mode m.

[0060] The multimodal knowledge graph completion model is trained using cross entropy loss; for all triples, the decoder provides the probability of the corresponding triple being true using the ComplEx model, which is obtained by applying softmax to the output of the scoring function, and the modality The probability of obtaining is denoted as y m (t|(h, r), the calculation formula is as follows:

[0061] y m (t|(h, r))=softmax(ψ final (h, r, t)) Formula 19;

[0062] The CE loss calculation formula for mode m is as follows:

[0063]

[0064] The multimodal knowledge graph completion method uses the basic model ComplEx to simultaneously train the three modalities of structural modality, text modality and image modality to learn the intra-modal representations respectively, and obtain three independent modal embeddings; after passing through the modal fusion layer, two fused embeddings are obtained, and then the three single modal embeddings and the two fused embeddings are predicted and processed respectively to calculate different loss terms. The loss calculation of the basic function ComplEx model is as follows:

[0065]

[0066] The advantages of the present invention are:

[0067] 1. This paper proposes an innovative modal hierarchical fusion model (MHF), which can learn external features in stages from easy to difficult, and effectively fuse complementary information of different modalities for link prediction. MHF contains independent learning layers and modal fusion layers, which adjust the confidence of external information by scaling factors and adaptively adjust the importance of each modality during the inference process. In addition, the model adopts a comprehensive loss function, which aims to simultaneously consider the joint role of multimodal representations in prediction.

[0068] 2. The modal fusion layer designed by the present invention utilizes the splitting and fusion of the real and imaginary parts in the complex space, fully capturing the complementarity between different modalities to achieve effective fusion of multimodal information.

[0069] 3. The present invention designs a novel factor interaction regularizer as a semantic constraint layer. Starting from the triple itself, it uses the semantic information implicit in the head-tail entity conversion properties to suppress overfitting and enhance the generalization ability of the model.

[0070] The technical advantages of the present invention are:

[0071] 1. The present invention advocates that the model learns from easy to difficult. The MHF model demonstrates excellent performance in processing multimodal information in the knowledge graph through multi-level learning of independent learning layers and modal fusion layers. In the complex space, the introduction of the feature fusion network effectively improves the expressive power of the model and provides a new idea for the comprehensive and accurate representation of entities in the knowledge graph.

[0072] 2. Existing models either ignore the role of visual modality in the reasoning process and directly use visual modality, which may introduce unnecessary noise; or rely on separate learning modules to process visual information, which does not fully realize its potential value. To address this challenge, this paper proposes a new modality embedding technology that integrates structural data with external data to more reasonably apply visual data and text data.

[0073] 3. Traditional regularizers avoid overfitting by only limiting the embedding size, achieving a balance between low computational cost and model performance. The present invention expands the basic regularizer function and designs a loss function containing multiple regularization terms. While obtaining more semantic constraint information, the computational complexity is well controlled. The semantic transfer property between the head and tail entities is used to adjust the model to enhance adaptability.

[0074] The MHF model of the present invention has multimodal characteristics and has wide application potential in multiple fields, and is particularly suitable for the following scenarios:

[0075] ① Question and answer system:

[0076] Can be used for multimodal question answering: Using multimodal data, we can build more powerful question answering systems that can understand entities and relationships in natural language and use images and numerical information to answer questions.

[0077] Can be used for knowledge graph enhancement: Multimodal data can be used to enhance the knowledge graph in the question-answering system so that it contains richer information, thereby improving the accuracy and practicality of the question-answering system.

[0078] ②Recommendation system:

[0079] Can be used for multimodal recommendations: Using multimodal data, a more personalized recommendation system can be built, such as making recommendations based on user interests, preferences, and entity relationships in the knowledge graph.

[0080] Can be used for knowledge graph-driven recommendations: Through multimodal knowledge graphs, knowledge-based recommendation systems can be built, such as making recommendations based on the associations between entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0082] Attached Figure 1 It is a schematic diagram of the architecture of the multimodal knowledge graph completion model MHF described in the present invention. DETAILED DESCRIPTION

[0083] As shown in the figure, a multimodal knowledge graph completion method based on modal hierarchical fusion is used in a question-answering system, a recommendation system, or a knowledge graph for completing or enhancing a multimodal data set. The completion method uses a multimodal knowledge graph completion model MHF based on modal hierarchical fusion to introduce visual features and text features. The multimodal knowledge graph completion model MHF includes:

[0084] Independent learning layer: independently learn the structure embedding X from the knowledge graph of the dataset segmented by multiple modalities S , visual embedding x v , Text EmbeddingX t , while using a scaling factor to adjust the scale of external features;

[0085] Modality fusion layer: The structural features are fused with the visual features and text features respectively to obtain two fused embeddings, which are included in the final loss calculation;

[0086] Semantic Constraint Layer: The five embeddings mentioned above are further processed by the factor interaction regularizer, and the additional five regularization loss terms generated by the processing are calculated in the final loss.

[0087] The information fields of the multiple multimodal data sets include geography, people, companies, audiovisual works, and medicines, and the numerical features and image information of the entities contained therein provide data resources for the learning of the multimodal knowledge graph completion model. There are a large number of association relationships between the knowledge graphs of each multimodal data set, in which entities point to the same entity. The multimodal knowledge graph completion model MHF obtains the required prior knowledge information from such association relationships.

[0088] The independent learning layer uses the pre-trained encoder as a fixed feature extractor to obtain modality-specific entity representations and randomly initializes the structural features e s, use random numbers that conform to the normal distribution to initialize the relation embeddings of different modalities;

[0089] The encoder includes an image encoder, which performs preprocessing on the images in the data set, including resizing, converting to tensors, and standardization, to ensure that the image data is normalized to the range of [-1, 1], and then obtains the feature representation f of the image by loading the pre-trained deep learning model. v , and then store the image path and its corresponding feature representation in a dictionary; the feature representation of the image f v After projection, the initial visual embedding of the entity is obtained, which can be expressed as:

[0090] e v ′=W v f v Formula 1;

[0091] Among them, W v is a transformation matrix specific to visual features. Assume that visual information and textual information are not always related to the factual triples, that is, external information carries uncertainty. In order to alleviate uncertainty, a scaling factor η is used to train visual and textual modalities in a confidence-constrained manner: The formula is:

[0092]

[0093] The image encoder uses the ViT model to randomly extract a picture of an entity for encoding to generate an image embedding of the entity.

[0094] The encoder uses the BERT model to encode the entity text name and text description to capture the text information of the entity, form an independent text embedding, and generate an embedding vector for a given entity description; the text encoder of the encoder reads the description information of the entity from the specified data set file, and then generates the embedding vector of these entity descriptions; if the entity description does not exist or is empty, a randomly initialized embedding vector will be generated for the entity as its representation; the text encoding process adds special tags before and after the text, and then uses the BERT tokenizer to segment the words, and converts the segmentation results into corresponding vocabulary IDs; the text encoder truncates the input of more than 510 tokens to ensure that the total length after including two special tags does not exceed 512 to meet the input length limit of the BERT model, and then sets these vocabulary IDs and paragraph IDs to constants to represent the same sentence, and converts them into tensors and sends them to the pre-trained BERT model for forward propagation to obtain the hidden state of the model; the hidden state of the penultimate layer of the model is used to generate the embedding vector of each token, and the arithmetic average of these hidden state vectors is calculated to generate the text embedding vector representation f t ;

[0095] The text encoder uses the hidden state of the penultimate layer to generate the embedding vector of each word, and generates the text embedding vector representation by calculating the arithmetic mean of these hidden state vectors, providing a dense embedding representation for different entities or text paragraphs. The embedding representation captures the key information and contextual meaning of the text. The text embedding vector f t After projection, the initial text embedding of the entity is obtained, and the formula is as follows:

[0096] e t ′=W t f t Formula 3;

[0097] Where Wt 是 The text feature-specific transformation matrix processes the initial text embedding with a scaling factor η, and the formula is:

[0098]

[0099] The confidence of the prediction is limited by the scaling factor operation to soften the output distribution while maintaining the invariance of the prediction results; the multimodal knowledge graph completion model uses independent embedding layers and processing logic for each feature separately to process different feature modalities.

[0100] The multimodal knowledge graph completion model integrates information of different structural modalities, textual modalities, and visual modalities through a modal fusion layer to improve the model's comprehensive understanding and expression capabilities of entities, that is, by fusing the embedding of different modalities, the multimodal knowledge graph completion model can more comprehensively consider diverse information and more accurately represent the characteristics and semantic relationships of entities;

[0101] The core components of the modality fusion layer include the feature fusion network, which obtains two fused embeddings by combining structure embedding and text embedding, as well as structure embedding and image embedding; using x m Represents an embedding x of some modality m =e m ∪r m , x m contains the entity embedding and relation embedding in modality m; when the fusion structure embedding x3 and image embedding x v When , the calculation process of the fusion network is to embed the structure into x based on the ComplEx model. s and image embedding x v Split into real and imaginary parts and fuse them separately; first, define a simple sequential connection module S:

[0102] S(x)=σ(BN(Linear(Dropout(x;p)))) Formula 5;

[0103] In the formula, a random dropout layer with probability p is applied to the input vector x, where p is a hyperparameter and is adjusted according to the model requirements; Linear is a fully connected layer, BN is a batch normalization layer BatchNorm1D, and the activation function σ selects ReLU;

[0104] The sequential connection module supports multi-layer stacking. After the real or imaginary parts of the structure embedding and the image embedding pass through the sequential connection module, they are fused through element-by-element multiplication and then through the activation function to ensure nonlinearity and positive activation. The process can be repeated l times. The process of multi-layer module fusion is as follows:

[0105]

[0106] Where Re(·) represents the real part operation, Im(·) represents the imaginary part operation, the subscript re represents the real part of the embedding, the subscript im represents the real part of the embedding, l represents the number of stacked layers of sequentially connected modules, ⊙ represents element-by-element multiplication, and the activation function σ selects ReLU; after concatenating the real and imaginary parts of the fused embedding, the complete fused embedding x is obtained. sv : The formula is:

[0107] x sv =[x sv，re ;x sv，im ] Formula 8;

[0108] The semicolon in the formula represents the concatenation operation;

[0109] If the image embedding and text embedding have been scaled in the independent learning layers before fusion, then the two fusion forms with structure embedding, that is, the embedding methods combined with text embedding and image embedding respectively, do not need additional scaling operations; the two fused embeddings obtained by fusing structure embedding with text embedding and fusing structure embedding with image embedding need to be added to the loss calculation when making predictions.

[0110] In the semantic constraint layer, the factor interaction regularizer utilizes the semantic transfer properties between head entities and tail entities to improve the generalization ability of the multimodal knowledge graph completion model;

[0111] The semantic transfer property between the head entities and the tail entities is specifically: if the semantics of the two head entities in the triple are similar, then the semantics of the two tail entities linked by the same relationship should also be similar; conversely, if the semantics of the two head entities linked by the same relationship are not similar, then the two tail entities are also not similar; let h, r, t be used to represent the head entity, relationship, and tail entity of the triple, and h, r, t represent the corresponding embedding. For a given triple T i =(h i , r i , ti ), and T i Another triple T with the same relation j =(h j , r i , t j ), the loss calculation method of its regularization term is:

[0112] Similarity loss based on the similarity in the semantic transfer properties between the head and tail entities The calculation formula is:

[0113]

[0114] Among them, given T i , T j , It is the L2 norm constraint on the head entity and the tail entity in the triple, hr = h i r i Entity t i Embedding, tr = h j r i Entity t j Embedding of; Computation The similarity between the head and tail entities of each triple It is equivalent to calculating T i , T j The similarity between the two tail entities of ; by inference, we get t i ,t j With similar semantics, Minimize, reduce i ,t j The distance between embeddings, let t i ,t j The embeddings are more likely to be located in similar semantic spaces; the computational complexity of the model is reduced by converting the computation between two triplets into the computation between elements within a specific triplet;

[0115] Based on the difference in the semantic transfer properties between the head and tail entities, the difference loss The calculation formula is:

[0116]

[0117] in, The definition of is the same as above; the semantic space of the entity is bounded to ensure that the semantic space does not diverge; there is also a norm constraint According to the parallelogram law of vectors, the farther the semantic distance between hr and tr is, The smaller it is; when T i =(h i , ri , t i ), T j =(h j , r i , t j ) in the header entity h i 、h j If they are not similar, Minimize and enlarge t i ,t j The distance between embeddings is such that t i ,t j The semantic difference is greater;

[0118] Combining Formula 9 and Formula 10, and considering the similarities and differences in the semantic transfer properties between the head and tail entities, the loss calculation formula of the FI regularizer is as follows

[0119]

[0120] in, The definition of is the same as above. α is a hyperparameter used to adjust the weight of the L2 norm regularization term. γ is used to determine whether two entities belong to the same category. When γ = 1, they belong to the same category and the similarity in the semantic transfer property is used. When γ = 0, they do not belong to the same category and the difference in the semantic transfer property is used. The cosine is used to calculate the similarity γ of the two triple head entities. The formula is as follows:

[0121]

[0122] Among them, ∈ j It is a hyperparameter, which is used as the threshold for judging similarity and takes values ​​between [0,1]. i 、h j Taken from Equation 11.

[0123] The multimodal knowledge graph completion model MHF adopts the ComplEx decoder, and its scoring function calculation formula is as follows:

[0124]

[0125] Where (h, r, t)∈T, Re(·) and Im(·) represent the real and imaginary parts of the complex number, respectively; is the conjugate of the complex vector t; the standard component multilinear dot product<a,b,c> =∑ d a d b d c d , d is the vector dimension;

[0126] When the entity-level triple data is sparse, the multimodal knowledge graph completion model adjusts the modal weights at the relation level to balance the importance of each modality in the final prediction. It first separates the triples into Relation-aware sets Right now The following calculation steps are performed on each In the separate, RankBoost algorithm is applied to model the importance of the modality and generate weight ω for each modality of relation r. m (r), where The final prediction score is obtained by combining the scores of each modality to find the best modality combination; specifically:

[0127] Initialize all modal weights to 0, that is, ω m (r) = 0, The weight of each triple (h, r, e) in the RankBoost algorithm is initialized as ω m (r), D m (e) Updated in the iterative learning of the model; for the query (h, r, ?), the prediction results under each modality are counted to generate the weight ω of each modality of the relation r m (r): If the scores of other candidate tail entities e except the true tail entity are less than the score of the true tail entity t, they are considered to be correctly predicted, indicating function g m (e) is assigned a value of 1, otherwise it is an incorrect prediction, indicating function g m (e) is assigned a value of -1; the formula is as follows:

[0128]

[0129] The result of the statistical indicator function is used to measure the prediction accuracy of the model. The better the ranking prediction result of the triple (h, r, ?) under mode m, the higher the modal weight ω m The larger the value of m Need to go through iterations, the weight of the i-th iteration The calculation formula is as follows:

[0130] The value of each iteration is saved, and the maximum value is selected as Used to update the triple weight D m+1 (e) and the modal weight ω of relation r m (r):

[0131]

[0132] ωm (r) = ω m (r)+ω m Formula 17;

[0133] The multimodal knowledge graph completion method uses integrated reasoning to obtain the final prediction, chooses to directly combine scores to retain more original information, and applies different weights and combination strategies to more finely control the integration process. For each query, through independent learning layers, the decoder obtains three scores ψ from the three modalities. s (h, r, t), ψ v (h, r, t), ψ t (h, r, t) to directly reflect the advantages and limitations of the entity prediction modality; the calculation formula for the combined score is as follows:

[0134]

[0135] in, Contains structural modality, visual modality and textual modality, ψ m (h, r, t) represents the score of the triple under mode m, ω m (r) represents the weight of relation r under mode m.

[0136] The multimodal knowledge graph completion model is trained using cross entropy loss; for all triples, the decoder provides the probability of the corresponding triple being true using the ComplEx model, which is obtained by applying softmax to the output of the scoring function, and the modality The probability of obtaining is denoted as y m (t|(h, r), the calculation formula is as follows:

[0137] y m (t|(h, r))=softmax(ψ final (h, r, t)) Formula 19;

[0138] The CE loss calculation formula for mode m is as follows:

[0139]

[0140] The multimodal knowledge graph completion method uses the basic model ComplEx to simultaneously train the three modalities of structural modality, text modality and image modality to learn the intra-modal representations respectively, and obtain three independent modal embeddings; after passing through the modal fusion layer, two fused embeddings are obtained, and then the three single modal embeddings and the two fused embeddings are predicted and processed respectively to calculate different loss terms. The loss calculation of the basic function ComplEx model is as follows:

[0141]

[0142] The symbols in this example are explained as follows:

[0143]

[0144] Example:

[0145] In this example, the MHF model is evaluated on three mainstream multimodal benchmark datasets, including FB15K-237, DB15K, and YAGO15K. Relation triples are used as structural features, entity images are used as visual features, and entity descriptions are extracted from Wikidata as text features. FB15K-237 is a subset of the Freebase knowledge graph, mainly involving movies, actors, awards, sports, and other related content. It contains 14,951 entities and 592,213 relationship triples, and is mainly used for knowledge graph completion research, especially link prediction tasks. DB15K is a subset of the DBPedia knowledge graph. There are a large number of aligned relationships between the entities of DB15K and FB15K-237, mainly involving the numerical features and image information of the entities, and contains 14,777 entities and 99,028 relationship triples. YAGO15K is a subset of the Yago knowledge graph, which focuses on descriptive attributes of individuals, such as nationality, gender, and occupation, and contains 15,283 entities and 122,886 relationship triples. YAGO15K also has a large number of aligned relationships with the entities of FB15K-237, and contains numerical features and image information of entities.

[0146] The three datasets cover many fields such as geography, people, companies, audiovisual works, and medicines. They all contain numerical features and image information of entities, providing rich data resources for multimodal knowledge graph learning. There are a large number of sameAs relationships between the three datasets, that is, entities in different knowledge graphs point to the same entity. This provides valuable prior knowledge for entity matching and link prediction tasks. The three datasets have significant differences in the number of relationship types, sparsity, etc., which can better evaluate the robustness of the model. The statistical data of the dataset are shown in Table 2:

[0147] Table 2 Dataset details

[0148]

[0149] This example uses mean reciprocal rank (MRR), mean rank (MR) Hit@1, and Hit@10 as evaluation indicators.

[0150] For each test sample, the model predicts a set of candidate entities (answers), and MRR considers the ranking position of the correct entity in the candidate entity. The MRR value is obtained by summing the reciprocal rankings of the correct entities in all test samples and dividing it by the total number of test samples. The larger the MRR value, the better the model is. MR is similar to MRR. The MR value is obtained by summing the rankings of the correct entities in all test samples and dividing it by the total number of test samples. The larger the MRR value, the better the model is.

[0151] Hit@N is an indicator to measure whether there is a correct entity among the first N candidate entities predicted by the model. The calculation method is: for each test sample, if the first N candidate entities predicted by the model contain the correct entity, the Hit@N score of the test sample is 1, otherwise it is 0. The final evaluation indicator is the average of the Hit@N scores of all test samples. The larger the Hit@N value, the better the model effect.

[0152] This example selects two types of methods for comparison, namely, unimodal knowledge graph completion methods and multimodal knowledge graph completion methods. For unimodal models, the baselines used include:

[0153] TransE: defines relations as transformations between entities and designs an energy function of triples as the scoring function.

[0154] ConvE: Convert 1D entity and relation embeddings into 2D embeddings and use convolutional neural networks to model the interactions between entities and relations.

[0155] TuckER: Applying Tucker decomposition to capture high-level interactions between entity and relation embeddings.

[0156] For the multimodal model, the baselines used in this example include:

[0157] IKRL: Use the TransE energy function as the scoring function for each pair of modalities for joint prediction.

[0158] MKGC: Extending IKRL by combining different modalities to explicitly enforce consistency between modalities.

[0159] MKBE: adopts DistMult as the scoring function and designs GAN to predict the missing patterns.

[0160] IMF: TuckER is used as the scoring function. After learning the three modalities separately, a multimodal embedding that combines the three modalities is introduced for prediction.

[0161] This example uses NVIDIA3090 to run the experiments. The filtering setting of the widely used TransE is followed, that is, all correct entities in the answer set except the true query target are excluded, aiming to achieve a fair and accurate evaluation of the performance. ComplEx is used as the KGC decoder. Following RSME, this example uses pre-trained encoders ViT and BERT to extract visual and textual features of entities. The embedding dimension d is 4000 dimensions. The sequential connection module S in the modality fusion layer applies a random dropout layer with probability p, where p is set to 0.2. This example uses Adagrad to optimize the model. The hyperparameters are selected based on the best Hits@10 on the validation set.

[0162] Table 3 below shows the results of MHF and other baselines on the link prediction task on three datasets.

[0163] Table 3 Link prediction results on FB15K-237, YAGO15K and DB15K datasets

[0164]

[0165] The best result is in bold, and the second best result is underlined. “*” indicates that the original article did not report the result or some data was missing, and it was reproduced in this example. MHF performs well in all indicators of all three datasets, especially in MRR and Hit@1 indicators, which directly reflect the accuracy and efficiency of the model in predicting the correct answer. The average performance gain of the evaluation indicators MRR, Hits@1 and Hits@10 of all datasets also exceeds 5.4%. The MRR indicator result on DB15K increased by 8.7%, the largest among the three datasets. Not only that, the MR performance on DB15K increased significantly, with a gain of 26.8%. This example speculates that this is because the DB15K dataset has the largest number of relations, but the number of triplets in the training set, validation set and test set is the least among the three datasets, and the frequency of relations appearing in the triplets is relatively high. Since MHF allows each modality to have an independent relation embedding and adopts a relation-aware reasoning method, the relations are better trained, and the performance of the MR indicator has been greatly improved. MR simply calculates the average of the positions of all relevant items in the recommendation list. This indicator more intuitively reflects the average ranking position of all relevant queries. Therefore, a large gain in the MR indicator indicates that the model has made progress in improving the quality of the entire prediction list. Combined with the MRR on the DB15K dataset, the Hit@ indicator has also been greatly improved, illustrating the overall effectiveness of the MHF model. The unimodal method TuckER adopts various complex models to improve expressiveness and capture potential interactions. However, the results show that its performance is highly limited by the structural bias of the nature of the knowledge graph itself. Although these methods have achieved excellent results, MHF can easily outperform them significantly with a simpler model structure, which fully demonstrates its effectiveness.

[0166] Compared with the multimodal methods MKGC and MKBE that process different modal features separately, the MHF in this example jointly learns from different modalities through hierarchical fusion, which is conducive to modeling commonalities and complementarity at the same time. Compared with the most advanced multimodal completion method IMF (2023), which is also modal fusion, the fusion method of MHF is more scientific and reasonable. It not only combines the embeddings of all modalities, but also uses structural embedding as the backbone to utilize external features. In 10 of the 12 indicator results, MHF is significantly better than IMF. The excellent performance of MHF may be attributed to the fact that the feature fusion network can more effectively integrate multimodal information, extract and utilize complex relationships and entity features in the knowledge graph, and thus achieve better performance in the knowledge graph embedding task. This proves that the modal fusion layer successfully realizes the mutual interaction of feature information of different modalities, improves the consistency of the model's expression of entity information, and thus enhances the accuracy of knowledge completion.

[0167] In general, due to the effective fusion of multimodal information, the MHF model proposed in this example can model a more comprehensive interaction between different modalities while having commonalities and complementarities, thereby achieving a significant improvement in link prediction results.

Claims

1. A multimodal knowledge graph completion method based on modal hierarchical fusion can be used to complete or enhance the knowledge graph of a multimodal data set, and is characterized by: The completion method uses a multimodal knowledge graph completion model MHF based on modal hierarchical fusion to introduce visual features and text features. The multimodal knowledge graph completion model MHF includes: Independent learning layer: independently learn the structure embedding X from the knowledge graph of the dataset segmented by multiple modalities s , Visual EmbeddingX v , text embedding x t , while using a scaling factor to adjust the scale of external features; Modality fusion layer: The structural features are fused with the visual features and text features respectively to obtain two fused embeddings, which are included in the final loss calculation; Semantic Constraint Layer: The five embeddings mentioned above are further processed by the factor interaction regularizer, and the additional five regularization loss terms generated by the processing are calculated in the final loss.

2. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 1 is characterized by: The information fields of the multiple multimodal data sets include geography, people, companies, audiovisual works, and medicines, and the numerical features and image information of the entities contained therein provide data resources for the learning of the multimodal knowledge graph completion model. There are a large number of association relationships between the knowledge graphs of each multimodal data set, in which entities point to the same entity. The multimodal knowledge graph completion model MHF obtains the required prior knowledge information from such association relationships.

3. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 1 is characterized by: The independent learning layer uses the pre-trained encoder as a fixed feature extractor to obtain modality-specific entity representations and randomly initializes the structural features e s , use random numbers that conform to the normal distribution to initialize the relation embeddings of different modalities; The encoder includes an image encoder, which performs preprocessing on the images in the data set, including resizing, converting to tensors, and standardization, to ensure that the image data is normalized to the range of [-1, 1], and then obtains the feature representation f of the image by loading the pre-trained deep learning model. v , and then store the image path and its corresponding feature representation in a dictionary; the feature representation of the image f v After projection, the initial visual embedding of the entity is obtained, which can be expressed as: e v ′=W v f v Formula 1; Among them, W v is a transformation matrix specific to visual features. Assume that visual information and textual information are not always related to the factual triples, that is, external information carries uncertainty. In order to alleviate uncertainty, a scaling factor η is used to train visual and textual modalities in a confidence-constrained manner: The formula is: The image encoder uses the ViT model to randomly extract a picture of an entity for encoding to generate an image embedding of the entity.

4. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 3 is characterized by: The encoder uses the BERT model to encode the entity text name and text description to capture the text information of the entity, form an independent text embedding, and generate an embedding vector for a given entity description; the text encoder of the encoder reads the description information of the entity from the specified dataset file, and then generates the embedding vectors of these entity descriptions; if the entity description does not exist or is empty, a randomly initialized embedding vector is generated for the entity as its representation; The text encoding process adds special tags before and after the text, then uses BERT's tokenizer to segment the text and converts the segmentation results into corresponding vocabulary IDs; the text encoder truncates inputs with more than 510 tokens to ensure that the total length after including two special tags does not exceed 512 to meet the input length limit of the BERT model. Subsequently, these vocabulary IDs and paragraph IDs are uniformly set as constants to represent the same sentence, and then converted into tensors and sent to the pre-trained BERT model for forward propagation to obtain the hidden state of the model; the hidden state of the penultimate layer of the model is then used to generate the embedding vector of each token, and the arithmetic average of these hidden state vectors is calculated to generate the text embedding vector representation f t。 5. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 4 is characterized by: The text encoder uses the hidden state of the penultimate layer to generate the embedding vector of each word, and generates the text embedding vector representation by calculating the arithmetic mean of these hidden state vectors, providing a dense embedding representation for different entities or text paragraphs. The embedding representation captures the key information and contextual meaning of the text. The text embedding vector f t After projection, the initial text embedding of the entity is obtained, and the formula is as follows: e t ′=W t f t Formula 3; Where W t is a text feature-specific transformation matrix, which processes the initial text embedding with a scaling factor η, and the formula is: The confidence of the prediction is limited by the scaling factor operation to soften the output distribution while maintaining the invariance of the prediction results; the multimodal knowledge graph completion model uses independent embedding layers and processing logic for each feature separately to process different feature modalities.

6. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 3 is characterized by: The multimodal knowledge graph completion model integrates information of different structural modalities, textual modalities, and visual modalities through a modal fusion layer to improve the model's comprehensive understanding and expression capabilities of entities, that is, by fusing the embedding of different modalities, the multimodal knowledge graph completion model can more comprehensively consider diverse information and more accurately represent the characteristics and semantic relationships of entities; The core components of the modality fusion layer include the feature fusion network, which obtains two fused embeddings by combining structure embedding and text embedding, as well as structure embedding and image embedding; using x m Embedding representing a certain modality x m contains the entity embedding and relation embedding in modality m; when the fused structure embedding x s and image embedding x v When , the calculation process of the fusion network is to embed the structure into x based on the ComplEx model. s and image embedding x v Split into real and imaginary parts and fuse them separately; first, define a simple sequential connection module S: S(x)=σ(BN(Linear(Dropout(x;p)))) Formula 5; In the formula, a random dropout layer with probability p is applied to the input vector x, where p is a hyperparameter and is adjusted according to the model requirements; Linear is a fully connected layer, BN is a batch normalization layer BatchNorm1D, and the activation function σ selects ReLU; the sequential connection module supports multi-layer stacking, and the real or imaginary parts of the structure embedding and the image embedding are fused through element-by-element multiplication after passing through the sequential connection module, and then through the activation function to ensure nonlinearity and positive activation; the process can be repeated l times, and the process of multi-layer module fusion is as follows: Where Re(·) represents the real part operation, Im(·) represents the imaginary part operation, the subscript re represents the real part of the embedding, the subscript im represents the real part of the embedding, l represents the number of stacked layers of sequentially connected modules, ⊙ represents element-by-element multiplication, and the activation function σ selects ReLU; after concatenating the real and imaginary parts of the fused embedding, the complete fused embedding x is obtained. sv : The formula is: x su =[x sv，re ;x sv，im ] Formula 8; The semicolon in the formula represents the concatenation operation; If the image embedding and text embedding have been scaled in the independent learning layers before fusion, then the two fusion forms with structure embedding, that is, the embedding methods combined with text embedding and image embedding respectively, do not need additional scaling operations; the two fused embeddings obtained by fusing structure embedding with text embedding and fusing structure embedding with image embedding need to be added to the loss calculation when making predictions.

7. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 6 is characterized by: In the semantic constraint layer, the factor interaction regularizer utilizes the semantic transfer properties between head entities and tail entities to improve the generalization ability of the multimodal knowledge graph completion model; The semantic transfer property between the head entities and the tail entities is specifically: if the semantics of the two head entities in a triple are similar, then the semantics of the two tail entities linked by the same relationship should also be similar; Conversely, if the two head entities linked by the same relation are semantically dissimilar, then the two tail entities are also dissimilar; Let h, r, t represent the head entity, relation, and tail entity of the triple, and h, r, t represent the corresponding embedding. For a given triple T i =(h i , r i , t i ), and T i Another triple T with the same relation j =(h j , r i , t j ), the loss calculation method of its regularization term is: Similarity loss based on the similarity in the semantic transfer properties between the head and tail entities The calculation formula is: Among them, given T i 、T j , It is the L2 norm constraint on the head entity and the tail entity in the triple, hr = h i r i Entity t i Embedding, tr = h j r i Entity t j Embedding of; Computation The similarity between the head and tail entities of each triple It is equivalent to calculating T i 、T j The similarity between the two tail entities of ; by inference, we get t i ,t j With similar semantics, Minimize, reduce i ,t j The distance between embeddings, let t i ,t j The embeddings are more likely to be located in similar semantic spaces; the computational complexity of the model is reduced by converting the computation between two triplets into the computation between elements within a specific triplet; Based on the difference in the semantic transfer properties between the head and tail entities, the difference loss The calculation formula is: in, The definition of is the same as above; the semantic space of the entity is bounded to ensure that the semantic space does not diverge; there is also a norm constraint According to the parallelogram law of vectors, the farther the semantic distance between hr and tr is, The smaller it is; when T j =(h i , r i , t i ), T j =(h j , r i , t j ) in the header entity h i 、h j If they are not similar, Minimize and enlarge t i ,t j The distance between embeddings is such that t i ,t j The semantic difference is greater; Combining Formula 9 and Formula 10, and considering the similarities and differences in the semantic transfer properties between the head and tail entities, the loss calculation formula of the FI regularizer is as follows in, The definition of is the same as above. α is a hyperparameter used to adjust the weight of the L2 norm regularization term. γ is used to determine whether two entities belong to the same category. When γ = 1, they belong to the same category and the similarity in the semantic transfer property is used. When γ = 0, they do not belong to the same category and the difference in the semantic transfer property is used. The cosine is used to calculate the similarity γ of the two triple head entities. The formula is as follows: Among them, ∈ j It is a hyperparameter, which is used as the threshold for judging similarity and takes values ​​between [0,1]. i 、h j Taken from Equation 11.

8. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 1 is characterized by: The multimodal knowledge graph completion model MHF adopts the ComplEx decoder, and its scoring function calculation formula is as follows: in, Re(·) and Im(·) represent the real and imaginary parts of the complex number, respectively; is the conjugate of the complex vector t; the standard component multilinear dot product<a,b,c> =Σ d a d b d c d , d is the vector dimension; When the entity-level triple data is sparse, the multimodal knowledge graph completion model adjusts the modal weights at the relation level to balance the importance of each modality in the final prediction. It first separates the triples into Relation-aware sets Right now The following calculation steps are performed on each In the separate, RankBoost algorithm is applied to model the importance of the modality and generate a weight ω for each modality of the relation r. m (r), where The final prediction score is obtained by combining the scores of each modality to find the best modality combination; specifically: Initialize all modal weights to 0, that is, The weight of each triple (h, r, e) in the RankBoost algorithm is initialized as ω m (r), D m (e) Updated in the iterative learning of the model; for the query (h, r, ?), the prediction results under each modality are counted to generate the weight ω of each modality of the relation r m (r): If the scores of other candidate tail entities e except the true tail entity are less than the score of the true tail entity t, they are considered to be correctly predicted, indicating function g m (e) is assigned a value of 1, otherwise it is an incorrect prediction, indicating function g m (e) is assigned a value of -1; the formula is as follows: The result of the statistical indicator function is used to measure the prediction accuracy of the model. The better the ranking prediction result of the triple (h, r, ?) under mode m, the higher the modal weight ω m The larger the value of m Need to go through iterations, the weight of the i-th iteration The calculation formula is as follows: The value of each iteration is saved, and the maximum value is selected as Used to update the triple weight D m+1 (e) and the modal weight ω of relation r m (r): ω m ψ(r) = ω m ψ(r) + ω m Formula 17; The multimodal knowledge graph completion method uses integrated reasoning to obtain the final prediction, chooses to directly combine scores to retain more original information, and applies different weights and combination strategies to more finely control the integration process. For each query, through independent learning layers, the decoder obtains three scores ψ from the three modalities. s (h, r, t), ψ v (h, r, t), ψ t (h, r, t) to directly reflect the advantages and limitations of the entity prediction modality; the calculation formula for the combined score is as follows: in, Contains structural modality, visual modality and textual modality, ψ m (h, r, t) represents the score of the triple under mode m, ω m (r) represents the weight of relation r under mode m.

9. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 1, characterized in that: The multimodal knowledge graph completion model is trained using cross entropy loss; for all triples, the decoder provides the probability of the corresponding triple being true using the ComplEx model, which is obtained by applying softmax to the output of the scoring function, and the modality The probability of obtaining is denoted as y m (t|(h, r), the calculation formula is as follows: the m (t|(h,r))=softmax((ψ final (h,r,t)) 19: The CE loss calculation formula for mode m is as follows:

10. The multimodal knowledge graph completion method based on modality hierarchical fusion according to claim 9, characterized in that: The multimodal knowledge graph completion method uses the basic model ComplEx to simultaneously train the three modalities of structural modality, text modality and image modality to learn the intra-modal representations respectively, and obtain three independent modal embeddings; after passing through the modal fusion layer, two fused embeddings are obtained, and then the three single modal embeddings and the two fused embeddings are predicted and processed respectively to calculate different loss terms. The loss calculation of the basic function ComplEx model is as follows:

Images