A Multimodal Knowledge Graph Completion Method Based on Modal Hierarchy Fusion

Abstract

This invention proposes a multimodal knowledge graph completion method based on modal hierarchical fusion. It uses a multimodal knowledge graph completion model (MHF) to incorporate visual and textual features. The model includes: an independent learning layer that independently learns structural embeddings, visual embeddings, and textual embeddings from knowledge graphs of multimodal segmented datasets, while adjusting the scale of external features using scaling factors; a modal fusion layer that fuses structural features with visual and textual features respectively, resulting in two fused embeddings, which are included in the final loss calculation; and a semantic constraint layer that further processes the aforementioned five embeddings using a factor interaction regularizer, with five additional regularization loss terms generated during the processing included in the final loss. This invention integrates structural data with external data, thereby more rationally applying visual and textual data, effectively improving the model's expressive power and providing a new approach for the comprehensive and accurate representation of entities in knowledge graphs.

Description

Technical Field

[0001] This invention relates to the field of knowledge representation and reasoning technology of knowledge graphs, and in particular to a multimodal knowledge graph completion method based on modal hierarchical fusion. Background Technology

[0002] Traditional knowledge graph completion models, including translation-based and neural network-based methods, face the problem of structural bias between triples. These models are typically modality-specific and optimized for single-modality applications. Directly applying these models to multimodal scenarios, considering only the knowledge graph structure, often results in suboptimal performance, primarily due to the lack of mechanisms for fusing additional multimodal information (e.g., text, images). To address this issue, recent research has focused on enriching datasets and proposing novel MMKGC models to capture multimodal information for link prediction.

[0003] Existing MMKGC methods typically employ a shared relation embedding strategy, which tightly couples relations between different modalities. One approach is to fuse multimodal information of entities into a single embedding and then learn a unified relation representation based on the entity embedding. Another approach is to fuse external modal information of entities (i.e., removing structural modalities) into a single embedding and then learn a unified relation embedding to represent intramodal and intermodal relations. However, such studies require projecting all modalities into a unified relation space and leveraging commonalities between modalities for prediction. This approach struggles to preserve modality-specific knowledge, resulting in an inability to effectively model complex intermodal interactions to capture complementarity.

[0004] When dealing with the MMKGC task, it was found that the internal correlations of entity images are usually high, while the connection information with other related entities is relatively low. Existing models either ignore the role of visual modality in the inference process and directly use visual modality, which may introduce unnecessary noise; or they rely on separate learning modules to process visual information, failing to fully realize its potential value. Summary of the Invention

[0005] This invention proposes a multimodal knowledge graph completion method based on modal hierarchical fusion. By integrating structural data and external data, it can more rationally apply visual and textual data, effectively improving the expressive power of the model and providing a new approach for the comprehensive and accurate representation of entities in knowledge graphs.

[0006] The present invention adopts the following technical solution.

[0007] A multimodal knowledge graph completion method based on modal hierarchical fusion is used in question-answering systems, recommendation systems, or to complete or enhance the knowledge graph of a multimodal dataset. This completion method uses a multimodal knowledge graph completion model (MHF) based on modal hierarchical fusion to incorporate visual and textual features. The MHF model includes:

[0008] Independent learning layers: Independently learn structural embeddings x from the knowledge graph of a dataset segmented from multiple modalities. s Visual Embedding x v Text Embedding X t Meanwhile, a scaling factor is used to adjust the size of the external features;

[0009] Modality fusion layer: The structural features are fused with visual features and textual features respectively to obtain two fused embeddings, which are then included in the final loss calculation;

[0010] Semantic constraint layer: The five embeddings mentioned above are further processed by the factor interaction regularizer. The five additional regularization loss terms generated during the processing are calculated in the final loss.

[0011] The information domains of the multiple multimodal datasets include geography, people, companies, audio-visual works, and pharmaceuticals. The numerical features and image information of the entities contained therein provide data resources for the learning of the multimodal knowledge graph completion model. There are a large number of relationships between the knowledge graphs of the various multimodal datasets, with entities pointing to the same entity. The multimodal knowledge graph completion model (MHF) obtains the necessary prior knowledge information from these relationships.

[0012] The independent learning layer uses a pre-trained encoder as a fixed feature extractor to obtain modality-specific entity representations and randomly initializes structural features e. s The relation embeddings of different modalities are initialized using random numbers that conform to a normal distribution;

[0013] The encoder includes an image encoder, which performs preprocessing on the images in the dataset, including resizing, converting to tensors, and standardization, to ensure that the image data is normalized to the range [-1, 1]. Then, by loading a pre-trained deep learning model, the feature representation f of the image is obtained. v Then, the image path and its corresponding feature representation are stored in a dictionary; the image feature representation f v The initial visual embedding of the entity obtained after projection is expressed by the formula:

[0014] e v ′=W v f v Formula 1;

[0015] Among them, Wv This is a transformation matrix specific to visual features. Assuming that visual and textual information are not always correlated with fact triples (i.e., external information carries uncertainty), a scaling factor η is used to train the visual and textual modalities with confidence constraints to mitigate this uncertainty. The formula is:

[0016]

[0017] The image encoder uses the ViT model to randomly extract an image of the entity for encoding, generating an image embedding of the entity.

[0018] The encoder uses the BERT model to encode entity text names and descriptions to capture the textual information of entities, forming independent text embeddings and generating embedding vectors for a given entity description. The encoder's text encoder reads entity descriptions from a specified dataset file and then generates embedding vectors for these entity descriptions. If an entity description is missing or empty, a randomly initialized embedding vector is generated as its representation. The text encoding process involves adding special markers before and after the text, then using BERT's word segmenter for word segmentation, and converting the segmentation results into corresponding word IDs. The text encoder truncates inputs exceeding 510 words to ensure the total length, including the two special markers, does not exceed 512, meeting the input length limit of the BERT model. Subsequently, these word IDs and paragraph IDs are uniformly set as constants to represent the same sentence, converted into tensors, and fed into the pre-trained BERT model for forward propagation to obtain the model's hidden states. The hidden states of the penultimate layer are then used to generate embedding vectors for each word, and the arithmetic mean of these hidden state vectors is calculated to generate the text embedding vector representation f. t ;

[0019] The text encoder uses the penultimate hidden state to generate the embedding vector for each word, and generates a text embedding vector representation by calculating the arithmetic mean of these hidden state vectors. This provides a dense embedding representation for different entities or text paragraphs, capturing key information and contextual meaning of the text. The text embedding vector f t The initial text embedding of the entity is obtained after projection, as shown in the following formula:

[0020] e t ′=W t f t Formula 3;

[0021] Among them W t It is a transformation matrix specific to text features. The initial text embedding is processed using a scaling factor η, and the formula is:

[0022]

[0023] The confidence level of the prediction is limited by scaling factor operations to soften the output distribution while maintaining the invariance of the prediction results; the multimodal knowledge graph completion model uses independent embedding layers and processing logic for each feature separately to process different feature modalities in a separate manner.

[0024] The multimodal knowledge graph completion model integrates information from different structural modalities, text modalities, and visual modalities through a modality fusion layer to improve the model's comprehensive understanding and expression of entities. In other words, by embedding and fusing different modalities, the multimodal knowledge graph completion model can more comprehensively consider diverse information and more accurately represent the features and semantic relationships of entities.

[0025] The core component of the modality fusion layer includes a feature fusion network, which combines structural embedding and text embedding, as well as structural embedding and image embedding, to obtain two fused embeddings; using x m Embedding x representing a certain modality m =e m ∪r m , x m It includes entity embeddings and relation embeddings in modality m; when fusing structural embedding x3 and image embedding x... v At that time, the computation process of the fusion network is as follows: based on the ComplEx model, the structural embedding x3 and the image embedding x... v The components are split into real and imaginary parts and then merged separately. First, a simple sequential connection module S is defined:

[0026] S(x)=σ(BN(Linear(Dropout(x;p)))) Formula 5;

[0027] In the formula, a random deactivation layer with probability p is applied to the input vector x, where p is a hyperparameter that is adjusted according to the model requirements; Linear is a fully connected layer, BatchNorm1D is a batch normalization layer, and ReLU is chosen as the activation function σ; the sequential connection module supports multi-layer stacking, and the real or imaginary parts of the structural embedding and image embedding are fused through element-wise multiplication after passing through the sequential connection module, and then fused through an activation function to ensure nonlinearity and positive activation; the process can be repeated l times, and the multi-layer module fusion process is as follows:

[0028]

[0029] Where Re(·) represents the operation of extracting the real part, Im(·) represents the operation of extracting the imaginary part, the subscript re indicates the real part of the embedding, the subscript im indicates the real part of the embedding, l represents the stacking level of the sequentially connected modules, ⊙ indicates element-wise multiplication, and the activation function σ is ReLU; after concatenating the real and imaginary parts of the fused embedding, the complete fused embedding x is obtained. sv The formula is:

[0030] x sv =[x sv，re ;xs v，im ] Formula 8;

[0031] The semicolon in the formula represents a concatenation operation;

[0032] If both image embeddings and text embeddings have been scaled in their independent learning layers before fusion, then the two fusion methods with structural embeddings—that is, embedding methods that combine with text embeddings and image embeddings respectively—do not require additional scaling operations. When fusing structural embeddings and text embeddings, or fusing structural embeddings and image embeddings, the two fused embeddings obtained need to be added to the loss calculation.

[0033] In the semantic constraint layer, the factor interaction regularizer utilizes the semantic transitive properties between head entities and tail entities to improve the generalization ability of the multimodal knowledge graph completion model.

[0034] The semantic transitivity between head entities and tail entities is specifically defined as follows: if two head entities within a triple are semantically similar, then two tail entities linked by the same relation should also be semantically similar; conversely, if two head entities linked by the same relation are semantically dissimilar, then the two tail entities are also dissimilar. Let h, r, t represent the head entity, relation, and tail entity of a triple, and h, r, t represent the corresponding embeddings. For a given triple T... i =(h i r i , t i ), and with T i Another triplet T with the same relationship j =(h j r i , t h The loss calculation method for its regularization term is as follows:

[0035] Similarity loss is based on the similarity in the semantic transitivity between head and tail entities. The calculation formula is:

[0036]

[0037] Where T is given i T j, Apply L2 norm constraints to the head and tail entities in the triple, hr = h i r i Consider it as entity t i Embedded, tr=h j r i Consider it as entity t j Embedding; computation Similarity between the head and tail entities of each triple This is equivalent to calculating T. i T j The similarity between the two tail entities; t is obtained through reasoning. i t j Having similar semantics, by Minimize, shrink t i t j The distance between embeddings, let t i t j The embeddings are more likely to be located in similar semantic spaces; the computational cost of the model is reduced by transforming the computation between two triples into the computation between elements within a specific triple.

[0038] Based on the differences in semantic transitivity between head and tail entities, the difference loss is used. The calculation formula is:

[0039]

[0040] in, The definition is the same as above; the semantic space of an entity is bounded to ensure that the semantic space does not diverge; it also has norm constraints. According to the parallelogram law of vectors, the greater the semantic distance between hr and tr, the better. The smaller the value, the smaller the value; when T i =(h i r i , t i ), T j =(h j r i , t j The head entity h in ) i h j When they are dissimilar, by Minimize, increase t i t j The distance between embeddings makes t i t j The semantic differences are greater;

[0041] Combining Formulas 9 and 10, and considering the similarity and differences in the semantic transitivity between the head and tail entities, the loss calculation formula for the FI regularizer is as follows:

[0042]

[0043] Where S is defined as above, α is a hyperparameter used to adjust the weight of the L2 norm regularization term; γ is used to determine whether two entities belong to the same category. When γ = 1, they belong to the same category, and the similarity in the semantic transitive property is used; when γ = 0, they do not belong to the same category, and the difference in the semantic transitive property is used; the similarity γ between the head entities of two triplets is calculated using cosine similarity, as shown in the following formula:

[0044]

[0045] Where, ∈ j It is a hyperparameter used as a threshold for judging similarity, and its value is between [0,1]. i h j Taken from Formula 11.

[0046] The multimodal knowledge graph completion model MHF uses a ComplEx decoder, and its scoring function is calculated as follows:

[0047]

[0048] in, Re(·) and Im(·) represent taking the real and imaginary parts of a complex number, respectively; It is the conjugate of a complex vector t; the standard component multilinear dot product<a,b,c> =∑ d a d b d c d d is the vector dimension;

[0049] When entity-level triple data is sparse, the multimodal knowledge graph completion model adjusts the modality weights at the relation level to balance the importance of each modality in the final prediction. It first sets the triples according to relation r. Divided into relation-aware sets Right now The following calculation steps are for each In a separate study, the RankBoost algorithm was applied to model modal importance, generating a weight ω for each modality of relation r. m (r), where The final prediction score is obtained by combining the scores of each modality, thus finding the optimal combination of modalities; specifically:

[0050] Initialize all modal weights to 0, i.e., ω m(r) = 0, The weights of each triple (h, r, e) in the RankBoost algorithm are initialized as follows: ω m (r), D m (e) Update the model during iterative learning; for the query (h, r, ?), calculate the prediction results for each modality to generate the modal weights ω for relation r. m (r): If the score of candidate tail entity e (excluding the true tail entity) is less than the score of the true tail entity t, it is considered a correct prediction, indicated by the function g. m (e) Assigning a value of 1 indicates a false prediction, while the opposite indicates an incorrect prediction, indicated by the function g. m (e) is assigned the value -1; the formula is as follows:

[0051]

[0052] The result of the statistical indicator function is used to measure the model's prediction accuracy. A better prediction result for the triple (h, r, ?) under mode m is indicated by a higher mode weight ω. m The larger the value, the greater the modal weight ω. m Need to go through The weights for the i-th iteration. The calculation formula is as follows:

[0053]

[0054] The value of each iteration is saved, and the maximum value is selected as the [value]. Used to update the triple weight D m+1 (e) and the modal weights ω of relation r m (r):

[0055]

[0056] ω m (r)=ω m (r)+ω m Formula 17;

[0057] The multimodal knowledge graph completion method utilizes ensemble reasoning to obtain the final prediction. It selects direct score combination to retain more original information and applies different weights and combination strategies to more finely control the ensemble process. For each query, through independent learning layers, the decoder obtains three scores ψ from the three modalities. s (h, r, t), ψ v (h, r, t), ψ t (h, r, t) is used to directly reflect the advantages and limitations of entity prediction modes; the formula for calculating the combined score is as follows:

[0058]

[0059] in, Includes structural modality, visual modality, and text modality, ψ m (h, r, t) represents the score of the triple in mode m, ω m (r) represents the weight of relation r in mode m.

[0060] The multimodal knowledge graph completion model is trained using cross-entropy loss; for all triples, the decoder uses the ComplEx model to provide the probability that the corresponding triple is true, which is obtained by applying softmax to the output of the scoring function, and will be modally... The probability of obtaining it is represented by y. m (t|(h,r)) is calculated using the following formula:

[0061] y m (t|(h,r))=softmax(ψ) final Formula 19 (h, r, t)

[0062] The formula for calculating the CE loss of mode m is as follows:

[0063]

[0064] The proposed multimodal knowledge graph completion method uses the base model ComplEx to simultaneously train three modalities—structural, textual, and image—to learn intra-modal representations, resulting in three independent modal embeddings. After passing through a modality fusion layer, two fused embeddings are obtained. Then, prediction processing is performed on the three single-modal embeddings and the two fused embeddings respectively to calculate different loss terms. The loss calculation for the base function ComplEx model is as follows:

[0065]

[0066] The advantages of this invention are:

[0067] 1. This invention proposes an innovative Modal Hierarchical Fusion Model (MHF), which learns external features in stages, from easy to difficult, and effectively fuses complementary information from different modalities for link prediction. MHF comprises independent learning layers and a modal fusion layer, adjusting the confidence level of external information through a scaling factor and adaptively adjusting the importance of each modality during inference. Furthermore, the model employs a comprehensive loss function designed to simultaneously consider the combined effects of multimodal representations in prediction.

[0068] 2. The modal fusion layer designed in this invention utilizes the splitting and fusion of real and imaginary parts in complex space to fully capture the complementarity between different modes, thereby achieving effective fusion of multimodal information.

[0069] 3. This invention designs a novel factor interaction regularizer as a semantic constraint layer. Starting from the triples themselves, it uses the semantic information implicit in the head and tail entity transformation properties to suppress overfitting and enhance the generalization ability of the model.

[0070] The technical advantages of this invention are as follows:

[0071] 1. This invention advocates for models to learn from easy to difficult. The MHF model, through multi-layered learning with independent learning layers and modality fusion layers, demonstrates excellent performance in processing multimodal information in knowledge graphs. In the complex space, the introduction of feature fusion networks effectively improves the model's expressive power, providing a new approach for the comprehensive and accurate representation of entities in knowledge graphs.

[0072] 2. Existing models either neglect to evaluate the role of visual modalities in the reasoning process and directly utilize visual modalities, potentially introducing unnecessary noise; or they rely on separate learning modules to process visual information, failing to fully realize its potential value. To address this challenge, this invention proposes a novel modality embedding technique that integrates structural data with external data, thereby more rationally applying visual and textual data.

[0073] 3. Traditional regularizers avoid overfitting by limiting only the embedding size, achieving a balance between low computational cost and model performance. This invention extends the functionality of basic regularizers by designing a loss function that includes multiple regularization terms. This obtains more semantic constraint information while effectively controlling computational complexity. Furthermore, it leverages the semantic transitivity between head and tail entities to adjust the model and enhance its adaptability.

[0074] The MHF model of this invention has multimodal characteristics and has broad application potential in multiple fields, especially suitable for the following scenarios:

[0075] ① Question and Answer System:

[0076] It can be used for multimodal question answering: By utilizing multimodal data, more powerful question answering systems can be built, which can understand entities and relationships in natural language and use image and numerical information to provide answers.

[0077] It can be used for knowledge graph enhancement: By utilizing multimodal data, the knowledge graph in the question-answering system can be enhanced to contain richer information, thereby improving the accuracy and usability of the question-answering system.

[0078] ② Recommendation system:

[0079] It can be used for multimodal recommendation: By utilizing multimodal data, more personalized recommendation systems can be built, such as making recommendations based on users' interests, preferences, and entity relationships in knowledge graphs.

[0080] It can be used for knowledge graph-driven recommendations: Through multimodal knowledge graphs, knowledge-based recommendation systems can be built, such as making recommendations based on the relationships between entities. Attached Figure Description

[0081] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0082] Appendix Figure 1 This is a schematic diagram of the architecture of the multimodal knowledge graph completion model (MHF) described in this invention. Detailed Implementation

[0083] As shown in the figure, a multimodal knowledge graph completion method based on modal hierarchical fusion is used in question-answering systems, recommendation systems, or to complete or enhance the knowledge graph of a multimodal dataset. This completion method uses a multimodal knowledge graph completion model (MHF) based on modal hierarchical fusion to introduce visual and textual features. The MHF multimodal knowledge graph completion model includes:

[0084] Independent learning layer: Independently learns structural embeddings X from the knowledge graph of a dataset segmented from multiple modalities. S Visual Embedding x v Text Embedding X t Meanwhile, a scaling factor is used to adjust the size of the external features;

[0085] Modality fusion layer: The structural features are fused with visual features and textual features respectively to obtain two fused embeddings, which are then included in the final loss calculation;

[0086] Semantic constraint layer: The five embeddings mentioned above are further processed by the factor interaction regularizer. The five additional regularization loss terms generated during the processing are calculated in the final loss.

[0087] The information domains of the multiple multimodal datasets include geography, people, companies, audio-visual works, and pharmaceuticals. The numerical features and image information of the entities contained therein provide data resources for the learning of the multimodal knowledge graph completion model. There are a large number of relationships between the knowledge graphs of the various multimodal datasets, with entities pointing to the same entity. The multimodal knowledge graph completion model (MHF) obtains the necessary prior knowledge information from these relationships.

[0088] The independent learning layer uses a pre-trained encoder as a fixed feature extractor to obtain modality-specific entity representations and randomly initializes structural features e. sThe relation embeddings of different modalities are initialized using random numbers that conform to a normal distribution;

[0089] The encoder includes an image encoder, which performs preprocessing on the images in the dataset, including resizing, converting to tensors, and standardization, to ensure that the image data is normalized to the range [-1, 1]. Then, by loading a pre-trained deep learning model, the feature representation f of the image is obtained. v Then, the image path and its corresponding feature representation are stored in a dictionary; the image feature representation f v The initial visual embedding of the entity obtained after projection is expressed by the formula:

[0090] e v ′=W v f v Formula 1;

[0091] Among them, W v This is a transformation matrix specific to visual features. Assuming that visual and textual information are not always correlated with fact triples (i.e., external information carries uncertainty), a scaling factor η is used to train the visual and textual modalities with confidence constraints to mitigate this uncertainty. The formula is:

[0092]

[0093] The image encoder uses the ViT model to randomly extract an image of the entity for encoding, generating an image embedding of the entity.

[0094] The encoder uses the BERT model to encode entity text names and descriptions to capture the textual information of entities, forming independent text embeddings and generating embedding vectors for a given entity description. The encoder's text encoder reads entity descriptions from a specified dataset file and then generates embedding vectors for these entity descriptions. If an entity description is missing or empty, a randomly initialized embedding vector is generated as its representation. The text encoding process involves adding special markers before and after the text, then using BERT's word segmenter for word segmentation, and converting the segmentation results into corresponding word IDs. The text encoder truncates inputs exceeding 510 words to ensure the total length, including the two special markers, does not exceed 512, meeting the input length limit of the BERT model. Subsequently, these word IDs and paragraph IDs are uniformly set as constants to represent the same sentence, converted into tensors, and fed into the pre-trained BERT model for forward propagation to obtain the model's hidden states. The hidden states of the penultimate layer are then used to generate embedding vectors for each word, and the arithmetic mean of these hidden state vectors is calculated to generate the text embedding vector representation f. t ;

[0095] The text encoder uses the penultimate hidden state to generate the embedding vector for each word, and generates a text embedding vector representation by calculating the arithmetic mean of these hidden state vectors. This provides a dense embedding representation for different entities or text paragraphs, capturing key information and contextual meaning of the text. The text embedding vector f t The initial text embedding of the entity is obtained after projection, as shown in the following formula:

[0096] e t ′=W t f t Formula 3;

[0097] Among them Wt 是 The transformation matrix specific to the text features is used to process the initial text embedding with a scaling factor η, as shown in the formula:

[0098]

[0099] The confidence level of the prediction is limited by scaling factor operations to soften the output distribution while maintaining the invariance of the prediction results; the multimodal knowledge graph completion model uses independent embedding layers and processing logic for each feature separately to process different feature modalities in a separate manner.

[0100] The multimodal knowledge graph completion model integrates information from different structural modalities, text modalities, and visual modalities through a modality fusion layer to improve the model's comprehensive understanding and expression of entities. In other words, by embedding and fusing different modalities, the multimodal knowledge graph completion model can more comprehensively consider diverse information and more accurately represent the features and semantic relationships of entities.

[0101] The core component of the modality fusion layer includes a feature fusion network, which combines structural embeddings and text embeddings, as well as structural embeddings and image embeddings, to obtain two fused embeddings; using x m Embedding x representing a certain modality m =e m ∪r m , x m It includes entity embeddings and relation embeddings in modality m; when fusing structural embedding x3 and image embedding x... v At that time, the computation process of the fusion network is as follows: based on the ComplEx model, the structure is embedded into x. s and image embedding x v The components are split into real and imaginary parts and then merged separately. First, a simple sequential connection module S is defined:

[0102] S(x)=σ(BN(Linear(Dropout(x;p)))) Formula 5;

[0103] In the formula, a random deactivation layer with probability p is applied to the input vector x, where p is a hyperparameter that is adjusted according to the model requirements; Linear is a fully connected layer, BN is a batch normalization layer (BatchNorm1D), and the activation function σ is ReLU.

[0104] The sequential connection module supports multi-level stacking. The real or imaginary parts of the structural embedding and image embedding are fused through element-wise multiplication after passing through the sequential connection module, and then an activation function is applied to ensure non-linearity and positive activation. The process can be repeated l times. The multi-level module fusion process is as follows:

[0105]

[0106] Where Re(·) represents the operation of extracting the real part, Im(·) represents the operation of extracting the imaginary part, the subscript re indicates the real part of the embedding, the subscript im indicates the real part of the embedding, l represents the stacking level of the sequentially connected modules, ⊙ indicates element-wise multiplication, and the activation function σ is ReLU; after concatenating the real and imaginary parts of the fused embedding, the complete fused embedding x is obtained. sv The formula is:

[0107] x sv =[x sv，re ;x sv，im ] Formula 8;

[0108] The semicolon in the formula represents a concatenation operation;

[0109] If both image embeddings and text embeddings have been scaled in their independent learning layers before fusion, then the two fusion methods with structural embeddings—that is, embedding methods that combine with text embeddings and image embeddings respectively—do not require additional scaling operations. When fusing structural embeddings and text embeddings, or fusing structural embeddings and image embeddings, the two fused embeddings obtained need to be added to the loss calculation.

[0110] In the semantic constraint layer, the factor interaction regularizer utilizes the semantic transitive properties between head entities and tail entities to improve the generalization ability of the multimodal knowledge graph completion model.

[0111] The semantic transitivity between head entities and tail entities is specifically defined as follows: if two head entities within a triple are semantically similar, then two tail entities linked by the same relation should also be semantically similar; conversely, if two head entities linked by the same relation are semantically dissimilar, then the two tail entities are also dissimilar. Let h, r, t represent the head entity, relation, and tail entity of a triple, and h, r, t represent the corresponding embeddings. For a given triple T... i =(h i r i , ti ), and with T i Another triplet T with the same relationship j =(h j r i , t j The loss calculation method for its regularization term is as follows:

[0112] Similarity loss is based on the similarity in the semantic transitivity between head and tail entities. The calculation formula is:

[0113]

[0114] Where T is given i T j , Apply L2 norm constraints to the head and tail entities in the triple, hr = h i r i Consider it as entity t i Embedded, tr=h j r i Consider it as entity t j Embedding; computation Similarity between the head and tail entities of each triple This is equivalent to calculating T. i T j The similarity between the two tail entities; t is obtained through reasoning. i t j Having similar semantics, by Minimize, shrink t i t j The distance between embeddings, let t i t j The embeddings are more likely to be located in similar semantic spaces; the computational cost of the model is reduced by transforming the computation between two triples into the computation between elements within a specific triple.

[0115] Based on the differences in semantic transitivity between head and tail entities, the difference loss is used. The calculation formula is:

[0116]

[0117] in, The definition is the same as above; the semantic space of an entity is bounded to ensure that the semantic space does not diverge; it also has norm constraints. According to the parallelogram law of vectors, the greater the semantic distance between hr and tr, the better. The smaller the value, the smaller the value; when T i =(h i ri , t i ), T j =(h j r i , t j The head entity h in ) i h j When they are dissimilar, by Minimize, increase t i t j The distance between embeddings makes t i t j The semantic differences are greater;

[0118] Combining Formulas 9 and 10, and considering the similarity and differences in the semantic transitivity between the head and tail entities, the loss calculation formula for the FI regularizer is as follows:

[0119]

[0120] in, The definition is the same as above. α is a hyperparameter used to adjust the weight of the L2 norm regularization term; γ is used to determine whether two entities belong to the same category. When γ = 1, they belong to the same category, and the similarity property in semantic transitivity is used; when γ = 0, they do not belong to the same category, and the difference property in semantic transitivity is used; the similarity γ between the head entities of two triplets is calculated using cosine similarity, as shown in the following formula:

[0121]

[0122] Where, ∈ j It is a hyperparameter used as a threshold for judging similarity, and its value is between [0,1]. i h j Taken from Formula 11.

[0123] The multimodal knowledge graph completion model MHF uses a ComplEx decoder, and its scoring function is calculated as follows:

[0124]

[0125] Where (h, r, t)∈T, Re(·) and Im(·) represent taking the real and imaginary parts of the complex number, respectively; It is the conjugate of a complex vector t; the standard component multilinear dot product<a,b,c> =∑ d a d b d c d d is the vector dimension;

[0126] When entity-level triple data is sparse, the multimodal knowledge graph completion model adjusts the modality weights at the relation level to balance the importance of each modality in the final prediction. It first sets the triples according to relation r. Divided into relation-aware sets Right now The following calculation steps are for each In a separate study, the RankBoost algorithm was applied to model modal importance, generating a weight ω for each modality of relation r. m (r), where The final prediction score is obtained by combining the scores of each modality, thus finding the optimal combination of modalities; specifically:

[0127] Initialize all modal weights to 0, i.e., ω m (r) = 0, The weights of each triple (h, r, e) in the RankBoost algorithm are initialized as follows: ω m (r), D m (e) Update the model during iterative learning; for the query (h,r,?), calculate the prediction results for each modality to generate the modal weights ω for relation r. m (r): If the score of candidate tail entity e (excluding the true tail entity) is less than the score of the true tail entity t, it is considered a correct prediction, indicated by the function g. m (e) Assigning a value of 1 indicates a false prediction, while the opposite indicates an incorrect prediction, indicated by the function g. m (e) is assigned the value -1; the formula is as follows:

[0128]

[0129] The result of the statistical indicator function is used to measure the model's prediction accuracy. A better prediction result for the triple (h, r, ?) under mode m is indicated by a higher mode weight ω. m The larger the value, the greater the modal weight ω. m Need to go through The weights for the i-th iteration. The calculation formula is as follows:

[0130] The value of each iteration is saved, and the maximum value is selected as the [value]. Used to update the triple weight D m+1 (e) and the modal weights ω of relation r m (r):

[0131]

[0132] ωm (r)=ω m (r)+ω m Formula 17;

[0133] The multimodal knowledge graph completion method utilizes ensemble reasoning to obtain the final prediction. It selects direct score combination to retain more original information and applies different weights and combination strategies to more finely control the ensemble process. For each query, through independent learning layers, the decoder obtains three scores ψ from the three modalities. s (h, r, t), ψ v (h, r, t), ψ t (h, r, t) is used to directly reflect the advantages and limitations of entity prediction modes; the formula for calculating the combined score is as follows:

[0134]

[0135] in, Includes structural modality, visual modality, and text modality, ψ m (h, r, t) represents the score of the triple in mode m, ω m (r) represents the weight of relation r in mode m.

[0136] The multimodal knowledge graph completion model is trained using cross-entropy loss; for all triples, the decoder uses the ComplEx model to provide the probability that the corresponding triple is true, which is obtained by applying softmax to the output of the scoring function, and will be modally... The probability of obtaining it is represented by y. m (t|(h,r)) is calculated using the following formula:

[0137] y m (t|(h,r))=softmax(ψ) final Formula 19 (h, r, t)

[0138] The formula for calculating the CE loss of mode m is as follows:

[0139]

[0140] The proposed multimodal knowledge graph completion method uses the base model ComplEx to simultaneously train three modalities—structural, textual, and image—to learn intra-modal representations, resulting in three independent modal embeddings. After passing through a modality fusion layer, two fused embeddings are obtained. Then, prediction processing is performed on the three single-modal embeddings and the two fused embeddings respectively to calculate different loss terms. The loss calculation for the base function ComplEx model is as follows:

[0141]

[0142] The symbols used in this example are explained below:

[0143]

[0144] Example:

[0145] In this example, the MHF model is evaluated on three mainstream multimodal benchmark datasets: FB15K-237, DB15K, and YAGO15K. Relation triples are used as structural features, entity images as visual features, and entity descriptions are extracted from Wikidata as textual features. FB15K-237 is a subset of the Freebase knowledge graph, primarily covering content related to movies, actors, awards, sports, etc., containing 14,951 entities and 592,213 relation triples. It is mainly used for knowledge graph completion research, especially link prediction tasks. DB15K is a subset of the DBPedia knowledge graph. There are numerous alignment relationships between the entities in DB15K and FB15K-237, mainly involving numerical features and image information of entities, containing 14,777 entities and 99,028 relation triples. YAGO15K is a subset of the Yago knowledge graph, primarily focused on descriptive attributes of individuals such as nationality, gender, and occupation. It contains 15,283 entities and 122,886 relation triples. YAGO15K also has extensive alignment relationships with entities in FB15K-237 and includes numerical features and image information of the entities.

[0146] The three datasets cover a wide range of fields, including geography, people, companies, audio-visual works, and pharmaceuticals. They all contain numerical features and image information of entities, providing rich data resources for multimodal knowledge graph learning. There are numerous "sameAs" relationships among the three datasets, meaning entities in different knowledge graphs point to the same entity. This provides valuable prior knowledge for entity matching and link prediction tasks. The three datasets differ significantly in the number of relation types and their sparsity, allowing for better evaluation of model robustness. Statistical data for the datasets are shown in Table 2.

[0147] Table 2 Dataset Details

[0148]

[0149] This example uses Mean Reciprocal Rank (MRR), Mean Rank (MR) Hit@1, and Hit@10 as evaluation metrics.

[0150] For each test sample, the model predicts a set of candidate entities (the answer). MRR considers the rank of the correct entity among these candidate entities. The MRR value is obtained by summing the inverse ranks of the correct entities across all test samples and dividing by the total number of test samples. A higher MRR value indicates a better model performance. Similarly, MR is calculated by summing the rank of the correct entities across all test samples and dividing by the total number of test samples. A higher MRR value indicates a better model performance.

[0151] Hit@N is a metric that measures whether the correct entity exists among the top N candidate entities predicted by the model. It is calculated as follows: for each test sample, if the correct entity is among the top N candidate entities predicted by the model, the Hit@N score for that test sample is 1; otherwise, it is 0. The final evaluation metric is the average of the Hit@N scores for all test samples. A higher Hit@N value indicates a better model performance.

[0152] This example compares two types of methods: unimodal knowledge graph completion methods and multimodal knowledge graph completion methods. For unimodal models, the baselines used include:

[0153] TransE defines relations as transformations between entities and designs an energy function for triples as a scoring function.

[0154] ConvE: Transforms 1D entity and relation embeddings into 2D embeddings and uses convolutional neural networks to model the interactions between entities and relations.

[0155] TuckER: Applies Tucker decomposition to capture high-level interactions between entity and relation embeddings.

[0156] For multimodal models, the baselines used in this example include:

[0157] IKRL: Joint prediction is performed using the TransE energy function as the scoring function for each pair of modes.

[0158] MKGC: Extends IKRL by combining different modalities to explicitly achieve consistency between modalities.

[0159] MKBE: It uses DistMult as the scoring function and designs a GAN to predict missing patterns.

[0160] IMF: Using TuckER as the scoring function, after learning the three modalities separately, a multimodal embedding that integrates the three modalities is introduced for prediction.

[0161] This example uses an NVIDIA 3090 to run experiments. It follows the widely used TransE filtering setup, excluding all correct entities in the answer set except the true query target, aiming for a fair and accurate evaluation of performance. ComplEx is used as the KGC decoder. Following RSME, this example employs pre-trained encoders ViT and BERT to extract visual and textual features of entities. The embedding dimension d is 4000. The sequential connection module S in the modality fusion layer applies a randomly deactivated layer with probability p, where p is set to 0.2. Adagrad is used to optimize the model. Hyperparameters are chosen based on the best Hits@10 on the validation set.

[0162] Table 3 below shows the results of the link prediction task on the three datasets for MHF and other baselines.

[0163] Table 3 Link prediction results on the FB15K-237, YAGO15K, and DB15K datasets

[0164]

[0165] The best results are indicated in bold, and the second-best results are indicated by underline. "*" indicates that the original text did not report the results, or that some data was missing, and the results were reproduced in this example. MHF performed exceptionally well across all metrics on all three datasets, particularly on MRR and Hit@1 metrics, which directly reflect the accuracy and efficiency of the model in predicting correct answers. The average performance gain across all datasets for the evaluation metrics MRR, Hits@1, and Hits@10 exceeded 5.4%. The MRR metric on DB15K showed the largest increase of 8.7% among the three datasets. Furthermore, the MR performance on DB15K saw a significant increase of 26.8%. This is presumably because the DB15K dataset has the most relations, but the fewest triples in the training, validation, and test sets, resulting in a relatively high frequency of relations appearing in triples. Because MHF allows each modality to have independent relation embeddings and employs relation-aware inference methods, relations are trained more effectively, leading to a significant improvement in MR metric performance. MR simply calculates the average position of all relevant items in the recommendation list, a metric that more intuitively reflects the average rank position of all relevant queries. Therefore, a large gain in the MR metric indicates that the model has made progress in improving the quality of the entire prediction list. Combined with MRR on the DB15K dataset, the Hit@ metric also shows a significant improvement, demonstrating the overall effectiveness of the MHF model. The unimodal method TuckER employs various complex models to improve expressive power and capture potential interactions. However, results show that its performance is highly limited by structural biases inherent in the nature of knowledge graphs themselves. Despite the excellent results achieved by these methods, MHF can easily outperform them significantly with a simpler model structure, fully demonstrating its effectiveness.

[0166] Compared to multimodal methods MKGC and MKBE, which process different modal features separately, MHF in this example learns jointly from different modalities through hierarchical fusion, which is beneficial for simultaneously modeling commonalities and complementarities. Compared to the state-of-the-art multimodal completion method IMF (2023), which also involves modal fusion, MHF's fusion method is more scientific and reasonable. It does not simply combine the embeddings of all modalities together, but uses structural embedding as the backbone and utilizes external features. In 10 out of 12 metrics, MHF significantly outperforms IMF. MHF's superior performance may be attributed to the feature fusion network's ability to more effectively integrate multimodal information, extract and utilize complex relationships and entity features in the knowledge graph, thereby achieving better performance in knowledge graph embedding tasks. This demonstrates that the modal fusion layer successfully achieves the interaction of feature information from different modalities, improves the consistency of the model's representation of entity information, and thus enhances the accuracy of knowledge completion.

[0167] Overall, due to the effective fusion of multimodal information, the MHF model proposed in this example can model a more comprehensive interaction between different modalities, while possessing both commonalities and complementarities, thereby achieving a significant improvement in link prediction performance.

Claims

1. A multimodal knowledge graph completion method based on modal hierarchical fusion, which can be used to complete or enhance the knowledge graph of a multimodal dataset, characterized by: The completion method uses a multimodal knowledge graph completion model (MHF) based on modal hierarchical fusion to incorporate visual and textual features. The multimodal knowledge graph completion model (MHF) includes: Independent learning layers: Independently learn structural embeddings x from the knowledge graph of a dataset segmented from multiple modalities. s Visual Embedding x v Text embedding x t Meanwhile, a scaling factor is used to adjust the size of the external features; Modality fusion layer: The structural features are fused with visual features and textual features respectively to obtain two fused embeddings, which are then included in the final loss calculation; Semantic constraint layer: The five embeddings mentioned above are further processed by the factor interaction regularizer. The five additional regularization loss terms generated during the processing are calculated in the final loss. The information domains of the multiple multimodal datasets include geography, people, companies, audio-visual works, and pharmaceuticals. The numerical features and image information of the entities they contain provide data resources for the learning of the multimodal knowledge graph completion model (MHF). There are numerous relationships between the knowledge graphs of each multimodal dataset, with entities pointing to the same entity. The MHF model extracts the necessary prior knowledge information from these relationships. The MHF model integrates information from different structural modalities, textual modalities, and visual modalities through a modality fusion layer. The core component of the modality fusion layer includes a feature fusion network, which combines structural embedding and text embedding, as well as structural embedding and image embedding, to obtain two fused embeddings; using x m Embedding representing a certain mode x m It includes entity embeddings and relation embeddings in modality m; when fusing structural embedding x3 and image embedding x... v At that time, the computation process of the fusion network is as follows: based on the ComplEx model, the structural embedding x3 and the image embedding x3 are combined. v The components are split into real and imaginary parts and then merged separately. First, a simple sequential connection module S is defined: S(x)=σ(BN(Linear(Dropout(x,p)))) Formula 5; In the formula, a random deactivation layer with probability p is applied to the input vector x, where p is a hyperparameter that is adjusted according to the model requirements; Linear is a fully connected layer, BatchNorm1D is a batch normalization layer, and ReLU is chosen as the activation function σ; the sequential connection module supports multi-layer stacking. The real or imaginary parts of the structural embedding and image embedding are fused through element-wise multiplication after passing through the sequential connection module, and then fused using an activation function to ensure nonlinearity and positive activation; the process is repeated l times. The multi-layer module fusion process is as follows: Where Re(·) represents the operation of extracting the real part, Im(·) represents the operation of extracting the imaginary part, the subscript re indicates the real part of the embedding, the subscript im indicates the real part of the embedding, l represents the stacking level of the sequentially connected modules, ⊙ indicates element-wise multiplication, and the activation function σ is ReLU; after concatenating the real and imaginary parts of the fused embedding, the complete fused embedding x is obtained. sv The formula is: x sv =[x sv，re ;x sv，im ] Formula 8; The semicolon in the formula represents a concatenation operation; If both image embeddings and text embeddings have been scaled in their independent learning layers before fusion, then the two fusion methods with structural embeddings—that is, embedding methods that combine with text embeddings and image embeddings respectively—do not require additional scaling operations. When fusing structural embeddings and text embeddings, or fusing structural embeddings and image embeddings, the two fused embeddings obtained need to be added to the loss calculation.

2. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 1, characterized in that: The independent learning layer uses a pre-trained encoder as a fixed feature extractor to obtain modality-specific entity representations and randomly initializes structural features e. s The relation embeddings of different modalities are initialized using random numbers that conform to a normal distribution; The encoder includes an image encoder, which performs preprocessing on the images in the dataset, including resizing, converting to tensors, and standardization, to ensure that the image data is normalized to the range [-1, 1]. Then, by loading a pre-trained deep learning model, the feature representation f of the image is obtained. v Then, the image path and its corresponding feature representation are stored in a dictionary; the image feature representation f v The initial visual embedding of the entity obtained after projection is expressed by the formula: e v ′=W v f v Formula 1; Among them, W v This is a transformation matrix specific to visual features. Assuming that visual and textual information are not always correlated with fact triples (i.e., external information carries uncertainty), a scaling factor η is used to train the visual and textual modalities with confidence constraints to mitigate this uncertainty. The formula is: The image encoder uses the ViT model to randomly extract an image of the entity for encoding, generating an image embedding of the entity.

3. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 2, characterized in that: The encoder uses the BERT model to encode entity text names and text descriptions to capture the text information of entities, forming independent text embeddings and generating embedding vectors for a given entity description. The encoder's text encoder reads the entity description information from a specified dataset file and then generates embedding vectors for these entity descriptions. If the entity description does not exist or is empty, a randomly initialized embedding vector will be generated for the entity as its representation. The text encoding process involves adding special markers before and after the text, then using BERT's word segmenter to segment the text and converting the segmentation results into corresponding word IDs. The text encoder truncates inputs exceeding 510 words to ensure the total length, including the two special markers, does not exceed 512, meeting the input length limit of the BERT model. Subsequently, these word IDs and paragraph IDs are set to constants to represent the same sentence, converted into tensors, and fed into the pre-trained BERT model for forward propagation to obtain the model's hidden states. The hidden states of the penultimate layer are then used to generate the embedding vector for each word, and the arithmetic mean of these hidden state vectors is calculated to generate the text embedding vector representation f. t .

4. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 3, characterized in that: The text encoder uses the penultimate hidden state to generate the embedding vector for each word, and generates a text embedding vector representation by calculating the arithmetic mean of these hidden state vectors. This provides a dense embedding representation for different entities or text paragraphs, capturing key information and contextual meaning of the text. The text embedding vector f t The initial text embedding of the entity is obtained after projection, as shown in the following formula: e t ′=W t f t Formula 3; Among them W t It is a transformation matrix specific to text features. The initial text embedding is processed using a scaling factor η, and the formula is: The confidence level of the prediction is limited by scaling factor operations to soften the output distribution while maintaining the invariance of the prediction results; the multimodal knowledge graph completion model uses independent embedding layers and processing logic for each feature separately to process different feature modalities in a separate manner.

5. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 1, characterized in that: In the semantic constraint layer, the factor interaction regularizer utilizes the semantic transitive properties between head entities and tail entities to improve the generalization ability of the multimodal knowledge graph completion model. The semantic transfer properties between head entities and tail entities are as follows: if two head entities within a triple have similar semantics, then two tail entities linked by the same relation should also have similar semantics. Conversely, if two head entities linked by the same relation are semantically dissimilar, then the two tail entities are also dissimilar. Let h, r, t represent the head entity, relation, and tail entity of a triple, and let h, r, t represent the corresponding embeddings. For a given triple T i =(h i r i , t i ), and with T i Another triplet T with the same relationship j =(h j r i , t j The loss calculation method for its regularization term is as follows: Similarity loss is based on the similarity in the semantic transitivity between head and tail entities. The calculation formula is: Where T is given i T j , Apply L2 norm constraints to the head and tail entities in the triple, hr = h i r i Consider it as entity t i Embedded, tr=h j r i Consider it as entity t j Embedding; computation Similarity between the head and tail entities of each triple This is equivalent to calculating T. i T j The similarity between the two tail entities; t is obtained through reasoning. i t j Having similar semantics, by Minimize, shrink t i t j The distance between embeddings, let t i t j The embeddings are more likely to be located in similar semantic spaces; the computational cost of the model is reduced by transforming the computation between two triples into the computation between elements within a specific triple. Based on the differences in semantic transitivity between head and tail entities, the difference loss is used. The calculation formula is: in, The definition is the same as above; the semantic space of an entity is bounded to ensure that the semantic space does not diverge; it also has norm constraints. According to the parallelogram law of vectors, the greater the semantic distance between hr and tr, the better. The smaller T is; i =(h i ,r i ,t i ), T j =(h j ,r i ,t j The head entity h in ) i h j When they are dissimilar, by Minimize, increase t i t j The distance between embeddings makes t i t j The semantic differences are greater; Combining Formulas 9 and 10, and considering the similarity and differences in the semantic transitivity between the head and tail entities, the loss calculation formula for the FI regularizer is as follows: in, The definition is the same as above. α is a hyperparameter used to adjust the weight of the L2 norm regularization term; γ is used to determine whether two entities belong to the same category. When γ = 1, they belong to the same category, and the similarity property in semantic transitivity is used; when γ = 0, they do not belong to the same category, and the difference property in semantic transitivity is used; the similarity γ between the head entities of two triplets is calculated using cosine similarity, as shown in the following formula: Where, ∈ j It is a hyperparameter used as a threshold for judging similarity, and its value is between [0,1].

6. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 1, characterized in that: The multimodal knowledge graph completion model MHF uses a ComplEx decoder, and its scoring function is calculated as follows: in, Re(·) and Im(·) represent taking the real and imaginary parts of a complex number, respectively; It is the conjugate of a complex vector t; the standard component multilinear dot product<a,b,c> =∑ d a d b d c d d is the vector dimension; When entity-level triple data is sparse, the multimodal knowledge graph completion model adjusts the modality weights at the relation level to balance the importance of each modality in the final prediction. It first sets the triples according to relation r. Divided into relation-aware sets Right now The following calculation steps are for each In a separate study, the RankBoost algorithm was applied to model modal importance, generating a weight ω for each modality of relation r. m (r), where The final prediction score is obtained by combining the scores of each modality, thus finding the optimal combination of modalities; specifically: Initialize all modal weights to 0, i.e. The weights of each triple (h, r, e) in the RankBoost algorithm are initialized as follows: ω m (r), D m (e) Update the model during iterative learning; for the query (h, r, ?), calculate the prediction results for each modality to generate the modal weights ω for relation r. m (r): If the score of candidate tail entity e (excluding the true tail entity) is less than the score of the true tail entity t, it is considered a correct prediction, indicated by the function g. m (e) Assigning a value of 1 indicates a false prediction, while the opposite indicates an incorrect prediction, indicated by the function g. m (e) is assigned the value -1; the formula is as follows: The result of the statistical indicator function is used to measure the model's prediction accuracy. A better prediction result for the triple (h, r, ?) under mode m is indicated by a higher mode weight ω. m The larger the value, the greater the modal weight ω. m Need to go through The weights for the i-th iteration. The calculation formula is as follows: The value of each iteration is saved, and the maximum value is selected as the [value]. Used to update the triple weight D m+1 (e) and the modal weights ω of relation r m (r): ω m (r) = ω m (r) + ω m Formula 17; The multimodal knowledge graph completion method utilizes ensemble reasoning to obtain the final prediction. It selects direct score combination to retain more original information and applies different weights and combination strategies to more finely control the ensemble process. For each query, through independent learning layers, the decoder obtains three scores ψ from the three modalities. s (h, r, t), ψ v (h, r, t), ψ t (h, r, t) is used to directly reflect the advantages and limitations of entity prediction modes; the formula for calculating the combined score is as follows: in, Includes structural modality, visual modality, and text modality, ψ m (h,r,t) represents the score of the triple in mode m, ω m (r) represents the weight of relation r in mode m.

7. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 1, characterized in that: The multimodal knowledge graph completion model is trained using cross-entropy loss; for all triples, the decoder uses the ComplEx model to provide the probability that the corresponding triple is true, which is obtained by applying softmax to the output of the scoring function, and will be modally... The probability of obtaining it is represented by y. m (t|(h,r)), the calculation formula is as follows: the m (t|(h,r))=softmax(ψ final (h,r,t)) 19: The formula for calculating the CE loss of mode m is as follows:

8. The multimodal knowledge graph completion method based on modal hierarchical fusion according to claim 7, characterized in that: The proposed multimodal knowledge graph completion method uses the base model ComplEx to simultaneously train three modalities—structural, textual, and image—to learn intra-modal representations, resulting in three independent modal embeddings. After passing through a modality fusion layer, two fused embeddings are obtained. Then, prediction processing is performed on the three single-modal embeddings and the two fused embeddings respectively to calculate different loss terms. The loss calculation for the base function ComplEx model is as follows:

Images