Knowledge graph completion method cooperatively driven by multi-modal information and gating attention
By combining TuckER and BERT with a gated attention mechanism, the problem of insufficient representation of sparse entities and complex relationships in knowledge graphs is solved, realizing dynamic fusion and collaborative modeling of multimodal information, and improving the integrity and reasoning ability of knowledge graphs.
Patent Information
- Application Number
- CN202511039730.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-28
- Publication Date
- 2025-10-31
AI Technical Summary
Existing knowledge graph completion methods are insufficient in representing sparse entities and complex relationships, and multimodal fusion strategies cannot dynamically adjust modal importance, resulting in insufficient reasoning and generalization capabilities.
The TuckER model is used for structured encoding, combined with the BERT model for text semantic encoding, and the structure and semantic information are dynamically fused through a gating attention mechanism to generate the final multimodal feature representation.
It enhances the ability to represent sparse entities and complex relationships, strengthens the collaborative modeling capability of multimodal information, and improves the integrity and reasoning ability of knowledge graphs.
Smart Images
Figure CN120874995A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of knowledge graph completion technology, specifically a knowledge graph completion method driven by multimodal information and gated attention. Background Technology
[0002] Knowledge graph completion (KGC) aims to infer missing links based on existing triples (head entity, relation, tail entity) to improve the completeness and reasoning ability of knowledge graphs, thereby supporting downstream applications such as intelligent question answering and recommendation systems. Existing knowledge graph completion methods mainly include structured embedding-based methods, methods that fuse entity description information, and multimodal fusion methods. Structured embedding-based methods learn the structural information of triples to map entities and relations into low-dimensional vectors, modeling the interaction between entities and relations. Methods that fuse entity description information introduce textual descriptions of entities and use pre-trained language models to extract semantic information, compensating for the lack of structural information. Multimodal fusion methods combine structural information and textual semantics, integrating features from both modalities through methods such as concatenation. The above methods still have the following drawbacks in practical applications: 1. Structured embedding methods rely on explicit connections in the graph, making it difficult to handle sparse entities, cold-start nodes, and complex relationship patterns, thus limiting their representational capabilities; 2. Although text augmentation models introduce entity description information to compensate for structural deficiencies, the significant differences between text and structural modalities mean that simple splicing or stacking can easily introduce redundancy or even noise, and some methods ignore structural information, resulting in insufficient reasoning and generalization capabilities; 3. Although existing multimodal fusion methods combine structure and semantics, the fusion strategies mostly adopt static splicing strategies, which cannot dynamically adjust the importance of different modalities according to the context, making it difficult to fully explore the complementarity between modalities. Summary of the Invention
[0003] The purpose of this invention is to provide a knowledge graph completion method driven by multimodal information and gated attention in order to solve the problems mentioned in the background art.
[0004] To achieve the above objectives, this invention provides the following technical solution: a knowledge graph completion method driven by multimodal information and gated attention, comprising the following steps: Step 1, data preparation and preprocessing; Step 2, TuckER-based structured encoding; Step 3, BERT-based text semantic encoding; Step 4, gated attention fusion; Step 5, decoder prediction and model optimization.
[0005] In step one above, the text description data of triples, entities and relations in the knowledge graph are prepared, and the data is preprocessed.
[0006] In step two above, the TuckER model is used as a structural encoder to map head entities, relations, and tail entities into low-dimensional vectors. Through the interaction of third-order kernel tensors, a structured representation of head entities and relations is generated.
[0007] In step three above, the BERT pre-trained model is used as a text encoder. The input sequence of head entity and relation text combination is used as input. The classification label position vector is extracted as global semantic features and converted into text semantic representation by a multilayer perceptron.
[0008] In step four above, the structured representation obtained in step two and the text semantic representation obtained in step three are first processed by feature concatenation and normalization, then dynamic weights are generated through a gating network, and finally the two representations are weighted and fused to obtain the final fused feature.
[0009] In step five above, based on the entity embedding matrix and the final fusion features obtained in step four, the matching score of the candidate tail entity is calculated, converted into a probability by Sigmoid, and the model parameters are optimized by binary cross-entropy loss. In the inference stage, the entity with the highest score is selected as the completion result.
[0010] Preferably, in step one, the preprocessing specifically involves: processing the triplet... The Xavier uniform initialization method is used to generate entity embedding matrices and relation embedding matrices, and the text description is formatted, where the triples include head entities. ,relation Tail-end entity The formatting process specifically involves extracting the textual descriptions of the entities and relations corresponding to the triples, and denoting them as the natural language descriptions of the head entities. Definitional description of the relationship Natural language description of tail entities By selecting a combination of head entity and relational text as the input sequence, the standard BERT input format can be constructed, which can be expressed by the following formula:
[0011]
[0012] in, These are classification tags used to aggregate global semantic information. This represents a sentence boundary separator, defining the boundary between entity descriptions and relation descriptions.
[0013] Preferably, in step two, the TuckER model first extracts head entities from the entity embedding matrix and the relation embedding matrix. ,relation Tail-end entity The vector representations of are denoted as . , , And construct the scoring function:
[0014]
[0015] in, This indicates that performing a modal product with the relation vector on the second dimension of the kernel tensor generates a relation-specific transformation matrix. The scoring function can be simplified to the following bilinear form:
[0016]
[0017] The structured representation formed after the interaction between the head entity and the relation is defined as:
[0018]
[0019] in, The head entity and relation interaction features output by the structured encoder contain structural semantic information of the knowledge graph.
[0020] Preferably, step three specifically involves: after inputting the input sequence of the head entity and relational text combination, generating the embedded layer. The input is fed into the BERT encoder, where a network composed of multiple Transformer layers uses a self-attention mechanism to capture the semantic dependencies between different locations. It selects the classification label location vector as the global semantic feature and outputs the corresponding context vector sequence. After undergoing linear transformation and nonlinear compression through a multilayer perceptron, the final text representation vector is obtained. Specifically, it can be expressed using the following formula:
[0021]
[0022] in, For word embedding, For location embedding, For sentence / segment embedding;
[0023]
[0024] in, For classification tags A vector of position.
[0025]
[0026] in, and These are the weight matrix and bias term of the projection layer, respectively.
[0027] Preferably, in step four, feature splicing and normalization can be represented by the following formula:
[0028]
[0029] in, To concatenate the normalized feature representation, For normalization processing, This involves concatenating vectors.
[0030] Preferably, in step four, generating dynamic weights through a gating network specifically involves: firstly, ... The input undergoes a linear transformation followed by ReLU activation to extract nonlinear features. A second linear transformation and Sigmoid activation are then applied to output the gating coefficients, which can be expressed by the following formula:
[0031]
[0032] in, , This is the weight matrix. , For bias terms, This is the Sigmoid activation function, used to compress the output to the [0,1] interval.
[0033] Preferably, in step four, the process of weighted fusion of the two representations can be expressed by the following formula:
[0034]
[0035] in, For the final fusion feature, This is the weighted structural representation. This is the weighted text representation. for The weighting coefficients.
[0036] Preferably, in step five, the matching score of the candidate tail entity can be calculated using the following formula:
[0037]
[0038] in, This represents the score of each entity in the current triple as the tail entity. The entity embedding matrix contains vector representations of all candidate entities.
[0039] Preferably, in step five, the Sigmoid transformation into probability can be expressed by the following formula:
[0040]
[0041] in, To predict probabilities, This is a Sigmoid conversion.
[0042] Preferably, in step five, the optimization of model parameters using the binary cross-entropy loss can be expressed by the following formula:
[0043]
[0044] in, Indicates the first Each entity is the real tail entity under the current triplet. The model provides the predicted probability; during the inference phase, the entity with the highest score is selected as the completion result. Specifically, during the inference phase, given... Yes, the model utilizes fused representations. The algorithm performs matching calculations with all entity embeddings, sorts them according to the prediction scores, selects the entity with the highest score as the prediction result, and completes the knowledge graph completion task.
[0045] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention introduces the BERT text encoder to mine the contextual semantic information in entity and relation descriptions, providing semantic supplementation for sparse entities and enhancing the ability to represent complex relations; it uses TuckER as a structural encoder to capture the interaction patterns between entities and relations, ensuring effective modeling of structural information; and it achieves dynamic weighted integration of structural and semantic embeddings through a gating attention mechanism, which can dynamically adjust the importance weights of different modalities according to the context, thereby improving the collaborative modeling capability of multimodal information. Attached Figure Description
[0046] Figure 1 This is a flowchart of the method of the present invention;
[0047] Figure 2 This is a diagram of the overall architecture of the HST-KG model of the present invention;
[0048] Figure 3 This is a diagram of the text encoder architecture of the present invention;
[0049] Figure 4 This is a schematic diagram of the gating attention fusion layer of the present invention. Detailed Implementation
[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0051] Please see the appendix Figure 1 -Appendix Figure 4 The present invention provides an embodiment of a knowledge graph completion method driven by multimodal information and gated attention, comprising the following steps: Step 1, data preparation and preprocessing; Step 2, TuckER-based structured encoding; Step 3, BERT-based text semantic encoding; Step 4, gated attention fusion; Step 5, decoder prediction and model optimization.
[0052] In step one above, the textual description data of triples, entities, and relations in the knowledge graph are prepared, and the data is preprocessed; specifically, the preprocessing involves: processing the triples... The Xavier uniform initialization method is used to generate entity embedding matrices and relation embedding matrices, and the text description is formatted, where the triples include head entities. ,relation Tail-end entity The formatting process specifically involves extracting the textual descriptions of the entities and relations corresponding to the triples, and denoting them as the natural language descriptions of the head entities. Definitional description of the relationship Natural language description of tail entities By selecting a combination of head entity and relational text as the input sequence, the standard BERT input format can be constructed, which can be expressed by the following formula:
[0053]
[0054] in, These are classification tags used to aggregate global semantic information. Indicates a sentence boundary separator, defining the boundary between entity descriptions and relation descriptions;
[0055] In step two above, the TuckER model is used as a structural encoder to map head entities, relations, and tail entities into low-dimensional vectors. Through interaction modeling with a third-order kernel tensor, a structured representation of the head entities and relations is generated. Specifically, the TuckER model first extracts the head entities from the entity embedding matrix and the relation embedding matrix. ,relation Tail-end entity The vector representations of are denoted as . , , And construct the scoring function:
[0056]
[0057] in, This indicates that performing a modal product with the relation vector on the second dimension of the kernel tensor generates a relation-specific transformation matrix. The scoring function can be simplified to the following bilinear form:
[0058]
[0059] The structured representation formed after the interaction between the head entity and the relation is defined as:
[0060]
[0061] in, The head entity and relation interaction features output by the structured encoder contain structural semantic information of the knowledge graph;
[0062] In step three above, a BERT pre-trained model is used as a text encoder. The input sequence consists of a combination of head entities and relational text. Classification label position vectors are extracted as global semantic features, which are then converted into a text semantic representation via a multilayer perceptron. Specifically, after inputting the input sequence of head entities and relational text, the resulting data is processed by an embedding layer to generate... The input is fed into the BERT encoder, where a network composed of multiple Transformer layers uses a self-attention mechanism to capture the semantic dependencies between different locations. It selects the classification label location vector as the global semantic feature and outputs the corresponding context vector sequence. After undergoing linear transformation and nonlinear compression through a multilayer perceptron, the final text representation vector is obtained. Specifically, it can be expressed using the following formula:
[0063]
[0064] in, For word embedding, For location embedding, For sentence / segment embedding;
[0065]
[0066] in, For classification tags A vector of position.
[0067]
[0068] in, and These are the weight matrix and bias term of the projection layer, respectively;
[0069] In step four above, the structured representation obtained in step two and the text semantic representation obtained in step three are first processed by feature concatenation and normalization, then dynamic weights are generated through a gating network, and finally the two representations are weighted and fused to obtain the final fused feature. The feature concatenation and normalization can be expressed by the following formula:
[0070]
[0071] in, To concatenate the normalized feature representation, For normalization processing, For vector concatenation; the dynamic weights are generated through a gating network, specifically: firstly, ... The input undergoes a linear transformation followed by ReLU activation to extract nonlinear features. A second linear transformation and Sigmoid activation are then applied to output the gating coefficients, which can be expressed by the following formula:
[0072]
[0073] in, , This is the weight matrix. , For bias terms, The Sigmoid activation function is used to compress the output to the [0,1] interval; the weighted fusion of the two representations can be expressed by the following formula:
[0074]
[0075] in, For the final fusion feature, This is the weighted structural representation. This is the weighted text representation. for Weighting coefficients;
[0076] In step five above, based on the entity embedding matrix and the final fusion features obtained in step four, the matching score of the candidate tail entity is calculated, converted into a probability using a sigmoid function, and the model parameters are optimized using a binary cross-entropy loss. During the inference phase, the entity with the highest score is selected as the completion result. The matching score of the candidate tail entity can be expressed by the following formula:
[0077]
[0078] in, This represents the score of each entity in the current triple as the tail entity. The entity embedding matrix contains vector representations of all candidate entities; after being converted to probabilities by the Sigmoid function, it can be expressed by the following formula:
[0079]
[0080] in, To predict probabilities, The Sigmoid transformation is used; the model parameters optimized using binary cross-entropy loss can be expressed by the following formula:
[0081]
[0082] in, Indicates the first Each entity is the real tail entity under the current triplet. The model provides the predicted probability; during the inference phase, the entity with the highest score is selected as the completion result. Specifically, during the inference phase, given... Yes, the model utilizes fused representations. The algorithm performs matching calculations with all entity embeddings, sorts them according to the prediction scores, selects the entity with the highest score as the prediction result, and completes the knowledge graph completion task.
[0083] Experimental Example 1:
[0084] To verify the effectiveness of the proposed HST-KG (Hybrid Struct Text KG Model) model in the knowledge graph completion task, link prediction experiments were conducted on two benchmark datasets, FB15k-237 and WN18RR, and compared with several representative mainstream models. The datasets FB15k-237 and WN18RR are shown in Table 1, and the comparison results are shown in Table 2. The experimental details are as follows: All models were trained under the same hardware environment and hyperparameter range to ensure fairness and reproducibility of the comparison. For embedding initialization, the entity and relation vectors were initialized using the Xavier uniform initialization method. The embedding dimension in both the structural encoder and semantic encoder was set to 200 to ensure a good balance between expressive power and computational efficiency. The optimizer was Adam, with adaptive adjustment of the learning rate, and an initial learning rate of 0.001. To prevent overfitting, an L2 regularization term was introduced during training, and its coefficients were obtained through grid search. Optimization was performed on the validation set, with a training batch size of 1024 and a maximum training epoch of 1000. Training was terminated early if the MRR metric on the validation set showed no improvement, with a maximum of 10 training epochs. For training sample construction, a negative sampling strategy was used to generate several negative triples for each positive triple. Specifically, the head or tail entity was fixed, and an entity at another position was replaced to construct negative examples. Binary cross-entropy loss was used for optimization, prompting the model to learn the discriminative boundary between positive and negative samples. To further improve training efficiency and model generalization ability, Dropout was introduced into each layer of the model, and normalization constraints were applied to the attention weights in the fusion module. Furthermore, in the gated attention layer, sigmoid activation was added to the calculation of the gating vector to enhance the dynamic adjustment capability of features from different sources during semantic fusion. All experiments were implemented using the PyTorch framework and on NVIDIA RTX. The experiment was run on a 4060 GPU. To ensure the stability of the results, each experiment was repeated three times, and the average value was reported. The specific model implementation and evaluation scripts followed an open and reproducible experimental paradigm. Several commonly used evaluation metrics in the field were used, including MRR (Mean Reciprocal Rank) and Hits@K (K=1,3,10). During the evaluation process, a test triplet was given. The model first depends on the head entity. and relationships Calculate the scores of all candidate tail entities and sort them accordingly; then record the actual tail entities. The rank in the sorted list is then used to calculate the following metric: MRR. This metric represents the average of the inverse ranks of all test samples, defined as follows:
[0085]
[0086] in The total number of test samples, Indicates the first The ranking of the true tail entities in each triplet is an indicator that can comprehensively reflect the overall predictive ranking performance of the model.
[0087] Hits@K (K=1,3,10) represents the proportion of true tail entities appearing in the top K predicted rankings, defined as follows:
[0088]
[0089] in, Hits@K is an indicator function that takes the value 1 when the condition is true and 0 otherwise. Hits@K measures the model’s ability to hit the correct entities in the Top-K range. The larger the value of K, the higher the tolerance for error.
[0090] As shown in Table 2, the proposed model HST-KG achieved optimal results across all metrics on both datasets. On the FB15k-237 dataset, HST-KG achieved an MRR of 0.361, Hits@1 of 0.271, Hits@3 of 0.422, and Hits@10 of 0.552, surpassing both the structure-based model TuckER (0.353 MRR) and the graph neural network model DRRGAT (0.360 MRR), and also outperforming the fusion model SAttLE (0.3). The result (58 MRR) demonstrates that HST-KG effectively combines structured and semantic information, exhibiting stronger generalization and expressive power in modeling complex entity relationships. On the WN18RR dataset, HST-KG also demonstrates leading performance, with an MRR of 0.485. It achieves MRRs of 0.448, 0.517, and 0.603 for Hits@1, Hits@3, and Hits@10 respectively, surpassing all other compared methods, especially in Hits@10 compared to the second-best model ISA-KGC (0.5). The 95) improvement of 0.8 percentage points indicates that HST-KG has stronger stability and accuracy in predicting tail entities. Further analysis of the performance of different types of models reveals that traditional embedding methods such as TransE and RotatE perform well on simple structures, but struggle to capture high-order relational semantics, resulting in relatively low overall performance. Graph neural network models (R-GCN, CompGCN, DRRGAT) enhance entity representation through neighbor information modeling, but suffer from information loss when dealing with semantically sparse or long-distance relationships. Fusion models (ISA-KGC, SAttLE) alleviate the problem of structural and semantic separation to some extent, but most adopt static splicing or unified mapping, lacking dynamic fusion mechanisms. In contrast, HST-KG extracts entity-relation interaction features using the structural encoder TuckER, enhances semantic perception using the pre-trained language model BERT, and achieves adaptive information fusion through a gated attention mechanism, effectively avoiding redundant feature interference and semantic overfitting. Ultimately, it achieves the best results on both datasets, verifying the effectiveness and robustness of the proposed method.
[0091] Experimental Example 2:
[0092] To deeply analyze the specific contributions of each component module in the proposed HST-KG model to the overall performance, ablation experiments were conducted. The structured encoder, text semantic representation, and gated attention fusion module were removed sequentially from the model. Tests were then performed on the FB15K-237 and WN18RR datasets. The results are shown in Table 3. Table 3 shows that the complete HST-KG model outperforms its ablation version in all metrics on both datasets, indicating that structural information, text semantic information, and the fusion mechanism all play crucial roles in the link prediction task. A comparison of the results of HST-KG and HSTw / oText further demonstrates this. It can be seen that removing the text module leaves the model with only information from the structured encoder TuckER, leading to an overall performance decrease. On FB15K-237, the MRR drops from 0.361 to 0.343, and on Hits@10 from 0.552 to 0.530. On WN18RR, the MRR drops from 0.485 to 0.466. This indicates that semantic description effectively complements structural modeling, especially when entity names are ambiguous or training samples are sparse. The contextual information provided by BERT helps the model learn more comprehensive representations. The results comparing HST-KG and HSTw / oStructure demonstrate this. The comparison shows that removing the structural module has a more significant impact on model performance, with MRR dropping to 0.330 (FB15K-237) and 0.452 (WN18RR), respectively. This indicates that relying solely on textual information is insufficient to capture structural patterns and inter-entity interactions in knowledge graphs. The significant decrease in metrics such as Hits@1 and Hits@3 further demonstrates the irreplaceable importance of structural representation in determining the accuracy of entity identification. Removing the gated fusion mechanism (HSTw / oGating) and replacing it with direct concatenation of structural and textual representations, although still retaining bimodal information, still results in overall performance inferior to the original model. On 15K-237, the MRR decreased to 0.347, and on Hits@10 it decreased to 0.540; on WN18RR, the MRR decreased to 0.471. These results indicate that the gating mechanism effectively mitigates redundant interference between different modal information. By adaptively adjusting the weights of structural and semantic features, the fused representation becomes more robust and discriminative. In summary, the structural encoder provides basic relational reasoning capabilities, the text module introduces additional semantic prior knowledge, and the gating fusion mechanism enhances the effectiveness of collaborative modeling between the two modalities. The ablation experiments fully validate the rationality and synergistic advantages of the HST-KG multimodal modeling design.
[0093] Table 1 Experimental Dataset
[0094]
[0095] Table 2 HST-KG Experimental Results
[0096]
[0097] Table 3 Results of HST-KG Ablation Experiment
[0098]
[0099] Based on the above, the advantages of this invention are that, when used, it constructs a multimodal coding architecture that integrates TuckER tensor decomposition structure and BERT expression, thereby achieving joint modeling of structural information and text semantics. This provides more complete representation support for sparse entities and complex relationships, solving the problems of insufficient ability of existing structured embedding methods to handle sparse entities and complex relationships, and text enhancement models ignoring structural information or introducing redundant noise through simple fusion. At the same time, a gated attention fusion mechanism is designed, which can dynamically adjust the contribution weights of structural embedding and semantic embedding according to the semantic and structural features of different samples, overcoming the defect that static splicing of multimodal methods cannot dynamically adapt to modal importance.
[0100] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A knowledge graph completion method driven by multimodal information and gated attention, including the following steps: Step 1: Data preparation and preprocessing; Step 2: TuckER-based structured encoding; Step 3: BERT-based text semantic encoding; Step 4: Gated attention fusion; Step 5: Decoder prediction and model optimization; Its key features are: In step one above, the text description data of triples, entities and relations in the knowledge graph are prepared, and the data is preprocessed. In step two above, the TuckER model is used as a structural encoder to map head entities, relations, and tail entities into low-dimensional vectors. Through the interaction of third-order kernel tensors, a structured representation of head entities and relations is generated. In step three above, the BERT pre-trained model is used as a text encoder. The input sequence of head entity and relation text combination is used as input. The classification label position vector is extracted as global semantic features and converted into text semantic representation by a multilayer perceptron. In step four above, the structured representation obtained in step two and the text semantic representation obtained in step three are first processed by feature concatenation and normalization, then dynamic weights are generated through a gating network, and finally the two representations are weighted and fused to obtain the final fused feature. In step five above, based on the entity embedding matrix and the final fusion features obtained in step four, the matching score of the candidate tail entity is calculated, converted into a probability by Sigmoid, and the model parameters are optimized by binary cross-entropy loss. In the inference stage, the entity with the highest score is selected as the completion result.
2. The knowledge graph completion method driven by multimodal information and gated attention as described in claim 1, characterized in that: In step one, the preprocessing specifically involves: processing the triplet... The Xavier uniform initialization method is used to generate entity embedding matrices and relation embedding matrices, and the text description is formatted, where the triples include head entities. ,relation Tail-end entity The formatting process specifically involves extracting the textual descriptions of the entities and relations corresponding to the triples, and denoting them as the natural language descriptions of the head entities. Definitional description of the relationship Natural language description of tail entities By selecting a combination of head entity and relational text as the input sequence, the standard BERT input format can be constructed, which can be expressed by the following formula: in, These are classification tags used to aggregate global semantic information. This represents a sentence boundary separator, defining the boundary between entity descriptions and relation descriptions.
3. The knowledge graph completion method driven by multimodal information and gated attention as described in claim 1, characterized in that: In step two, specifically: the TuckER model first extracts head entities from the entity embedding matrix and the relation embedding matrix. ,relation Tail-end entity The vector representations of are denoted as . , , And construct the scoring function: in, This indicates that performing a modal product with the relation vector on the second dimension of the kernel tensor generates a relation-specific transformation matrix. The scoring function can be simplified to the following bilinear form: The structured representation formed after the interaction between the head entity and the relation is defined as: in, The head entity and relation interaction features output by the structured encoder contain structural semantic information of the knowledge graph.
4. The knowledge graph completion method driven by multimodal information and gated attention as described in claim 1, characterized in that: In step three, specifically: after inputting the input sequence consisting of the head entity and relational text, the data is generated by the embedding layer. The input is fed into the BERT encoder, where a network composed of multiple Transformer layers uses a self-attention mechanism to capture the semantic dependencies between different locations. It selects the classification label location vector as the global semantic feature and outputs the corresponding context vector sequence. After undergoing linear transformation and nonlinear compression through a multilayer perceptron, the final text representation vector is obtained. Specifically, it can be expressed using the following formula: in, For word embedding, For location embedding, For sentence / segment embedding; in, For classification tags A vector of position. in, and These are the weight matrix and bias term of the projection layer, respectively.
5. The knowledge graph completion method driven by multimodal information and gated attention as described in claim 1, characterized in that: In step four, feature concatenation and normalization can be represented by the following formula: in, To concatenate the normalized feature representation, For normalization processing, This involves concatenating vectors.
6. The knowledge graph completion method driven by multimodal information and gated attention as described in claim 1, characterized in that: In step four, generating dynamic weights through a gating network specifically involves: firstly, ... The input undergoes a linear transformation followed by ReLU activation to extract nonlinear features. A second linear transformation and Sigmoid activation are then applied to output the gating coefficients, which can be expressed by the following formula: in, , This is the weight matrix. , For bias terms, This is the Sigmoid activation function, used to compress the output to the [0,1] interval.
7. The knowledge graph completion method driven by multimodal information and gated attention as described in claim 1, characterized in that: In step four, the process of weighted fusion of the two representations can be expressed by the following formula: in, For the final fusion feature, This is the weighted structural representation. This is the weighted text representation. for The weighting coefficients.
8. The knowledge graph completion method driven by multimodal information and gated attention as described in claim 1, characterized in that: In step five, the matching score of the candidate tail entity can be calculated using the following formula: in, This represents the score of each entity in the current triple as the tail entity. The entity embedding matrix contains vector representations of all candidate entities.
9. The knowledge graph completion method driven by multimodal information and gated attention as described in claim 1, characterized in that: In step five, the Sigmoid function conversion to probability can be expressed by the following formula: in, To predict probabilities, This is a Sigmoid conversion.
10. The knowledge graph completion method driven by multimodal information and gated attention as described in claim 1, characterized in that: In step five, the optimization of model parameters using the binary cross-entropy loss can be expressed by the following formula: in, Indicates the first Each entity is the real tail entity under the current triplet. The model provides the predicted probability; during the inference phase, the entity with the highest score is selected as the completion result. Specifically, during the inference phase, given... Yes, the model utilizes fused representations. The algorithm performs matching calculations with all entity embeddings, sorts them according to the prediction scores, selects the entity with the highest score as the prediction result, and completes the knowledge graph completion task.