Knowledge graph completion method based on multi-mode visual angle perception and deep neural network

Abstract

The invention relates to the field of knowledge graph completion, provides a knowledge graph completion method based on multi-modal visual angle perception and a deep neural network, and aims to solve the problems of weak multi-modal information expression ability, rough fusion mode and insufficient structural reasoning ability in the prior art. According to the method, structure information, text description and visual image information of an entity in a knowledge graph are obtained, structure, text and image modal input is constructed respectively, and a graph neural network, a pre-training language model and a visual encoder are adopted for feature coding; weighted fusion and semantic enhancement of multi-modal features are realized through a visual angle fusion mechanism and hierarchical attention processing; cross-modal contrast learning is introduced to improve modal consistency; and carrying out triple reasoning by using a uniform Transform encoder, and verifying a completion result by scores. According to the method, multi-modal semantics are effectively integrated, the entity representation capability and the triple prediction accuracy are improved, the model robustness is enhanced, and the method is suitable for application scenes such as intelligent question answering and recommendation systems and has remarkable practical value and popularization prospects.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph completion, particularly to the field of knowledge graph completion technology based on multimodal fusion, which involves unified modeling of text, image and structural information. Background Technology

[0002] Knowledge graphs are structured semantic networks used to represent entities and their relationships, and they have wide applications in fields such as natural language processing, intelligent question answering, and recommendation systems. However, practically constructed knowledge graphs often contain a large number of missing facts, resulting in insufficient structural and semantic completeness, which in turn affects the effectiveness of downstream applications. Therefore, predicting and completing missing triples has become an important research task in this field, known as knowledge graph completion.

[0003] In recent years, with the development of deep learning technology, traditional rule-based methods have been gradually replaced by embedded methods (such as TransE, DistMult, and ComplEx). These methods learn low-dimensional vector representations of entities and relations to complete the completion task. However, these methods usually rely only on structural information and are difficult to fully capture the external semantic features of entities.

[0004] To enhance the richness of knowledge graph representations, some studies have introduced external modal information such as text descriptions, images, and audio, forming the so-called Multimodal Knowledge Graph Completion (Multimodal KGC) method. Current methods attempt to fuse encoded images and text into entity representations, such as the MKGformer model. However, these methods generally suffer from the following problems: Unreasonable modal alignment assumptions: Most current models assume that different modalities (such as images and text) express the same semantics, only concatenating or averaging in the vector space, without considering the perspective differences in semantic representation of the same entity across modalities; Limited structural information modeling capabilities: Some models simplify the graph structure to static adjacency embeddings, lacking path-level reasoning modeling, resulting in insufficient structural reasoning capabilities; Lack of controllable modal information fusion mechanisms: Most fusion strategies are static concatenations, unable to dynamically perceive and allocate the importance of modalities, affecting the model's generalization ability and robustness.

[0005] In summary, existing technologies still face many challenges in unifying the modeling of multimodal semantic information and improving the completion reasoning ability, and there is an urgent need for a completion method with better semantic understanding and modal adaptability. Summary of the Invention

[0006] Purpose of the invention: In order to overcome the problems of weak multimodal information expression ability, crude modal fusion method and insufficient structural reasoning ability in the existing technology, this invention provides a knowledge graph completion method based on multimodal perspective perception and deep neural network, so as to improve the representation ability of entities and relations in knowledge graph and the prediction accuracy of missing triples, and achieve more interpretable and robust completion results.

[0007] The technical solution adopted in this invention is: a knowledge graph completion method based on multimodal perspective perception and deep neural networks, comprising the following steps:

[0008] Step 1: For entities in the knowledge graph, obtain their structural information (such as adjacent entities and relationships), textual description information (such as definitions and encyclopedia descriptions), and visual image information, and construct the original inputs for the structural modality, textual modality, and image modality respectively;

[0009] Step 2: Encode the structural modes of the data using a graph neural network (such as R-GCN);

[0010] Step 3: Semantically encode the entity description text using a pre-trained language model (such as BERT) for the text modality of the data;

[0011] Step 4: Extract visual feature representations of entity-related images from the image modalities of the data using a visual encoder (such as CLIP);

[0012] Step 5: For different modalities, use a viewpoint fusion mechanism to represent entity features;

[0013] Step 6: Perform hierarchical attention processing on the features;

[0014] Step 7: Design cross-modal contrastive learning to enhance modal representation;

[0015] Step 8: Combine the head entity, relation, and tail entity to be predicted (or reversed) into a unified Transformer encoder for sequence input;

[0016] Step 9: Calculate the triple score using the output obtained from the Transformer, and use the score to verify the triple completion result;

[0017] Furthermore, regarding step 1, this embodiment of the invention uses the FB15K-237 and WN18RR datasets, commonly used in knowledge graph completion tasks. These datasets contain descriptive information about entities and relationships. During the data preprocessing stage, when an entity or relationship has a clear natural language description, that description is preferentially used as the text modality input; if there is no description, the entity name or relationship name is used as the text information input. Simultaneously, images that semantically match the entities are retrieved from open image repositories (such as Wikimedia Commons, OpenImages, etc.) as image modality data, and a structured adjacency list of triples is constructed as the structured modality input.

[0018] Furthermore, for step 2, structural modality modeling employs graph neural networks (such as R-GCN) to capture the contextual relationships and multi-hop structural semantic information of entities within the knowledge graph. In practical implementation, the adjacency subgraphs of entities (generally set to 1-2 hop neighbors) are selected to construct a local graph structure, and relation-aware graph convolution is used to weighted aggregate neighbor information, thereby obtaining the structural modality embedding representation.

[0019] Furthermore, for step 3, the text modality encoding employs a pre-trained language model, such as BERT-base. First, the entity description text is segmented and embedded. Then, BERT is used for context-aware modeling to extract the [CLS] position information as the entity's text embedding vector. In cases where some descriptions are lengthy, they are truncated to the maximum input length (e.g., 512 tokens) to improve computational efficiency.

[0020] Furthermore, for step 4, the encoding of image modalities uses a pre-trained model such as CLIP-ViT-B / 32. Each entity can correspond to multiple images. First, each image is input into the CLIP image encoder to extract patch-level visual vectors. Then, average pooling or attention aggregation mechanisms are used to fuse the features of multiple images to form the visual modal representation of the entity.

[0021] Furthermore, regarding step 5, this invention employs a modal perspective fusion mechanism, using a gated attention network to weightedly fuse the embedding vectors from three modalities: structure, text, and image. Each modality corresponds to a learnable perspective weight, which is dynamically adjusted based on the current input context to adapt to the semantic emphasis of different entities in different modalities. For example, for abstract entities (such as "culture"), the text modality may dominate, while for concrete objects (such as "bicycle"), the image modality contributes more.

[0022] Furthermore, regarding step 6, a hierarchical attention mechanism was designed to enhance the semantic expressiveness and robustness of the fused representation. Within the modal layer, local attention is used to strengthen the semantic dependency structure within the modality; in the inter-modal layer, cross-modal attention is introduced to enhance the integration of complementary information between modalities, thereby generating a more discriminative unified entity representation.

[0023] Furthermore, regarding step 7, to avoid inconsistencies or information shifts between different modalities, this invention designs a cross-modal contrastive learning module. Different modal embeddings of the same entity are used as positive samples, and modal embeddings between different entities are used as negative samples. InfoNCE is used as the optimization objective to bring positive samples closer together and distance negative samples, thereby improving the interoperability and consistency between modal representations.

[0024] Furthermore, regarding step 8, during the model training phase, the head entity representation, relation representation, and the predicted tail entity position are constructed into an embedding sequence, which is then input into a unified Transformer encoder for interactive semantic modeling. This sequence typically takes the form [h fused ],[r],[MASK], where [MASK] represents the entity to be predicted.

[0025] Furthermore, in step 9, the Transformer output is scored by the prediction module, representing the confidence level of the current triple (h,r,t). The scoring function uses a dot product interaction, and the prediction results are used to generate a ranking of candidate entities. The training loss is the cross-entropy loss, supporting a sampling-based full-graph training strategy.

[0026] Beneficial effects:

[0027] The dynamic knowledge graph completion method based on multimodal perspective perception and deep neural networks described in this invention effectively integrates the structural information, textual description information, and image information of entities. It employs a modal perspective fusion mechanism and a hierarchical attention network to fully explore the complementary semantics between different modalities, enhancing the expressive and discriminative capabilities of entity representations. Simultaneously, by introducing a cross-modal contrastive learning strategy, it strengthens the semantic consistency between modalities and the model's robustness to modal omissions. End-to-end triple modeling and reasoning are implemented within a unified Transformer framework, improving the accuracy and stability of the completion results. This method is particularly suitable for knowledge representation and reasoning tasks in multimodal scenarios and has broad practical value and promising prospects for application in intelligent question answering, recommendation systems, and semantic search. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a knowledge graph completion process based on multimodal perspective perception and deep neural networks disclosed in an embodiment of the present invention;

[0029] Figure 2 This is a diagram of the R-GCN network structure for feature extraction in this invention;

[0030] Figure 3 This is a diagram of the BERT network structure for feature extraction in this invention;

[0031] Figure 4 This is a diagram of the VIT network structure for feature extraction in this invention;

[0032] Figure 5 This is the core framework diagram of the knowledge graph completion model of multimodal perspective perception and deep neural network in this invention. Detailed Implementation

[0033] The specific operational steps of a knowledge graph completion method based on multimodal perspective perception and deep neural networks according to the present invention will be described in more detail below with reference to the accompanying drawings. To further clarify the present invention, it should be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

[0034] A knowledge graph completion method based on multimodal perspective perception and deep neural networks, such as Figure 1-5 As shown, it includes the following steps:

[0035] Step 1: For entities in the knowledge graph, obtain their structural information, textual description information, and visual image information, and construct the original inputs for the structural modality, textual modality, and image modality respectively;

[0036] Furthermore, for each entity or relation in the knowledge graph, a corresponding text description is provided. If a text description exists, it is used; otherwise, the entity's name and its own name are used. For example, the original triple (h, r, t) is replaced with text to obtain the text triple (h...). text ,r text ,t text ).

[0037] (h,r,t)=(h text ,r text ,t text ) Formula (1)

[0038] Where h, r, and t represent the head entity, relation, and tail entity in the original triple, respectively; h text r text t text This indicates that the head entity, relation, and tail entity have been replaced with textual information based on knowledge.

[0039] Step 2: Encode the structural modes of the data using a graph neural network (R-GCN);

[0040] In knowledge graph completion tasks, the structural modalities of the data are encoded using a graph neural network (R-GCN) to obtain structural embedding representations. This process aggregates the representations of entity neighbors to the target entity through relation-aware graph convolution operations, effectively preserving multi-hop semantic information. The number of layers in the R-GCN directly affects the model's ability to model multiple relations. A single layer can only capture information from direct neighbors (1 hop), resulting in insufficient semantic expression. Two to three layers can effectively aggregate information from 1-hop and 2-hop (or even 3-hop) neighbors, balancing semantic richness and computational complexity. More than three layers may lead to oversmoothing, causing entity representations to become too similar and losing discriminative information. Therefore, in most knowledge graph completion tasks, a two-layer structure can capture the semantic information of 1-hop and 2-hop neighbors, making it suitable for the local structural features of most knowledge graphs.

[0041] Define the neighbor set Let r represent an adjacent entity connected to entity h through relation r. The structural embedding calculation formula is as follows:

[0042]

[0043] in, The structural modal embedding of entity h is represented by σ, which is a nonlinear activation function, and W is the structural modal embedding of entity h. r It is the weight matrix corresponding to relation r, c h,r This is the normalization coefficient.

[0044] Each layer uses relation-aware convolution to aggregate neighbor entity representations, with an output dimension of 256. It has 2 layers of R-GCN, with ReLU activation function and Dropout of 0.2 to prevent overfitting.

[0045] Step 3: Semantically encode the entity description text using a pre-trained language model (such as BERT) for the text modality of the data;

[0046] For text modalities of entities or relations in the data, the descriptive text of the entities or relations is input into the tokenizer of the BERT-base model. The tokenizer converts the text into a sequence of tokens and adds special markers [CLS] and [SEP] to the beginning and end of the sequence, respectively, forming an input in the format [CLS], w_1, w_2, ..., w_n, [SEP], where w_i represents a word and n is the sequence length, with a maximum of 512 tokens. If the text length exceeds this limit, it is truncated.

[0047] The BERT-base model employs a multi-layer Transformer architecture, containing 12 encoder layers, each with a 768-dimensional hidden representation and 12 self-attention heads, for a total of approximately 110 million parameters. The input token sequence is processed layer by layer by the model to generate a context-dependent representation for each token. After encoding each token in the sequence, the model outputs a set of feature vectors, where the vector corresponding to the [CLS] marker is extracted as the semantic vector representation of the entity. This vector has 768 dimensions and represents the global semantic information of the descriptive text.

[0048] The input is natural language description text of entities, and the model's [CLS] output serves as the semantic vector representation of the entities. Let the input text be texth; after BERT encoding, its embedding representation is:

[0049]

[0050] Where [CLS] represents the global representation position in the extracted feature output.

[0051] Step 4: Extract visual feature representations of entity-related images from the image modalities of the data using a visual encoder (such as CLIP);

[0052] First, for the image set of entity h, containing several images (e.g., 1 to 10 images), each image is processed by the visual encoder of the CLIP model. The CLIP visual encoder adopts a Transformer-based structure, containing 12 layers of visual Transformer modules, each with a 768-dimensional hidden representation and 12 self-attention heads, for a total of approximately 86 million parameters. The input images are first preprocessed to a fixed resolution (e.g., 224×224 pixels) and then converted to a tensor format suitable for model input.

[0053] Let the set of images of entity h be {I1, I2, ..., I...} k}, its image embedding is

[0054]

[0055] CLIP(I) i ) indicates the image I i Visual feature encoding, α i For attention weights, let I represent the image I. i The importance of entity representation.

[0056] The CLIP visual encoder encodes each image layer by layer, generating a corresponding visual feature vector with 768 dimensions, representing the global visual information of the image. For the image set of entity h, the visual feature vector of each image is extracted by the encoder to obtain a set of feature representations. To comprehensively utilize the information from multiple images, an attention mechanism is used to aggregate these feature vectors. Specifically, the attention mechanism calculates the contribution weight of each image to the entity representation, and the weight reflects the difference in importance of each image. The weight calculation is based on the semantic relevance of the feature vectors and is generated using a self-attention mechanism, specifically implemented through a multi-head self-attention module with 12 heads.

[0057] Based on the visual feature vectors of each image and their corresponding attention weights, a weighted aggregation is performed to generate an image embedding vector representation of entity h with a dimension of 768. This image embedding vector integrates the visual information of the entity-related images and can be directly used for knowledge graph completion tasks, or fused with textual and structural modal representations to enhance the multimodal representation capability of entities.

[0058] Step 5: For different modalities, use a viewpoint fusion mechanism to represent entity features;

[0059] This invention employs a modal perspective fusion mechanism to fuse embeddings of structural, textual, and image modalities. A gated attention mechanism is introduced during the fusion process to learn the importance of each modality within the specific entity semantics, generating a unified fused representation.

[0060] The embedding vectors for three modalities are obtained. The structural modality embedding vector is generated using the Relation-Aware Graph Convolutional Network (R-GCN), with a dimension of 256, representing the structural information of the entity in the knowledge graph. The textual modality embedding vector is generated using the pre-trained language model BERT-base, with a dimension of 768, representing the natural language description information of the entity. The image modality embedding vector is generated using the visual encoder CLIP, with a dimension of 768, representing the visual information of the image related to the entity. The embedding vector for each modality corresponds to the semantic features of structure, text, and image, respectively.

[0061] α m =Softmax(W·[e struct ;e text ;e image ])

[0062]

[0063] Where, α m e represents the gating attention weights for each modality. m Let W represent the embedding vector for the corresponding mode, and W be the learnable parameter matrix.

[0064] A gated attention mechanism is employed to fuse the embedding vectors of three modalities. This mechanism learns the importance of each modality within a specific entity's semantics, generating corresponding attention weights. Weight calculation is based on a learnable parameter matrix with a dimension of 768×768, used to linearly transform the input embedding vectors to capture semantic relevance between modalities. The attention weights are normalized using a softmax function, generating weight values ​​for the three modalities, representing the contribution of the structural, textual, and image modalities to the fused representation. The weight values ​​range from 0 to 1, and the sum of the three values ​​is 1.

[0065] The fusion process is achieved through weighted summation, which involves multiplying the 256-dimensional embedding vector of the structural modality, the 768-dimensional embedding vector of the text modality, and the 768-dimensional embedding vector of the image modality with their corresponding attention weights and then summing the results. To ensure dimensionality consistency, the structural modality embedding vector is mapped to 768 dimensions through a linear transformation layer before fusion. The final unified fused representation is a 768-dimensional vector that integrates the semantic information of the three modalities.

[0066] Step 6: Perform hierarchical attention processing on the features;

[0067] For various modalities in the data, a hierarchical attention mechanism is designed. Local attention is used within a modality to model the internal dependencies of the same modality, and cross attention is introduced between modalities to enhance intermodal synergy, thereby optimizing the entity representation after fusion.

[0068] First, a local attention mechanism is applied to the embedding expression of each modality to model its internal semantic dependencies. For the structural modality, the input is an embedding vector generated by the relation-aware graph convolutional network R-GCN, with a dimension of 256. This embedding vector is divided into multiple sub-blocks through a linear transformation layer, with four sub-blocks, each with a dimension of 64 (256 ÷ 4). Subsequently, a self-attention mechanism is applied to process these sub-blocks, generating an aggregated representation of the modality's internal structure, while maintaining the output dimension of 256. The self-attention mechanism employs a 4-head self-attention module, each with a dimension of 64, to represent the semantic dependencies between sub-blocks.

[0069] structural modal embedding e struct Represented as multiple sub-blocks through a linear layer. Compute its intramodal aggregate representation using attention:

[0070]

[0071] Similarly, local attention modeling is performed for text and image modalities. For the text modality, the input is an embedding vector generated by the pre-trained language model BERT-base, with a dimension of 768. This embedding vector is divided into 8 sub-blocks through a linear transformation layer, each with a dimension of 96 (768 ÷ 8). Local attention modeling is then performed using an 8-head self-attention module (each head with a dimension of 96) to generate an aggregated representation of the text modality, with an output dimension of 768. For the image modality, the input is an embedding vector generated by the visual encoder CLIP, with a dimension of 768, also divided into 8 sub-blocks, each with a dimension of 96. Modeling is performed using an 8-head self-attention module, maintaining the output dimension of 768.

[0072]

[0073] Here, SelfAttn(·) represents the self-attention operation, used to capture the semantic dependencies between sub-representations within a modality; l, m, and k are the number of sub-blocks in the structure, text, and image modalities, respectively.

[0074] Next, in the intermodal layer, the enhanced representations from the intramodal layer are used as input, and a cross-attention mechanism is employed to integrate complementary information between the modalities. Specifically:

[0075]

[0076] Here, CrossAttn(·) represents the cross-modal attention mechanism, and the output is the enhanced unified entity representation.

[0077] The input consists of locally attention-enhanced representations of the structural (256-dimensional), text (768-dimensional), and image (768-dimensional) modalities. A cross-attention mechanism integrates complementary information from each modality. To unify the dimensions, the 256-dimensional representation of the structural modality is mapped to 768 dimensions through a linear transformation layer. The cross-attention mechanism uses a 12-head attention module, each with a dimension of 64 (768 ÷ 12), and calculates the semantic relevance between modalities through learnable query, key, and value matrices. The output is an enhanced unified entity representation with a dimension of 768, integrating semantic information from the structural, text, and image modalities.

[0078] Step 7: Design cross-modal contrastive learning to enhance modal representation;

[0079] A cross-modal contrastive learning module is constructed. Different modal representations of the same entity are used as positive samples, and modal representations of different entities are used as negative samples. The module is trained using the InfoNCE loss function.

[0080]

[0081] Where sim(·) represents the cosine similarity function, τ is the temperature coefficient, and N is the sample size. These are representations of the same entity in different modalities. It represents a negative sample.

[0082] Step 8: Combine the head entity, relation, and tail entity to be predicted (or reversed) into a unified Transformer encoder for sequence input;

[0083] Before constructing the Transformer input sequence, a gated fusion layer is introduced to address the gated fusion representation in step 5. And the enhanced representation of step 6 Generate the final entity representation The gated fusion layer optimizes the semantic expressiveness of entity representations by allocating weights between multimodal and unimodal representations.

[0084] Dynamic balancing via gating network and The contribution of [the technology / method] generates multimodal intermediate representations:

[0085]

[0086] Where β1∈[0,1] are the multimodal gating weights. This is the final entity representation.

[0087] The head entity, relation, and predicted tail entity position are constructed into a unified Transformer encoder for the sequence input. The encoder improves triple inference performance by interactively modeling the fused representation.

[0088]

[0089] S' = Transformer(L)

[0090] in It is a fused representation of the head entity, e r It is a relational representation, where [MASK] represents the tail entity location that needs to be predicted.

[0091] Step 9: Calculate the triple score using the output obtained from the Transformer, and use the score to verify the triple completion result;

[0092] The Transformer outputs a representation vector, and the triplet confidence is calculated using a scoring function. The scoring function is in the form of a vector dot product. The training process uses cross-entropy loss, defined as follows:

[0093]

[0094] The final total loss function is the joint optimization objective:

[0095]

[0096] Where λ is the importance weighting coefficient for adjusting the connection prediction loss and the contrast loss.

[0097] The Transformer model generates representation vectors from the input triple elements (head entity h, relation r, and tail entity t). The Transformer model employs a 6-layer encoder structure, with each layer having a 512-dimensional hidden representation and 8 self-attention heads. The embedding vectors of the head entity h and tail entity t are generated through the aforementioned multimodal fusion mechanism, with a dimension of 768. The embedding vector of relation r is generated through a pre-trained embedding layer, with a dimension of 256. To ensure dimensionality consistency, the relation embeddings are mapped to 768 dimensions using a linear transformation layer before score calculation.

[0098] The confidence score of the triple (h, r, t) is calculated using a scoring function. This scoring function is a vector dot product, specifically the dot product of the 768-dimensional representation vectors of the head entity h, r, and t, generating a scalar score that characterizes the plausibility of the triple. A higher score indicates a greater likelihood that the triple represents a true fact. The calculated score is used to verify the triple completion result. Candidate tail entities (or head entities) are ranked, and the highest-scoring triple is selected as the completed result.

[0099] During the testing process, the triple scores are no longer converted into categories. Instead, the calculated triple scores are ranked, and MR, MRR, and Hits@n are used as model evaluation metrics. The smaller the MR value and the larger the MRR and Hits@n values, the better the model performance.

[0100] The tests used the general datasets FB15K-237 and WN18RR for knowledge graph completion to conduct entity prediction experiments.

[0101] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A knowledge graph completion method based on multimodal perspective perception and deep neural networks, characterized in that, The steps are as follows: Step 1: For entities in the knowledge graph, obtain their structural information, textual description information, and visual image information, and construct the original inputs for the structural modality, textual modality, and image modality respectively; Step 2: Encode the structural modes using a graph neural network to obtain structural mode embedding representations; Step 3: Semantically encode the entity description text of the text modality using a pre-trained language model to obtain the text modality embedding representation; Step 4: Extract visual features from entity-related images using a visual encoder to obtain image modality embedding representation; Step 5: The embedding vectors of the structural modality, text modality, and image modality are weighted and fused using a viewpoint fusion mechanism to generate a unified entity feature representation; Step 6: Perform hierarchical attention processing on the unified entity feature representation to enhance the integration of intramodal semantic dependencies and intermodal complementary information; Step 7: Design a cross-modal contrastive learning module, using different modal representations of the same entity as positive samples and modal representations of different entities as negative samples, and use the InfoNCE loss function to enhance the consistency of modal representations; Step 8: Construct a sequence from the head entity representation, relation representation, and the position of the tail entity to be predicted, and input it into a unified Transformer encoder for interactive semantic modeling; Step 9: Calculate the triplet score using the output of the Transformer encoder, and use the score to verify the triplet completion result.

2. The knowledge graph completion method based on multimodal perspective perception and deep neural networks as described in claim 1, characterized in that, For step 1, obtaining the structural information includes constructing an adjacency subgraph of the entity. The text description information preferentially uses the natural language description of the entity. If there is no description, the entity name or relation name is used. The visual image information retrieves images that semantically match the entity from an open image library.

3. The knowledge graph completion method based on multimodal perspective perception and deep neural networks as described in claim 2, characterized in that, For step 2, the graph neural network adopts a relation-aware graph convolutional network (R-GCN), which performs weighted aggregation of entity neighbor information through relation-aware graph convolution operations, and retains multi-hop semantic information.

4. The knowledge graph completion method based on multimodal perspective perception and deep neural networks as described in claim 3, characterized in that, For step 3, BERT's tokenizer is used to denote the descriptions or names of entities and relationships, and each word is mapped to an ID according to the provided dictionary. The [CLS] location information is extracted by the BERT model as a text modality embedding representation. If the description text exceeds the maximum input length, it is truncated to 512 tokens.

5. The knowledge graph completion method based on multimodal perspective perception and deep neural networks as described in claim 4, characterized in that, For step 4, the visual encoder uses the CLIP-ViT-B / 32 model to extract patch-level visual vectors from multiple images corresponding to the entity, and then fuses them through an attention aggregation mechanism to generate an image modality embedding representation.

6. The knowledge graph completion method based on multimodal perspective perception and deep neural networks as described in claim 5, characterized in that, Regarding step 5, the view fusion mechanism uses a gated attention network to perform weighted fusion of the embedding vectors of structural, textual, and image modalities. Each modality corresponds to a learnable view weight, which is dynamically adjusted according to the input context.

7. The knowledge graph completion method based on multimodal perspective perception and deep neural networks as described in claim 6, characterized in that, For step 6, the hierarchical attention processing includes intramodal local attention modeling and intermodal cross-modal attention modeling. Intramodal local attention is used to capture semantic dependencies within a modality, while intermodal cross-modal attention is used to integrate complementary information between modalities to generate an enhanced unified entity representation.

8. The knowledge graph completion method based on multimodal perspective perception and deep neural networks as described in claim 7, characterized in that, For step 8, the cross-modal contrastive learning module uses different modal representations of the same entity as positive samples and modal representations of different entities as negative samples, and optimizes them using the InfoNCE loss function.

9. The knowledge graph completion method based on multimodal perspective perception and deep neural networks as described in claim 8, characterized in that, In step 9, the input sequence of the Transformer encoder is in the form of [h], [r], [MASK], where h is the fused representation of the head entity, r is the relation representation, and [MASK] represents the position of the tail entity to be predicted. In step 10, the triple score is calculated through dot product interaction, the training process uses cross-entropy loss, the testing process ranks the triple scores, and MR, MRR, and Hits@n are used as model evaluation metrics.

Images