Multi-modal knowledge completion method based on key information extraction and multi-granularity interaction

Abstract

The invention provides a multi-modal knowledge completion method based on key information extraction and multi-granularity interaction, and the method comprises the steps: carrying out the fine-granularity key information extraction of a multi-modal knowledge completion data set, and obtaining the keyword information in an entity description text and the key subgraph information in an entity image; based on the entity description text, the keyword information and the key subgraph information, carrying out data processing and visual redundancy feature aggregation on missing triple data, and constructing a training sample set; coding the training sample, and then carrying out multi-granularity cross-modal interaction to obtain a text feature and a multi-modal embedded visual feature; performing cross-modal fusion on the text features and the multi-modal embedded visual features to obtain cross-modal fusion features, combining all the cross-modal fusion features, performing multi-modal feature combination to obtain final multi-modal fusion features, and performing missing entity prediction; and constructing a total loss function based on a prediction result and a real label, and training the multi-modal knowledge completion model.

Description

Technical Field

[0001] The embodiments of this application relate to the field of knowledge completion technology, and in particular to a multimodal knowledge completion method based on key information extraction and multi-granularity interaction. Background Technology

[0002] Knowledge graphs (KGs) are a technology that represents entities and their relationships in the real world in a structured form. KGs provide a better ability to organize and understand information and are now widely used in the field of artificial intelligence, including multimedia reasoning, object detection, and recommender systems. KGs are incomplete due to the large amount of valuable knowledge missing or implicitly contained within them. This challenging problem has spurred research into knowledge graph completion (KGC) technology. With technological advancements, more and more KGs are incorporating multimodal information, making them closer to the real world. Specifically, multimodal information such as images and text can be seen as supplementary information to better represent knowledge triples, and thus can be used to enhance knowledge completion. Multimodal knowledge completion (MKGC) automatically discovers missing factual knowledge in a given graph by collaboratively modeling the structure and multimodal information of knowledge triples, thereby completing the knowledge completion process. Currently, most KGC tasks focus on entity prediction (link prediction), mainly following two technical paths: embedding-based methods or pre-trained model fine-tuning methods. Both of these methods can be used to solve non-multimodal knowledge completion and multimodal knowledge completion problems.

[0003] Embedded knowledge completion methods aim to embed entity-relation triples from a knowledge graph into one or more continuous representation spaces. A predefined scoring function is used to train the model on the known triples. This predefined scoring function calculates a relevance score between two entities based on their embedded representations. Depending on the scoring function, methods can be further categorized into distance-translation-based methods, tensor-factor-based methods, and neural network-based methods, among others.

[0004] Pre-trained model-based methods utilize large-scale multimodal data to pre-train Transformer architectures, fine-tuning them to adapt for downstream knowledge completion tasks. Compared to embedding-based methods (which typically freeze the pre-trained encoder), these methods can optimize encoder parameters to fully extract modal semantic information.

[0005] MKGformer proposes a multimodal knowledge completion model based on the Transformer architecture. It modifies the Transformer-based VIT and BERT to design a multi-level fusion multimodal encoding layer. The last few layers of VIT and BERT employ a multi-level fusion M-Encoder for integrated text-image entity modeling. A coarse-grained prefix guide reduces modal heterogeneity at the self-attention layer, and a fine-grained modal relevance perception fusion module aligns text and image information, mitigating the impact of visual noise. This completes the semantic modeling of multimodal triples, thereby predicting missing entities to achieve knowledge completion.

[0006] Chinese invention patent application number 202311392775.3 proposes a method for food safety knowledge graph completion based on multimodal information. This method constructs a food safety multimodal knowledge graph containing multimodal information and expands the scale of the food safety knowledge graph. It designs an attention-based multi-path multimodal feature encoder to obtain embedded representations of multimodal attributes of entities. By constructing a framework based on graph neural networks, it completes the implicit relations in the knowledge graph, retains the global structural information of the knowledge graph, and better captures the relational structure information of the knowledge graph. This achieves the fusion of semantic and structural information, improves the robustness of link prediction, and realizes high-quality food safety knowledge graph completion.

[0007] However, the inventors of this application have discovered the following drawbacks of the above method.

[0008] First, multimodal noise was not effectively filtered from the data source. Entities often contain multiple redundant images, with a large amount of redundant visual information interfering with the extraction of key entity regions. In entity text processing, long text descriptions contain irrelevant semantics, leading to the loss of core information and reducing the accuracy and semantic confidence of feature extraction.

[0009] Second, traditional methods mostly employ general attention mechanisms or simple fusion strategies for modal interaction, lacking guidance on key semantics and failing to accurately map from fine-grained entity key information to key semantic information in long entity text and entity images. This makes it difficult to bridge the semantic gaps between modalities, resulting in insufficient model representation capabilities for complex entities. Summary of the Invention

[0010] To address the aforementioned technical issues, embodiments of this application propose a multimodal knowledge completion method based on key information extraction and multi-granularity interaction. This method acquires high-confidence key image and text information from a data source using a pre-trained model and a fine-grained visual localization model. By designing a key information-guided multi-granularity cross-modal interaction method, it achieves multi-granularity alignment of the image and text information, guiding the model to learn entity image features under keyword features, thereby improving the model's ability to represent entities and ultimately enhancing knowledge completion performance.

[0011] To achieve the above objectives, embodiments of this application propose a multimodal knowledge completion method based on key information extraction and multi-granularity interaction, applicable to multimodal knowledge graphs. The method includes: fine-grained key information extraction from the multimodal knowledge completion dataset to obtain keyword information in entity description text and key subgraph information in entity images; wherein the multimodal knowledge completion dataset contains a large amount of entity description text and entity images; based on the entity description text, keyword information, and key subgraph information, data processing and visual redundancy feature aggregation are performed on missing triplet data to construct a training sample set; and a text encoder and a visual encoder are used to process the training samples in the training sample set. This process involves encoding the data and then performing multi-granularity cross-modal interactions guided by key information to obtain text features and multi-modal embedded visual features. Cross-modal fusion is then performed on the text features and multi-modal embedded visual features to obtain cross-modal fused features. All cross-modal fused features are combined and then merged using an aggregation FFN layer to obtain the final multi-modal fused features. Missing entity prediction is then performed based on these final multi-modal fused features. A total loss function is constructed based on the missing entity prediction results and the ground truth labels on the training samples. The multi-modal knowledge completion model is iteratively trained until convergence using the total loss function to obtain the trained multi-modal knowledge completion model, which is then used to perform the multi-modal knowledge completion task.

[0012] To achieve the above objectives, embodiments of this application also propose an electronic device, including a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement a multimodal knowledge completion method based on key information extraction and multi-granularity interaction as described above.

[0013] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a multimodal knowledge completion method based on key information extraction and multi-granularity interaction as described above.

[0014] Optionally, fine-grained key information extraction is performed on the multimodal knowledge completion dataset to obtain keyword information in entity description text and key subgraph information in entity images. This includes: using a large language model, based on preset prompts suitable for keyword extraction, extracting keywords related to image attributes from entity description text, and using BERT to encode the extracted keyword sequence to obtain keyword embedding vectors as keyword information; for entity images, using an object detector to extract key subgraph information most relevant to the keywords, inputting entity images and keywords into the Grounding-DINO model, and ensuring the quantity and quality of extracted key subgraphs by setting an effective subgraph confidence threshold.

[0015] Optionally, before performing data processing and visual redundancy feature aggregation on the missing triplet data based on entity description text, keyword information, and key subgraph information, the method further includes: randomly selecting an entity from the original triplet data and replacing it with a mask label to obtain the missing triplet data. The goal of the multimodal knowledge completion model is to predict the most likely entity corresponding to the mask label.

[0016] Optionally, based on entity description text, keyword information, and key subgraph information, data processing and visual redundancy feature aggregation are performed on the missing triplet data, including: The key subgraph information is embedded and encoded using a Vision Transformer as an image encoder to obtain the initial embedding of the key subgraph. , This can be expressed by the formula: , ; in, This indicates the number of patch embedding vectors after the key subgraph has been block-encoded. This represents the dimension of the embedding vector for each patch. Indicates the first Each patch embedding vector The dimension is space; Will The data is input into clustering units and the K-Means clustering algorithm is used to perform clustering. Perform clustering operations to obtain image embeddings for multiple categories. , This can be expressed by the formula: ; in, Indicates the first The initial set of embeddings for each cluster. Indicates the total number of clusters. This represents the K-Means clustering algorithm; Using a merging unit, pooling operations are used to merge and reduce the dimensionality of the image embeddings of each clustered group, extracting key sub-image embeddings after integrating redundant visual features. The clustered embedding is denoted as , is represented as: ; in, , This indicates a pooling operation. Indicates the first The number of patch embedding vectors in each cluster, all Clustered embeddings constitute entity image embeddings. , .

[0017] Optionally, the training samples in the training sample set are encoded using a text encoder and a visual encoder, and then multi-granular cross-modal interaction guided by key information is performed to obtain text features and multi-modal embedded visual features. This includes: using a text encoder to encode the triplet text sequence and keyword sequence in the training samples, and using a visual encoder to encode the image data in the training samples; using a multi-granular cross-modal interaction module guided by key information to process the outputs of the text encoder and the visual encoder, extracting coarse-grained text features by a self-attention unit, and using a cross-attention unit with text keyword features as attention guides to perform fine-grained interaction between the coarse-grained text features and image features through a multi-level attention architecture, ultimately obtaining text features and multi-modal embedded visual features.

[0018] Alternatively, for the attention mechanism, based on the input sequence vector Through the coefficient matrix , , ,Will Mapping to query vector Key vector Value vector In query vector Under the guidance of [the system / mechanism], attention calculation is completed, enabling interaction and fusion between data, and extracting global dependency features of the data. The calculation formula is as follows: ; in, This represents the attention mechanism. Represents the key vector The dimension; Multi-head attention mechanisms allow models to jointly pay attention to subspaces at different locations from different representations by performing attention functions in parallel on multiple heads and then concatenating them: ; ; in, This indicates a multi-head attention mechanism. This represents the multi-head attention coefficient matrix. Indicates the first One head, Set the text embedding for calculation The head is Keyword embedding The head is Visual Embedding The head is The multi-level attention calculation process guided by key information is as follows: ; ; Each embedding feature corresponds to a set of coefficient matrices and is consistent with the embedding feature subscripts. The fine-grained two-layer attention is divided into self-attention based on global triplet embedding and multi-level hybrid attention by connecting the key-value pairs of global triplet embedding and local key image embedding to local keyword embedding. Under the guidance of keywords, fine-grained interactive encoding between multiple modalities can be completed, thereby better representing the rich semantic features of triplets. Further expressed as: ; in, Indicates self-attention, Indicates cross attention. It represents the sum of normalized attention weights on the keyword embedding key-value pair vector, equivalent to adaptively adjusted self-attention and cross-attention.

[0019] Optionally, cross-modal fusion is performed on text features and multimodal embedded visual features to obtain cross-modal fused features. After combining all cross-modal fused features, a convergent FFN layer is used to merge the multimodal features to obtain the final multimodal fused features. Missing entity prediction is then performed based on the final multimodal fused features, including: The effectiveness of multimodal embedding visual features is measured by calculating the similarity between text features and multimodal embedding visual features. Each text feature... With all multimodal embedded visual features Similarity is calculated, and weights are obtained through the Softmax function. These weights are then aggregated to obtain cross-modal fusion features. Cross-modal fusion features This can be expressed by the formula: ; in, This represents the Softmax function; Combine all cross-modal fusion features This yields the final Fusion layer output aggregation embedding matrix. : ; in, Indicates the length of the original text sequence. Indicates the vector embedding dimension; Finally, the aggregation embedding matrix is... The multimodal features are merged into the modified aggregated FFN layer to obtain the final multimodal fused features. The aggregated FFN layer consists of two linear transformations and a ReLU activation function. ; in, Indicates the aggregation of FFN layers. , , , , These are all parameters to be learned. Represents the ReLU activation function; The output of the aggregated FFN layer is the final multimodal fusion feature. Then, using a mask prediction strategy, the output vector corresponding to the mask label is fed into the MLP network layer for missing entity prediction. Each possible entity is scored to obtain the entity set probability distribution. The entity with the highest score is the prediction result.

[0020] Optionally, during model training, since one-to-many and many-to-one relationships exist in the knowledge graph, the model is treated as a multi-label classification task, and the binary cross-entropy loss function is used as the total loss function. The total loss function is expressed by the formula: ; in, Indicates the number of entity categories. It is a probability distribution. This indicates that the model predicts the missing entity as the first... The probability of a class entity. One-hot encoding of real labels, due to the existence of one-to-many triples, means that the completed positive entity corresponding to each missing triple is not unique. For each candidate entity... If it can be used as a positive example to complete the triplet, then ,otherwise For positive examples, the similarity between the encoded representation of the model triples and the positive samples is maximized.

[0021] This application proposes a multimodal knowledge completion method based on key information extraction and multi-granularity interaction. Addressing the modal noise problem, it utilizes a large language model to generate text keyword sequences, improving the semantic confidence of entity descriptions. Simultaneously, it uses a visual localization model to lock key subgraphs of entities, designs a clustering and merging module to deeply mine and integrate image detail features, eliminate redundant visual noise, and extract fine-grained entity image semantics. To address the modal semantic gap problem, it proposes a key information-guided multi-granularity cross-modal interaction mechanism. Using keyword sequences as semantic bridges, it drives guided multi-granularity interactions between entity text, keyword sequences, and key subgraphs through hybrid attention, achieving deep alignment and fusion of cross-modal semantics and enhancing the model's representation ability of multimodal knowledge triples. In summary, this application obtains high-confidence key image and text information from the data source through a pre-trained model and a fine-grained visual localization model. By designing a key information-guided multi-granularity cross-modal interaction method, it achieves multi-granularity alignment of image and text information, guiding the model to learn entity image features under keyword features, thereby improving the model's entity representation ability and ultimately enhancing knowledge completion performance. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0023] Figure 1 This is a flowchart of a multimodal knowledge completion method based on key information extraction and multi-granularity interaction provided in one embodiment of this application; Figure 2 This is a structural diagram of a multimodal knowledge completion model provided in one embodiment of this application; Figure 3 This is a schematic diagram of a key information extraction process provided in one embodiment of this application; Figure 4 This is a structural diagram of a visual redundancy feature aggregation module provided in one embodiment of this application; Figure 5 This is a structural diagram of a multi-granularity cross-modal interaction module provided in one embodiment of this application; Figure 6 This is a structural diagram of an electronic device provided in another embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0025] The following sections will first introduce knowledge completion methods based on embedding and knowledge completion methods based on pre-trained models.

[0026] Embedsion-based knowledge completion methods aim to embed entity-relation triples from a knowledge graph into one or more continuous representation spaces. On known triples, a predefined scoring function can be used to train the model. This scoring function calculates the relevance score between two entities based on their embedded representations. Depending on the scoring function, these methods can be categorized into distance-translation-based, tensor decomposition-based, and neural network-based methods. Representative models include the relation-dependent translation scoring function, the TransE model which evaluates triple confidence using distance metrics, the DistMult model which projects entities and relations into a complex vector space, and the ConvE model, which first uses convolutional neural networks (CNNs) to explore the interaction between entity and relation embeddings. The performance of knowledge completion models largely depends on whether the model fully utilizes the information in the knowledge graph. The aforementioned models only use the structural or textual information of the knowledge graph and do not fully utilize the multimodal information within it. IKRL was the first to introduce visual information into knowledge representation, using an attention mechanism to integrate visual information with the structural information encoded by TransE. TransAE jointly learns visual and textual features into a unified knowledge embedding through an autoencoder. To address the visual noise problem in multimodal knowledge completion, RSME designed a forgetting gate for adaptive modality information selection to choose valuable images for multimodal knowledge completion (KGR). Meanwhile, the LAFA model dynamically evaluates the importance of images to entities and aggregates structural information from neighbors by linking perceptual fusion and aggregation modules.

[0027] Pre-trained model-based knowledge completion methods utilize large-scale multimodal data (such as text-image pairs) to pre-train a Transformer architecture, fine-tuning it to adapt to downstream knowledge completion tasks. Compared to embedding-based methods (which typically freeze the pre-trained encoder), these methods can optimize encoder parameters to fully extract modal semantic information. Single-modal knowledge completion methods based on pre-trained models are usually text-based, using fine-tuning of the pre-trained language model BERT. KG-BERT was the first method to model knowledge completion as a text classification task, constructing input sequences using textual entities and relations, encoding them through a pre-trained model, and predicting missing entities. Multimodal knowledge completion methods based on pre-trained models primarily convert MKG triples into token sequences, feeding them into a PLM for multimodal encoding learning. Due to the presence of visually noisy data in multimodal knowledge graphs and the high requirements for modal interaction fusion, designing effective multimodal pre-trained models for MKGC tasks is a crucial research issue. MKGformer proposes a multimodal knowledge completion model based on Transformer design. It achieves good results by modifying the Transformer-based VIT and BERT architectures to design a multi-level fusion multimodal encoding layer. SGMPT is the first to incorporate knowledge graph structural information into pre-trained Transformers. Through a structural encoder and a structure-guided fusion module, structural information is injected into text and visual features, effectively improving inference performance. MMKGR uses image filtering and image enhancement algorithms to handle image noise. In recent years, some contrastive learning strategies suitable for MKGC have been introduced into MKGC, enhancing the model's knowledge generalization ability through contrastive learning. Furthermore, generative methods are gaining popularity. MACO completes missing modal features using adversarial generative networks; NativE designs a relation-guided dual adaptive fusion module to alleviate modality imbalance; and VISTA introduces a modality-aware attention mechanism to dynamically allocate fusion weights for text and visual features.

[0028] For entities containing long text descriptions or multiple images, existing multimodal knowledge completion methods cannot fully extract their key semantic and visual information, resulting in insufficient model representation ability of multimodal knowledge triples. The first technical problem to be solved in this application is how to filter noise information in multimodal entity data, accurately locate key regions in entity images and core semantics in entity text, and improve the accuracy and semantic confidence of knowledge representation.

[0029] Existing multimodal knowledge completion methods neglect fine-grained interactions between entity text and images, failing to achieve deep semantic alignment and fusion between different modalities. This makes it difficult for models to bridge the semantic gap between modalities, thus limiting their ability to represent complex entities. The second technical problem this application aims to solve is how to achieve fine-grained interactions between entity text and images based on key entity information, thereby enhancing the representation ability of multimodal knowledge triples.

[0030] In view of this, one embodiment of this application proposes a multimodal knowledge completion method based on key information extraction and multi-granularity interaction, applicable to multimodal knowledge graphs and applied to electronic devices. The electronic device can be a terminal or a server; this embodiment and the following embodiments will use a server as an example for illustration. The implementation details of the multimodal knowledge completion method based on key information extraction and multi-granularity interaction proposed in this embodiment will be described in detail below. The following implementation details are provided for ease of understanding only and are not essential for implementing this solution.

[0031] The specific process of the multimodal knowledge completion method based on key information extraction and multi-granularity interaction proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 11: Fine-grained key information extraction is performed on the multimodal knowledge completion dataset to obtain keyword information in entity description text and key subgraph information in entity images.

[0032] In practical implementation, the multimodal knowledge completion model can be as follows: Figure 2 As shown, the system is mainly divided into a preprocessing module responsible for fine-grained key information extraction, a visual redundancy feature aggregation module (aggregation module) responsible for data processing and aggregating visual redundancy features, a multi-granularity cross-modal interaction module responsible for key information-guided multi-granularity cross-modal interaction, and an output module for final missing entity prediction. The preprocessing module extracts fine-grained key information from the multi-modal knowledge completion dataset, obtaining keyword information from entity description text and key subgraph information from entity images. The multi-modal knowledge completion dataset contains a large amount of entity description text and entity images.

[0033] The original data for the knowledge completion task is a triple containing entities and relations. The multimodal knowledge completion dataset mainly includes information from two modalities: entity images and entity description text. Multimodal knowledge graphs suffer from modal noise. To denoise the data source, key information needs to be extracted from the original data during preprocessing. This mainly includes keyword information from entity description text and key subgraph information from entity images. The key information extraction process can be as follows: Figure 3 As shown.

[0034] In one example, the preprocessing module first utilizes a Large Language Test (GPT) model to extract keywords related to image attributes from the entity description text based on pre-defined prompts suitable for keyword extraction. Then, it uses BERT to encode the extracted keyword sequence, obtaining keyword embedding vectors as keyword information. The extracted keywords serve two purposes: first, they guide the extraction of key subgraphs; second, the BERT-encoded keyword sequence serves as one of the core inputs to the subsequent multimodal fusion coding model. For entity images, the preprocessing module uses an object detector to extract key subgraph information most relevant to the keywords. The entity image and keywords are then input into the Grounding-DINO model, and a confidence threshold for effective subgraphs is set to ensure the quantity and quality of the extracted key subgraphs.

[0035] Step 12: Based on entity description text, keyword information, and key subgraph information, perform data processing and visual redundancy feature aggregation on the missing triplet data to construct a training sample set.

[0036] In practice, after extracting key information, the missing triplet data can be processed and visual redundancy features can be aggregated based on entity description text, keyword information, and key subgraph information to construct a training sample set.

[0037] Considering that an entity has multiple original image data sets and contains some noise, while the original text data has higher confidence and stronger semantic meaning than images, this embodiment uses the original entity description text as one of the core inputs, and the entity text keywords and entity image key sub-graphs extracted by the preprocessing module as the other two inputs. However, in order to make the data suitable for the multimodal knowledge completion model, a reasonable data generation strategy needs to be designed to construct the model input data samples.

[0038] The generation of training data is related to the model's objective. For the task of predicting missing entities in this embodiment, the traditional task objective is to evaluate the confidence of the input triples. Therefore, the model input needs to have complete triple information. Due to the application of pre-trained models in knowledge completion, a novel training strategy based on masking models has emerged, transforming the original triple scoring task into a multi-classification task. Specifically, an entity is randomly selected from the original triple data and replaced with a mask label to obtain the missing triple data. The goal of the multimodal knowledge completion model is to predict the most likely entity corresponding to the mask label. This masking strategy fully utilizes the "Mask Language Modeling (MLM)" capability of pre-trained Transformers to model the multimodal representation of entities in the knowledge graph by merging images and text, avoiding the situation where all candidate entities are replaced sequentially during the prediction stage, resulting in low computational efficiency. This embodiment divides the experimental process into two stages: pre-training and training, and designs corresponding training data generation strategies for each stage.

[0039] Furthermore, since the initially introduced regional images may contain a lot of repetitive image information, in order to extract more refined key regional image information and integrate redundant visual image information, this embodiment designs a visual redundancy feature aggregation module consisting of clustering and merging strategies, such as... Figure 4 As shown.

[0040] Furthermore, since the initially introduced regional images may contain a lot of repetitive image information, in order to extract more refined key regional image information and integrate redundant visual image information, this embodiment designs a visual redundancy feature aggregation module consisting of clustering and merging strategies, such as... Figure 4 As shown.

[0041] First, the key subgraph information is embedded and encoded using a Vision Transformer as an image encoder to obtain the initial embedding of the key subgraph. , This can be expressed by the formula: , ; in, This indicates the number of patch embedding vectors after the key subgraph has been block-encoded. This represents the dimension of the embedding vector for each patch. Indicates the first Each patch embedding vector The dimension is The space.

[0042] Next, The data is input into clustering units and the K-Means clustering algorithm is used to perform clustering. Perform clustering operations to obtain image embeddings for multiple categories. , This can be expressed by the formula: ; in, Indicates the first The initial set of embeddings for each cluster. Indicates the total number of clusters. This represents the K-Means clustering algorithm.

[0043] Then, a merging unit is used to merge and reduce the dimensionality of the image embeddings of each clustered group through pooling operations, extracting the key sub-image embeddings after integrating redundant visual features. The clustered embedding is denoted as , This can be expressed by the formula: ; in, , This indicates a pooling operation. Indicates the first The number of patch embedding vectors in each cluster. Ultimately, all Clustered embeddings constitute entity image embeddings. , .

[0044] Step 13: Encode the training samples in the training sample set using a text encoder and a visual encoder, and then perform multi-granularity cross-modal interaction guided by key information to obtain text features and multi-modal embedded visual features.

[0045] In the specific implementation, for model training, the training samples in the training sample set are encoded using a text encoder and a visual encoder, and then multi-granular cross-modal interaction guided by key information is performed to obtain text features and multi-modal embedded visual features.

[0046] For the input triplet information to be completed, the original input of the model can be obtained through the above data generation and processing strategies. This input includes multimodal data such as entity images and text. After feature learning and extraction from the input data, the missing entity can be predicted. This embodiment designs a multi-granularity cross-modal interaction module with both global and local granularities, such as... Figure 5 As shown.

[0047] First, the text encoder BERT is used to encode the triplet text sequence and the keyword sequence, and the visual encoder VIT is used to encode the image data, corresponding to... Figure 2The text process and vision process are defined in this embodiment. A natural modal gap exists between multimodal data, which is particularly evident in the data studied in this embodiment. To address this modal gap between text and vision, further cross-modal interaction aggregation is performed on the multimodal data in the triples, establishing a connection between the semantic information of the triples and key modal data, effectively capturing the complex semantic information of the triples. This embodiment proposes a key information-guided multimodal cross-modal interaction fusion module, which mainly utilizes key visual and textual information to effectively interact and aggregate multimodal information to obtain the final encoding result containing entity-related multimodal information.

[0048] In the key information-guided multimodal interaction and aggregation module, the complete text sequence of triples will use the autonomous power module to extract coarse-grained text features. Cross-modal interaction is mainly based on cross-attention, which uses text keyword features as attention guides and performs fine-grained interaction on text features and image features through a multi-level attention architecture.

[0049] In one example, for the attention mechanism, based on the input sequence vector Through the coefficient matrix , , ,Will Mapping to query vector Key vector Value vector In query vector Under the guidance of [the system / mechanism], attention calculation is completed, enabling interaction and fusion between data, and extracting global dependency features of the data. The calculation formula is as follows: ; in, This represents the attention mechanism. Represents the key vector Dimensions .

[0050] Multi-head attention mechanisms allow models to jointly pay attention to subspaces at different locations from different representations by performing attention functions in parallel on multiple heads and then concatenating them: ; ; in, This indicates a multi-head attention mechanism. This represents the multi-head attention coefficient matrix. Indicates the first One head.

[0051] Set the text embedding for calculation The head is Keyword embedding The head is Visual Embedding The head is The multi-level attention calculation process guided by key information is as follows: ; ; Each embedding feature corresponds to a set of coefficient matrices, and the indices are consistent with those of the embedding features. The fine-grained two-layer attention is divided into self-attention, which is mainly based on global triplet embedding. Figure 5 (left side), and multi-level hybrid attention by connecting key-value pairs of global triple embedding and local key image embedding to local keyword embedding ( Figure 5 (On the right side), it can complete fine-grained interactive encoding between multiple modalities under the guidance of keywords, thereby better representing the rich semantic features of triples.

[0052] The formula for the multi-level hybrid attention can be derived as follows: Further expressed as: ; in, Indicates self-attention, Indicates cross attention. It represents the sum of normalized attention weights on the keyword embedding key-value pair vector, equivalent to adaptively adjusted self-attention and cross-attention.

[0053] To address the problem of cross-modal semantic interaction between entity text and images, this embodiment considers in-depth interaction modeling at multiple granularities, including global text, global text and local keywords, and local keywords and local key subgraphs. A cross-modal interaction network based on multi-level hybrid attention is designed. The key is to use keyword information as a guide to lead the model to focus on more critical information under the attention model architecture. This solves the problem that existing technologies have failed to effectively achieve cross-modal interaction between entity text and entity images. It can effectively bridge the semantic gap between text and images, improve the model's ability to represent the semantics of triples, and break through the bottleneck of knowledge completion performance.

[0054] Step 14: Perform cross-modal fusion on text features and multimodal embedded visual features to obtain cross-modal fused features. After combining all cross-modal fused features, use the aggregation FFN layer to merge multimodal features to obtain the final multimodal fused features. Then, perform missing entity prediction based on the final multimodal fused features.

[0055] In the specific implementation, after obtaining the text features and multimodal embedded visual features, cross-modal fusion can be performed on the text features and multimodal embedded visual features to obtain cross-modal fused features. After combining all cross-modal fused features, the multimodal features are merged using the aggregation FFN layer to obtain the final multimodal fused features. Missing entity prediction is then performed based on the final multimodal fused features.

[0056] After encoding using multi-level attention operations (the embedding vector dimension remains unchanged after the attention operation, so it is still used) (This represents the output of the multi-layer attention encoding layer), and the multimodal data can be further perceptually aggregated through Fusion operations. Historical research shows that entity text has better confidence than images in multimodal knowledge graphs. Therefore, in this process, the effectiveness of multimodal embedding visual features is measured by calculating the similarity between text features and multimodal embedded visual features. Each text feature... With all multimodal embedded visual features Similarity is calculated, and weights are obtained through the Softmax function. These weights are then aggregated to obtain cross-modal fusion features. Cross-modal fusion features This can be expressed by the formula: ; in, This represents the Softmax function.

[0057] Similarly, combine all cross-modal fusion features This yields the final Fusion layer output aggregation embedding matrix. : ; in, Indicates the length of the original text sequence. Represents the vector embedding dimension.

[0058] Finally, the aggregation embedding matrix is... The multimodal features are merged into the modified aggregated FFN layer to obtain the final multimodal fused features. The aggregated FFN layer consists of two linear transformations and a ReLU activation function. ; in, Indicates the aggregation of FFN layers. , , , , These are all parameters to be learned. This represents the ReLU activation function.

[0059] The output of the aggregated FFN layer is the final multimodal fusion feature. Then, using a mask prediction strategy, the output vector corresponding to the mask label is fed into the MLP network layer for missing entity prediction. Each possible entity is scored to obtain the entity set probability distribution. The entity with the highest score is the prediction result.

[0060] Step 15: Construct a total loss function based on the missing entity prediction results and the real labels labeled on the training samples. Iterate the training of the multimodal knowledge completion model until convergence based on the total loss function to obtain the trained multimodal knowledge completion model, so as to perform the multimodal knowledge completion task.

[0061] In the specific implementation, after obtaining the prediction results of missing entities, a total loss function can be constructed based on the prediction results of missing entities and the real labels marked on the training samples. The multimodal knowledge completion model is then iteratively trained until convergence based on the total loss function to obtain the trained multimodal knowledge completion model, which is then used to perform the multimodal knowledge completion task.

[0062] During model training, since one-to-many and many-to-one relationships exist in the knowledge graph, the model is treated as a multi-label classification task, and the binary cross-entropy loss function is used as the total loss function. The total loss function is expressed by the formula: ; in, Indicates the number of entity categories. It is a probability distribution. This indicates that the model predicts the missing entity as the first... The probability of a class entity. One-hot encoding of real labels, due to the existence of one-to-many triples, means that the completed positive entity corresponding to each missing triple is not unique. For each candidate entity... If it can be used as a positive example to complete the triplet, then ,otherwise For positive examples, the similarity between the encoded representation of the model triples and the positive samples is maximized.

[0063] This embodiment proposes a multimodal knowledge completion method based on key information extraction and multi-granularity interaction. Addressing the modal noise problem, it utilizes a large language model to generate text keyword sequences, improving the semantic confidence of entity descriptions. Simultaneously, it uses a visual localization model to lock key subgraphs of entities, designs a clustering and merging module to deeply mine and integrate image detail features, eliminate redundant visual noise, and extract fine-grained entity image semantics. To address the modal semantic gap problem, it proposes a key information-guided multi-granularity cross-modal interaction mechanism. Using keyword sequences as semantic bridges, it drives guided multi-granularity interactions between entity text, keyword sequences, and key subgraphs through hybrid attention, achieving deep alignment and fusion of cross-modal semantics and enhancing the model's representation ability of multimodal knowledge triples. In summary, this embodiment obtains high-confidence key image and text information from the data source through a pre-trained model and a fine-grained visual localization model. By designing a key information-guided multi-granularity cross-modal interaction method, it achieves multi-granularity alignment of image and text information, guiding the model to learn entity image features under keyword features, thereby improving the model's entity representation ability and ultimately enhancing knowledge completion performance.

[0064] The steps described above are merely for clarity in describing the technical solution. In actual implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Any insignificant modifications or designs added to the algorithm or process, as long as they do not change the core of the algorithm or process, are also within the scope of protection of this application.

[0065] Another embodiment of this application provides an electronic device, such as Figure 6 As shown, it includes a processor 21 and a memory 22. The memory 22 stores instructions that the processor 21 can execute. When the processor 21 is configured to execute the instructions, the electronic device can realize a multimodal knowledge completion method based on key information extraction and multi-granularity interaction as described in the above method embodiment.

[0066] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0067] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0068] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement a multimodal knowledge completion method based on key information extraction and multi-granularity interaction as described in the above method embodiments.

[0069] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0070] It will be understood by those skilled in the art that the above embodiments are specific implementations of this application, and various changes in form and detail can be made in practical applications without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A multimodal knowledge completion method based on key information extraction and multi-granularity interaction, applicable to multimodal knowledge graphs, characterized in that, The method includes: Fine-grained key information extraction is performed on the multimodal knowledge completion dataset to obtain keyword information in entity description text and key subgraph information in entity images; the multimodal knowledge completion dataset contains a large number of entity description texts and entity images; Based on entity description text, keyword information, and key subgraph information, data processing and visual redundancy feature aggregation are performed on the missing triplet data to construct a training sample set. The training samples in the training sample set are encoded using a text encoder and a visual encoder, and then multi-granular cross-modal interaction guided by key information is performed to obtain text features and multi-modal embedded visual features. Cross-modal fusion is performed on text features and multimodal embedded visual features to obtain cross-modal fusion features. After combining all cross-modal fusion features, multimodal features are merged using an aggregated FFN layer to obtain the final multimodal fusion features. Missing entity prediction is then performed based on the final multimodal fusion features. A total loss function is constructed based on the missing entity prediction results and the real labels annotated on the training samples. The multimodal knowledge completion model is iteratively trained until convergence based on the total loss function to obtain the trained multimodal knowledge completion model, which is then used to perform the multimodal knowledge completion task.

2. The multimodal knowledge completion method based on key information extraction and multi-granularity interaction according to claim 1, characterized in that, Fine-grained key information extraction is performed on the multimodal knowledge completion dataset to obtain keyword information in entity description text and key subgraph information in entity images, including: Using a large language model, based on pre-defined prompts suitable for keyword extraction, keywords related to image attributes are extracted from entity description text. BERT is then used to encode the extracted keyword sequence to obtain keyword embedding vectors, which serve as keyword information. For entity images, an object detector is used to extract key subgraph information most relevant to the keywords. The entity images and keywords are then input into the Grounding-DINO model, and the number and quality of the extracted key subgraphs are ensured by setting an effective subgraph confidence threshold.

3. The multimodal knowledge completion method based on key information extraction and multi-granularity interaction according to claim 1, characterized in that, Before performing data processing and visual redundancy feature aggregation on the missing triplet data based on entity description text, keyword information, and key subgraph information, the method further includes: In the original triplet data, an entity is randomly selected and replaced with a mask label to obtain the missing triplet data. The goal of the multimodal knowledge completion model is to predict the most likely entity corresponding to the mask label.

4. The multimodal knowledge completion method based on key information extraction and multi-granularity interaction according to claim 3, characterized in that, Based on entity description text, keyword information, and key subgraph information, data processing and visual redundancy feature aggregation are performed on missing triplet data, including: The key subgraph information is embedded and encoded using a Vision Transformer as an image encoder to obtain the initial embedding of the key subgraph. , This can be expressed by the formula: ， ； in, This indicates the number of patch embedding vectors after the key subgraph has been block-encoded. This represents the dimension of the embedding vector for each patch. Indicates the first Each patch embedding vector The dimension is space; Will The data is input into clustering units and the K-Means clustering algorithm is used to perform clustering. Perform clustering operations to obtain image embeddings for multiple categories. , This can be expressed by the formula: ； in, Indicates the first The initial set of embeddings for each cluster. This represents the total number of clusters. This represents the K-Means clustering algorithm; Using a merging unit, pooling operations are used to merge and reduce the dimensionality of the image embeddings of each clustered group, extracting key sub-image embeddings after integrating redundant visual features. The clustered embedding is denoted as , represented as: ； in, , This indicates a pooling operation. Indicates the first The number of patch embedding vectors in each cluster, all Clustered embeddings constitute entity image embeddings. , .

5. The multimodal knowledge completion method based on key information extraction and multi-granularity interaction according to claim 1, characterized in that, The training samples in the training sample set are encoded using a text encoder and a visual encoder, and then multi-granularity cross-modal interaction guided by key information is performed to obtain text features and multi-modal embedded visual features, including: A text encoder is used to encode the triplet text sequences and keyword sequences in the training samples, and a visual encoder is used to encode the image data in the training samples. The multi-granularity cross-modal interaction module guided by key information processes the outputs of the text encoder and the visual encoder. The self-attention unit extracts coarse-grained text features, and the cross-attention unit uses text keyword features as attention guides. Through a multi-layered attention architecture, fine-grained interaction is performed between the coarse-grained text features and image features, ultimately obtaining text features and multi-modal embedded visual features.

6. The multimodal knowledge completion method based on key information extraction and multi-granularity interaction according to claim 5, characterized in that, For attention mechanisms, based on the input sequence vector Through the coefficient matrix , , ,Will Mapping to query vector Key vector Value vector In query vector Under the guidance of [the system / mechanism], attention calculation is completed, enabling interaction and fusion between data, and extracting global dependency features of the data. The calculation formula is as follows: ； in, This represents the attention mechanism. Represents the key vector The dimension; Multi-head attention mechanisms allow models to jointly pay attention to subspaces at different locations from different representations by performing attention functions in parallel on multiple heads and then concatenating them: ； ； in, This indicates a multi-head attention mechanism. This represents the multi-head attention coefficient matrix. Indicates the first One head; Set the text embedding for calculation The head is Keyword embedding The head is Visual Embedding The head is The multi-level attention calculation process guided by key information is as follows: ； ； Each embedding feature corresponds to a set of coefficient matrices and is consistent with the embedding feature subscripts. The fine-grained two-layer attention is divided into self-attention based on global triplet embedding and multi-level hybrid attention by connecting the key-value pairs of global triplet embedding and local key image embedding to local keyword embedding. Under the guidance of keywords, fine-grained interactive encoding between multiple modalities can be completed, thereby better representing the rich semantic features of triplets. Further expressed as: ； in, Indicates self-attention, Indicates cross attention. It represents the sum of normalized attention weights on the keyword embedding key-value pair vector, equivalent to adaptively adjusted self-attention and cross-attention.

7. A multimodal knowledge completion method based on key information extraction and multi-granularity interaction according to any one of claims 1 to 6, characterized in that, Cross-modal fusion is performed on text features and multimodal embedded visual features to obtain cross-modal fused features. All cross-modal fused features are combined, and a convergent FFN layer is used to merge the multimodal features to obtain the final multimodal fused features. Missing entity prediction is then performed based on the final multimodal fused features, including: The effectiveness of multimodal embedding visual features is measured by calculating the similarity between text features and multimodal embedding visual features. Each text feature... With all multimodal embedded visual features Similarity is calculated, and weights are obtained through the Softmax function. These weights are then aggregated to obtain cross-modal fusion features. Cross-modal fusion features This can be expressed by the formula: ； in, Represents the Softmax function; Combine all cross-modal fusion features This yields the final aggregate embedding matrix output by the Fusion layer. : ； in, Indicates the length of the original text sequence. Indicates the vector embedding dimension; Finally, the aggregation embedding matrix is... The multimodal features are merged into the modified aggregated FFN layer to obtain the final multimodal fused features. The aggregated FFN layer consists of two linear transformations and a ReLU activation function. ； in, Indicates the aggregation of FFN layers. , , , , These are all parameters to be learned. Represents the ReLU activation function; The output of the aggregated FFN layer is the final multimodal fusion feature. Then, using a mask prediction strategy, the output vector corresponding to the mask label is fed into the MLP network layer for missing entity prediction. Each possible entity is scored to obtain the entity set probability distribution. The entity with the highest score is the prediction result.

8. The multimodal knowledge completion method based on key information extraction and multi-granularity interaction according to claim 7, characterized in that, During model training, since one-to-many and many-to-one relationships exist in the knowledge graph, the model is treated as a multi-label classification task, and the binary cross-entropy loss function is used as the total loss function. The total loss function is expressed by the formula: ； in, Indicates the number of entity categories. It is a probability distribution. This indicates that the model predicts the missing entity as the first... The probability of a class entity. One-hot encoding of real labels, due to the existence of one-to-many triples, means that the completed positive entity corresponding to each missing triple is not unique. For each candidate entity... If it can be used as a positive example to complete the triplet, then ,otherwise For positive examples, the similarity between the encoded representation of the model triples and the positive samples is maximized.

9. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores instructions that the processor can execute, and the processor is configured to, when executing the instructions, enable the electronic device to implement a multimodal knowledge completion method based on key information extraction and multi-granularity interaction as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a multimodal knowledge completion method based on key information extraction and multi-granularity interaction as described in any one of claims 1 to 8.

Images