Method for aligning multi-modal knowledge graph entities based on neighbor perception modal generation

Abstract

The application discloses a kind of multi-modal knowledge graph entity alignment methods based on neighbor perception modal generation, belong to natural language processing and knowledge graph technical field.The application designs neighbor perception generator, aggregates the visual distribution of entity multi-hop neighbor as prior by graph attention network, and combines its own complete relationship and attribute modal as condition, utilizes conditional generation adversarial network to semantically reconstruct missing image features, effectively alleviate the uncertainty in modal completion.Secondly, introduce modal feature selection strategy, dynamically evaluate the correlation of each modal feature and alignment task during training process, freeze the gradient update of low correlation feature, so that the model adaptively focuses on high-value, high-credibility modal information for fusion.The application significantly improves the robustness and accuracy of entity alignment in the scene where image information is severely incomplete or quality is uneven through context-aware modal generation and intelligent feature screening.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and knowledge graph technology, specifically to a method for aligning entities in a multimodal knowledge graph based on neighbor-aware modality generation. Background Technology

[0002] Multimodal knowledge graphs, by deeply integrating multi-source information such as images and text on top of traditional symbolic triples, provide richer semantic representations for machines to understand the real world. Multimodal entity alignment, as a core technology for integrating multi-source multimodal knowledge graphs, aims to identify entities in different graphs that refer to the same real-world object, and is of great significance for advanced applications such as cross-modal search, intelligent question answering, and personalized recommendations.

[0003] However, real-world multimodal knowledge graphs commonly suffer from large-scale missing image modalities. Due to differences in data acquisition sources, copyright restrictions, or cultural variations, entity images are often missing or of uneven quality in one or both knowledge graphs. This asymmetric image loss not only reduces usable information but also undermines the complementarity between multimodalities, leading to a significant performance degradation of existing alignment methods.

[0004] To address image missingness, one of the mainstream paradigms is cross-modal generation and reconstruction. These methods (such as those based on variational autoencoders or generative adversarial networks) attempt to reconstruct the representation of missing images using the existing modalities of entities. However, these methods still have the following limitations: (1) Reconstruction relies solely on cross-modal statistical correlations, failing to fully utilize the structural contextual information of entities in the graph (such as the visual distribution of multi-hop neighbors), resulting in blurred, prototype-like, or even deviating features from the true visual semantics; (2) The use of a uniform reconstruction or filling mechanism lacks adaptive adjustment to the severity of entity missingness, leading to the introduction of noise rather than effective information into the reconstructed features.

[0005] To address the aforementioned issues, this invention proposes a multimodal knowledge graph entity alignment method based on neighbor-aware modality generation. Through structure-guided visual feature completion and dynamic modal feature selection, it significantly improves the alignment robustness and accuracy in scenarios with missing images. Summary of the Invention

[0006] This invention provides a method for aligning entities in a multimodal knowledge graph based on neighbor-aware modality generation to alleviate the uncertainty in modality reconstruction caused by missing image modalities.

[0007] The technical solution of this invention is: a multimodal knowledge graph entity alignment method based on neighbor-aware modality generation, the method comprising:

[0008] Step 1: Construct a multimodal encoder to independently encode the structural modality, relational modality, attribute modality, and image modality in the input multimodal knowledge graph, and obtain the initial feature representation of each modality;

[0009] Step 2: Design a neighbor perception generator. Use a graph attention network to aggregate the visual feature distribution of the target entity's multi-hop neighbors in the knowledge graph to generate a structure-enhanced visual representation. Combine the existing relation modality features and attribute modality features of the target entity to construct a multimodal conditional vector. Use the structure-enhanced visual representation and the multimodal conditional vector as input to a conditional generative adversarial network to perform semantic reconstruction of the missing image modality features.

[0010] Step 3: Introduce a modal feature selection strategy. During model training, dynamically calculate the cross-graph alignment correlation score of each entity in different modalities. Freeze the gradient update of low-correlation modal features based on the alignment correlation score, and perform weighted fusion of the unfrozen modal features according to their alignment correlation scores to obtain the joint representation of the entities.

[0011] Step 4: Based on the joint representation and reconstructed image modal features, the model is trained end-to-end using the cross-knowledge graph alignment loss function, and the trained model is used to perform alignment prediction of entities in the target knowledge graph.

[0012] Furthermore, Step 1 specifically includes the following:

[0013] Step 1.1, Structural Modality Encoding: A Graph Attention Network (GAT) is used to encode the adjacency matrix and randomly initialized entity representations of the knowledge graph. Multi-head attention mechanism is used to aggregate multi-hop neighbor information to obtain structural feature representations. The calculation formula is:

[0014]

[0015] in It is a physical entity The random initialization representation, It is the adjacency matrix of the knowledge graph. This is a learnable weight matrix.

[0016] Step 1.2, Relational Modal Encoding: A fully connected layer is used to transform the bag-of-words features of the entity's relations to obtain the relational feature representation. ;

[0017] Step 1.3, Attribute Modality Encoding: A fully connected layer is used to transform the bag-of-words features of the entity's attributes to obtain the attribute feature representation. ;

[0018] Step 1.4, Image Modality Coding: A pre-trained deep convolutional neural network (such as VGG-16) is used to extract the initial visual features of the entity association image to obtain the image feature representation. The missing parts are filled with zero vectors;

[0019] The relational modality, attribute modality, and image modality are all transformed using independent fully connected layers to avoid information pollution caused by mixing different modal information during the encoding stage.

[0020] Furthermore, Step 2 specifically includes the following:

[0021] Step 2.1, Neighborhood Information Aggregation: For target entities lacking image features, a graph attention network is used to aggregate the visual features of their L-hop neighbors to generate a structure-enhanced visual representation. The calculation formula is as follows:

[0022]

[0023] in For learnable weight matrix, Let be the adjacency matrix of the knowledge graph. The initial image features are filled with zero vectors to fill in the missing parts;

[0024] Step 2.2, Multimodal Conditional Vector Construction: Enhancing the visual representation of the structure. Relationship characteristics and attribute features Multimodal conditional vectors are generated by fusing data through a multilayer perceptron.

[0025]

[0026] in This indicates a vector concatenation operation.

[0027] Step 2.3, Conditional Generative Adversarial Network Reconstruction: Visual Representation Enhanced by the Structure and multimodal condition vector As input to the conditional generator, pseudo-image features are generated. Simultaneously, a discriminator is set up to distinguish between real image features and generated features, and to evaluate the semantic consistency between generated features and conditional vectors.

[0028] Furthermore, Step 3 specifically includes the following:

[0029] Step 3.1, Alignment Relevance Score Calculation: For each entity and each modal feature, calculate its maximum cosine similarity between the source and target knowledge graphs, and normalize it using a dynamic threshold to obtain the alignment relevance score. The calculation formula is:

[0030]

[0031] in , Dynamic threshold;

[0032] Step 3.2 Gradient Freeze: Truncate the gradients of modal features with alignment correlation scores of zero or below a preset threshold during backpropagation and stop their parameter updates;

[0033] Step 3.3, Weighted Fusion: For the modal features that are not frozen, a weighted sum is performed using their alignment correlation scores as weights to generate a joint representation of the entity.

[0034] Furthermore, Step 4 specifically includes the following:

[0035] Step 4.1: Adversarial loss is used to train the generator and discriminator, and its expression is:

[0036]

[0037] Step 4.2, Seed Alignment Loss: Utilizing known pre-aligned entity pairs, the loss function narrows the representation distance between them in each modality, including structural modality, relational modality, attribute modality, image modality, and the fused joint modality. The loss function is defined as:

[0038]

[0039] in For a pre-aligned set of entity pairs, Entities The structure, relationships, attributes, and image feature representations. The joint representation generated in Step 3.3.

[0040] The present invention also provides a multimodal knowledge graph entity alignment system based on neighbor-aware modality generation, the system comprising: a module for executing the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation.

[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation.

[0044] The beneficial effects of this invention are:

[0045] This invention proposes a multimodal knowledge graph entity alignment method based on neighbor-aware modality generation. By leveraging the graph structure context to guide semantic reconstruction of missing image features through a neighbor-aware generator, the uncertainty of modality completion is significantly reduced. Furthermore, a modality feature selection strategy dynamically focuses on high-value modality information, suppressing noise interference. Experiments on four public datasets—MMKG, OpenEA, and FB-YAGO15K—demonstrate stable performance even with image missing ratios as high as 80%. On the highly heterogeneous FB-YAGO15K dataset, Hits@1 reaches 61.1%, a 5.8 percentage point improvement over the current state-of-the-art method, PMF-MMEA, and the MRR reaches 0.732, a 4.9 percentage point improvement, validating its strong robustness and alignment accuracy in image missing scenarios. Attached Figure Description

[0046] Figure 1 This is a diagram illustrating the overall framework of the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation in this invention. Detailed Implementation

[0047] The method proposed in this embodiment was implemented on four publicly available multimodal entity alignment benchmark datasets: MMKG (FB-DB15K, FB-YAGO15K) and OpenEA (EN-FR-15K V2, EN-DE-15K V2).

[0048] The MMKG and OpenEA datasets are constructed based on multiple knowledge graphs, including FB15K, DB15K, YAGO15K, and DBP15K. These knowledge graphs contain structured triples (entity, relation, attribute) and corresponding image information for each entity. The datasets provide entity alignment pairs between different knowledge graphs through manual annotation or cross-language linking. Statistical information for each dataset is shown in Table 1.

[0049] Table 1. Statistical Description of the Dataset

[0050] like Figure 1 As shown, a multimodal knowledge graph entity alignment method based on neighbor-aware modality generation includes:

[0051] Step 1: Construct a multimodal encoder to independently encode the structural modality, relational modality, attribute modality, and image modality in the input multimodal knowledge graph, and obtain the initial feature representation of each modality;

[0052] Step 2: Design a neighbor perception generator. Use a graph attention network to aggregate the visual feature distribution of the target entity's multi-hop neighbors in the knowledge graph to generate a structure-enhanced visual representation. Combine the existing relation modality features and attribute modality features of the target entity to construct a multimodal conditional vector. Use the structure-enhanced visual representation and the multimodal conditional vector as input to a conditional generative adversarial network to perform semantic reconstruction of the missing image modality features.

[0053] Step 3: Introduce a modal feature selection strategy. During model training, dynamically calculate the cross-graph alignment correlation score of each entity in different modalities. Freeze the gradient update of low-correlation modal features based on the alignment correlation score, and perform weighted fusion of the unfrozen modal features according to their alignment correlation scores to obtain the joint representation of the entities.

[0054] Step 4: Based on the joint representation and reconstructed image modal features, the model is trained end-to-end using the cross-knowledge graph alignment loss function, and the trained model is used to perform alignment prediction of entities in the target knowledge graph.

[0055] Furthermore, Step 1 specifically includes the following:

[0056] Step 1.1, Structural Modality Encoding: A Graph Attention Network (GAT) is used to encode the adjacency matrix and randomly initialized entity representations of the knowledge graph. Multi-head attention mechanism is used to aggregate multi-hop neighbor information to obtain structural feature representations. The calculation formula is:

[0057]

[0058] in It is a physical entity The random initialization representation, It is the adjacency matrix of the knowledge graph. This is a learnable weight matrix.

[0059] Step 1.2, Relational Modality Encoding: A fully connected layer is used to perform linear transformation and dimensionality mapping on the bag-of-words features of entity relations, projecting the high-dimensional sparse relation features into a low-dimensional dense space to obtain the relation feature representation. The calculation formula is:

[0060]

[0061] in For entities Bag-of-relations features A fully connected layer specifically for relational modalities.

[0062] Step 1.3, Attribute Modality Encoding: A fully connected layer is used to perform a nonlinear transformation on the bag-of-words features of the entity's attributes to extract the semantic information of the attributes and obtain the attribute feature representation. The calculation formula is:

[0063]

[0064] in For entities Bag-of-attributes feature, A fully connected layer specifically for attribute modalities;

[0065] Step 1.4, Image Modality Coding: A pre-trained deep convolutional neural network (such as VGG-16) is used to extract the initial visual features of the entity association image to obtain the image feature representation. For entities missing in the image, they are filled with zero vectors, calculated using the following formula:

[0066]

[0067] in For entities The corresponding image data, This is a pre-trained visual feature extraction network. The relational modality, attribute modality, and image modality are all transformed using independent fully connected layers to avoid information contamination caused by mixing different modal information during the encoding stage.

[0068] The relational modality, attribute modality, and image modality are all transformed using independent fully connected layers to avoid information pollution caused by mixing different modal information during the encoding stage.

[0069] Furthermore, Step 2 specifically includes the following:

[0070] Step 2.1, Neighborhood Information Aggregation: For target entities lacking image features, a graph attention network is used to aggregate the visual features of their L-hop neighbors to generate a structure-enhanced visual representation. The calculation formula is as follows:

[0071]

[0072] in For learnable weight matrix, Let be the adjacency matrix of the knowledge graph. The initial image features are filled with zero vectors to fill in the missing parts;

[0073] Step 2.2, Multimodal Conditional Vector Construction: Enhancing the visual representation of the structure. Relationship characteristics and attribute features Multimodal conditional vectors are generated by fusing data through a multilayer perceptron.

[0074]

[0075] in This indicates a vector concatenation operation.

[0076] Step 2.3, Conditional Generative Adversarial Network Reconstruction: Visual Representation Enhanced by the Structure and multimodal condition vector As input to the conditional generator, pseudo-image features are generated. Simultaneously, a discriminator is set up to distinguish between real image features and generated features, and to evaluate the semantic consistency between generated features and conditional vectors.

[0077] Furthermore, Step 3 specifically includes the following:

[0078] Step 3.1, Alignment Relevance Score Calculation: For each entity and each modal feature, calculate its maximum cosine similarity between the source and target knowledge graphs, and normalize it using a dynamic threshold to obtain the alignment relevance score. The calculation formula is:

[0079]

[0080] in , Dynamic threshold;

[0081] Step 3.2 Gradient Freeze: Truncate the gradients of modal features with alignment correlation scores of zero or below a preset threshold during backpropagation and stop their parameter updates;

[0082] Step 3.3, Weighted Fusion: For the modal features that are not frozen, a weighted sum is performed using their alignment correlation scores as weights to generate a joint representation of the entity.

[0083] Step 4: Based on the joint representation and reconstructed image modal features, the model is trained end-to-end using the cross-knowledge graph alignment loss function, and the trained model is used to perform alignment prediction of entities in the target knowledge graph.

[0084] Furthermore, Step 4 specifically includes the following:

[0085] Step 4.1: Adversarial loss is used to train the generator and discriminator to ensure that the generated feature distribution is as close as possible to the real image feature distribution. Its expression is:

[0086]

[0087] in, For discriminator, For generator, For real image features, For structurally enhanced visual representation, This is the multimodal condition vector.

[0088] Step 4.2: Seed Alignment Loss utilizes the known pre-aligned entity pair set. This reduces the representational distance between them in each modality, including structural modality, relational modality, attribute modality, image modality, and the fused joint modality; the loss function is defined as:

[0089]

[0090] in, Entities The structure, relationships, attributes, and image feature representations. The joint representation generated in Step 3.3.

[0091] Step 4.3: The model's total loss function is a weighted sum of the adversarial loss and the seed alignment loss.

[0092]

[0093] in To balance the hyperparameters, a value of 0.1 is used in this invention. During training, the Adam optimizer is used to update the model parameters, with an initial learning rate of 0.001 and a batch size of 512.

[0094] The present invention also provides a multimodal knowledge graph entity alignment system based on neighbor-aware modality generation, the system comprising: a module for executing the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation.

[0095] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation.

[0096] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation.

[0097] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation.

[0098] Furthermore, the present invention constructs a training set, a validation set, and a test set, and employs an evaluation protocol based on nearest neighbor search to train and evaluate the model on a multimodal entity alignment task.

[0099] Furthermore, the present invention specifically includes the following:

[0100] (1) In this invention, the pre-labeled entity alignment pairs in each benchmark dataset are randomly divided into a training set (alignment seed), a validation set, and a test set according to a ratio of 30%, 10%, and 60%. The distribution of different datasets is kept consistent during the partitioning process, and the average result of 5 random partitions is reported. The training set is used for loss function optimization in Step 2, the validation set is used for hyperparameter tuning and early stopping, and the test set is used for final performance evaluation.

[0101] (2) During model training, an early stopping mechanism is used to prevent overfitting. Specifically, after each training epoch, the Hits@1 metric is calculated on the validation set. If the Hits@1 metric on the validation set does not improve for 20 consecutive epochs, training is stopped, and the model parameters with the best performance on the validation set are saved.

[0102] (3) After the model training is completed, alignment prediction is performed on the test set. The prediction stage uses a distance-based nearest neighbor search method: for the source map Each entity in Calculate its relationship with the target map The embedding distances of all entities in the dataset are calculated and sorted in ascending order. The entity ranked highest is the predicted alignment target. This process can be represented as:

[0103]

[0104] Furthermore, the present invention also employs multiple evaluation metrics to quantitatively assess the entity alignment results, including Hits@k, average reciprocal ranking, and average ranking.

[0105] Furthermore, the present invention specifically includes the following:

[0106] (1) This invention uses standard evaluation metrics in the field of entity alignment to evaluate the model's prediction performance. Assume the test set contains... Each source entity, for each source entity The correctly aligned target entity is The model predicts the ranking as The definitions of each indicator are as follows:

[0107] Hits@k: ranked highest The proportion of correctly aligned entities within a certain number is calculated using the following formula:

[0108]

[0109] in, For indicator functions, The values ​​are usually 1, 5, or 10.

[0110] Mean Reciprocal Rank (MRR): The average of the reciprocals of the correct entity rankings, calculated using the following formula:

[0111]

[0112] Rank Average (MR): The arithmetic mean of the correct entity rankings, calculated using the following formula:

[0113]

[0114] (2) In order to evaluate the comprehensiveness and fairness of the results, this invention reports the source spectrum to the target spectrum simultaneously. From target map to source map The evaluation metrics were calculated in two directions, and their average was taken as the final result. The final output of the experiment was the Hits@1, Hits@5, Hits@10, MRR, and MR metrics on each benchmark dataset.

[0115] Evaluation metrics: This study used Hits@1 as the primary evaluation metric, which directly reflects the model's ability to accurately align entities. Hits@5, Hits@10, MRR, and MR were also used to comprehensively measure the model's alignment performance under different levels of strictness. All experiments were run independently five times, and the mean ± standard deviation was reported. Paired t-tests (p < 0.05) were used to determine the significance of differences between models.

[0116] To verify the effectiveness of the model proposed in this invention, the method of this invention is compared with the following state-of-the-art baseline methods in the field of multimodal knowledge graph entity alignment, as follows:

[0117] EVA: It uses a simple concatenation method to fuse features and discovers new entity alignment pairs based on image similarity, which can maintain good alignment performance in scenarios where there is a lack of labeled alignment data.

[0118] MCLEA calculates the weights between modalities at the graph level, fuses features by weighted concatenation, and optimizes entity representation by constructing positive and negative sample pairs of aligned and unaligned entities, combined with a modal contrastive learning framework, effectively enhancing the model's discriminative ability in multimodal environments.

[0119] MSNEA: Introduces image information into TransE to guide the representation learning of entity attribute and relation modalities. By incorporating image features into attribute and relation modalities during representation, it enhances the expressive power of weak modalities in multimodal alignment.

[0120] MEAformer: Adopting the Transformer architecture, it dynamically models the importance weights of each modality at the entity level, realizing adaptive fusion of multimodal information and effectively improving representation quality and alignment performance.

[0121] UMAEA: To address the problem of missing images, a mechanism was designed to complete entity representations using fused representations, and modal weights were modeled at both the graph level and the entity level to enhance the robustness of the model in the case of incomplete modalities.

[0122] OTMEA: It combines entity-level attention weights to achieve fine-grained fusion and solves the modal heterogeneity problem in multimodal entity alignment through optimal transmission technology.

[0123] GSIEA: By using graph structure prefix injection and convolution enhancement, it extracts fine-grained local features and reduces modal noise, thereby mitigating the noise problem caused by heterogeneous multimodal data.

[0124] PMF-MMEA: It adopts a progressive modal freezing mechanism, which gradually freezes the parameter updates of features that are not related to alignment, and then uses an attention mechanism to weighted fuse features that are related to alignment, thereby improving the modal fusion effect and alignment performance.

[0125] Table 2 compares the overall performance of the proposed method with different baseline models on the multimodal entity alignment task (Hits@1 / Hits@10 / MRR).

[0126] The proposed method was effectively compared on four multimodal knowledge graph entity alignment benchmark datasets: FB-DB15K, FB-YAGO15K, EN-FR-15K V2, and EN-DE-15K V2. To verify the effectiveness of the proposed model, we compared it with several state-of-the-art baseline models, including EVA, MCLEA, MSNEA, MEAformer, UMAEA, OTMEA, GSIEA, and PMF-MMEA. The specific results are shown in Table 2, yielding the following key conclusions: First, the proposed model achieved state-of-the-art performance on all four datasets, significantly outperforming existing multimodal alignment methods. Second, compared to methods like EVA and MCLEA that employ simple concatenation or fixed-weighted fusion, the proposed method, along with methods like PMF-MMEA that utilize dynamic modality selection strategies, performs better, verifying the importance of dynamically adjusting modality weights at the entity level for improving alignment performance. In comparison with the current best model, PMF-MMEA, the proposed method achieved significant performance improvements on multiple datasets. Specifically, it achieves a 6.5 percentage point improvement on the Hits@1 metric of the FB-DB15K dataset and a 6.3 percentage point improvement on the more challenging cross-linguistic EN-DE-15K V2 dataset. Finally, the superior performance of the proposed model can be attributed to the following factors. Firstly, by designing a neighbor-aware generator (Step 2), which utilizes a graph attention network to aggregate the visual feature distribution of multi-hop neighbors, semantic reconstruction of missing image modalities is possible, thus alleviating the problem of incomplete images in multimodal knowledge graphs. Secondly, by introducing a modality feature selection strategy (Step 3), the alignment relevance score of each entity in each modality is dynamically calculated during training, the gradient updates of low-relevance modalities are frozen, and weighted fusion of effective modalities is performed, significantly reducing the interference of modal noise and further improving the accuracy and robustness of entity alignment.

[0127] Table 3 presents the experimental results under the robustness test scenario with 50% image modality loss. The results show that among existing mainstream models, UMAEA and PMF-MMEA employ specialized completion or freezing mechanisms for image loss problems, thus performing better than simpler methods like EVA when modalities are incomplete. PMF-MMEA, through progressive modal freezing, maintains relatively good alignment performance even with 50% image loss. However, these methods primarily rely on the entity's remaining modal information or global modal weights, failing to fully utilize the visual context provided by neighboring entities. The model proposed in this invention actively aggregates the visual feature distributions of multi-hop neighbors around the target entity through the neighbor-aware generator in Step 2, and combines this with multimodal conditional vectors to generate semantically consistent pseudo-image features, thereby achieving high-quality entity representation even with large-scale image loss. Experimental data show that, under the condition that 50% of the images in the FB-DB15K dataset are missing, the Hits@1 of the method of this invention reaches 53.4%, which is 6.7 percentage points higher than that of PMF-MMEA (46.7%), and the relative improvement is 14.3%, which fully demonstrates the superior robustness of the present invention in modal incomplete scenarios.

[0128] Table 3 presents the experimental results for the MATE task on a few-shot multimodal dataset. The results show that among existing mainstream models, VLP performs best on the MATE task, but it relies on additional data and multiple pre-training tasks, deviating from its initial goal of being based on low-resource data. Similarly, JML also improves performance by introducing additional data. VLP's superior performance on the MATE task compared to NVLP may be due to the fact that VLP's pre-training tasks are more aligned with the MATE task, thus improving model performance. The model proposed in this chapter utilizes contextual examples to stimulate the few-shot learning ability of large models and introduces aspect-based background information, providing additional contextual support for the large language model to extract aspect boundaries during fine-tuning. Simultaneously, constructing an aspect preference set further optimizes the aspect output of the large model, improving the performance of the MATE task.

[0129] Table 3. Comparison of alignment performance between the method of this invention and different baseline models under the condition of 50% image modality loss (Hits@1)

[0130] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for aligning entities in a multimodal knowledge graph based on neighbor-aware modality generation, characterized in that, The method includes the following steps: Step 1: Construct a multimodal encoder to independently encode the structural modality, relational modality, attribute modality, and image modality in the input multimodal knowledge graph, and obtain the initial feature representation of each modality; Step 2: Design a neighbor perception generator. Use a graph attention network to aggregate the visual feature distribution of the target entity's multi-hop neighbors in the knowledge graph to generate a structure-enhanced visual representation. Combine the existing relation modality features and attribute modality features of the target entity to construct a multimodal conditional vector. Use the structure-enhanced visual representation and the multimodal conditional vector as input to a conditional generative adversarial network to perform semantic reconstruction of the missing image modality features. Step 3: Introduce a modal feature selection strategy. During model training, dynamically calculate the cross-graph alignment correlation score of each entity in different modalities. Freeze the gradient update of low-correlation modal features based on the alignment correlation score, and perform weighted fusion of the unfrozen modal features according to their alignment correlation scores to obtain the joint representation of the entities. Step 4: Based on the joint representation and reconstructed image modal features, the model is trained end-to-end using the cross-knowledge graph alignment loss function, and the trained model is used to perform alignment prediction of entities in the target knowledge graph.

2. The multimodal knowledge graph entity alignment method based on neighbor-aware modality generation according to claim 1, characterized in that: Step 1 specifically includes the following: Step 1.1, Structural Modal Encoding: A graph attention network is used to encode the adjacency matrix and randomly initialized entity representations of the knowledge graph. Multi-head attention mechanism is used to aggregate multi-hop neighbor information to obtain structural feature representations. Step 1.2, Relational Modal Encoding: A fully connected layer is used to transform the bag-of-words features of the entity's relations to obtain the relational feature representation; Step 1.3, Attribute Modal Encoding: A fully connected layer is used to transform the bag-of-words features of the entity's attributes to obtain the attribute feature representation; Step 1.4, Image Modal Coding: A pre-trained deep convolutional neural network is used to extract the initial visual features of the entity association image to obtain the image feature representation; The relational modality, attribute modality, and image modality are all transformed using independent fully connected layers to avoid information pollution caused by mixing different modal information during the encoding stage.

3. The multimodal knowledge graph entity alignment method based on neighbor-aware modality generation according to claim 1, characterized in that: Step 2 specifically includes the following: Step 2.1, Neighborhood Information Aggregation: For target entities lacking image features, a graph attention network is used to aggregate the visual features of their L-hop neighbors to generate a structure-enhanced visual representation. The calculation formula is as follows: ； in For learnable weight matrix, Let be the adjacency matrix of the knowledge graph. The initial image features are filled with zero vectors to fill in the missing parts; Step 2.2, Multimodal Conditional Vector Construction: Enhancing the visual representation of the structure. Relationship characteristics and attribute features Multimodal conditional vectors are generated by fusion using a multilayer perceptron. ; Step 2.3, Conditional Generative Adversarial Network Reconstruction: Visual Representation Enhanced by the Structure and multimodal condition vector As input to the conditional generator, pseudo-image features are generated. Simultaneously, a discriminator is set up to distinguish between real image features and generated features, and to evaluate the semantic consistency between generated features and conditional vectors.

4. The multimodal knowledge graph entity alignment method based on neighbor-aware modality generation according to claim 1, characterized in that: Step 3 specifically includes the following: Step 3.1, Alignment Relevance Score Calculation: For each entity and each modal feature, calculate its maximum cosine similarity between the source and target knowledge graphs, and normalize it using a dynamic threshold to obtain the alignment relevance score. The calculation formula is: ； in , Dynamic threshold; Step 3.2 Gradient Freeze: Truncate the gradients of modal features with alignment correlation scores of zero or below a preset threshold during backpropagation and stop their parameter updates; Step 3.3, Weighted Fusion: For the modal features that are not frozen, a weighted sum is performed using their alignment correlation scores as weights to generate a joint representation of the entity.

5. The multimodal knowledge graph entity alignment method based on neighbor-aware modality generation according to claim 1, characterized in that: Step 4 specifically includes the following: Step 4.1: Adversarial loss is used to train the generator and discriminator, and its expression is: ； Step 4.2 Seed Alignment Loss: Using known pre-aligned entity pairs, the representation distance between them is reduced in each modality, including structural modality, relational modality, attribute modality, image modality, and fused joint modality.

6. A multimodal knowledge graph entity alignment system based on neighbor-aware modality generation, characterized in that, The system includes a module for performing the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation as described in any one of claims 1 to 4.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation as described in any one of claims 1 to 4.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation as described in any one of claims 1 to 4.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal knowledge graph entity alignment method based on neighbor-aware modality generation as described in any one of claims 1 to 4.

Images