A Semi-Supervised Multimodal Entity Alignment Method

Abstract

The present invention discloses a semi-supervised multi-modal entity alignment method, which relates to the technical field of knowledge graphs. It is characterized by the following steps: S1: Symbolic definition of the multi-modal entity alignment task; S2: Constructing a semi-supervised multi-modal entity alignment model; S21: The multi-modal knowledge feature embedding module extracts heterogeneous features through a heterogeneous encoder; S22: The cross-modal fusion module uses the cross-attention mechanism to model the complementarity and correlation between modalities and generates a joint representation; S23: The pseudo-label generation and momentum contrast learning module screens high-confidence pseudo-labels based on graph label propagation and enhances the stability of contrast learning by combining a momentum queue. The technical problem to be solved by the present invention is to provide a semi-supervised multi-modal entity alignment method. Multi-modal entity alignment aims to identify and connect information representing the same entity in different multi-modal knowledge graphs.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge graphs, and more specifically, to a semi-supervised multi-modal entity alignment method. Background Art

[0002] Knowledge Graph (KG), as a structured knowledge representation form, has been widely applied in tasks such as natural language processing, recommendation systems, and intelligent question answering. KG provides strong support for knowledge reasoning, information retrieval, and question answering by organizing entities (such as people, places, events, etc.) and their relationships. However, traditional knowledge graphs usually only contain single-modal information, such as text or structured data, and it is difficult to handle complex semantic associations involving multiple modalities (such as images, audio, video, etc.).

[0003] To make up for this deficiency, multi-modal knowledge graphs (MMKG) enhance the richness of entity representations by integrating information from different modalities (such as text, images, relationships, etc.). However, a single multi-modal knowledge graph still faces problems such as incomplete data coverage and uneven modal information. For example, the image or attribute information of some entities may be missing, resulting in challenges in cross-modal information integration. Therefore, the multi-modal entity alignment (MMEA) task has emerged, and its goal is to align equivalent entities between different multi-modal knowledge graphs to make up for the information deficiency of a single graph and build a more comprehensive multi-modal knowledge base.

[0004] Recent research has shown that introducing the visual modality can effectively improve the performance of the MMEA task. For example, the MMEA model achieves entity alignment by jointly using image and numerical features, further leveraging the role of the visual modality in long-tail entity matching, and proposing an unsupervised iterative learning strategy. However, the above existing research uses simple concatenation or weighted fusion methods for feature fusion, ignoring the relative importance between modalities. In addition, they mainly rely on the role of labeled samples while ignoring the role of unlabeled samples in alignment.

[0005] Entity alignment aims to identify objects that describe the same entity in different knowledge graphs, thereby achieving knowledge fusion. Traditional methods can be divided into two categories: (1) translation-based models (such as TransE), which learn entity embeddings through vector space translation assumptions; (2) graph neural network (GNN)-based models (such as GCN, GAT), which use structural information to enhance representation consistency.

[0006] With the explosion of multi-modal data, researchers have introduced visual and other means to improve alignment performance. For example, MCLEA optimizes cross-modal interaction through contrastive learning, while DFMKE designs a bidirectional fusion framework that combines early and late fusion strategies. However, existing methods still face challenges such as modal noise correlation and dynamic multi-modal modeling.

[0007] To reduce the dependence on labeled data, semi-supervised methods have emerged. IPTransE generates pseudo-labels through threshold-based filtering, but there is a risk of introducing noise; MRAEA adopts meta-relation-aware alignment but ignores potential matches between low-similarity entities; RAC integrates active learning to select high-value samples but requires manual intervention. Recent work (e.g., PCMEA) has mitigated the impact of noisy labels through momentum-based contrastive learning and pseudo-label calibration, providing new insights for semi-supervised EA. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a semi-supervised multi-modal entity alignment method. Multi-modal entity alignment aims to identify and connect information representing the same entity in different multi-modal knowledge graphs. Aiming at the problems of insufficient utilization of multi-modal information, challenges of modal heterogeneity, and single fusion strategy in existing methods, this paper proposes a semi-supervised multi-modal entity alignment model (Cross-modal Feature Fusion and Momentum Contrastive Learning for Semi-supervised Multi-modal Entity Alignment, CMMEA). This method combines cross-modal feature modeling and momentum contrastive learning to improve the alignment accuracy. First, heterogeneous encoders are used to extract features of each modality, and the complementarity and correlation between modalities are co-modeled in the fusion stage to enhance feature representation. Second, a pseudo-label iterative optimization mechanism is designed and combined with momentum contrastive learning to fully exploit labeled and unlabeled data and improve alignment robustness. Finally, experimental results on two multi-modal entity alignment benchmark datasets show that CMMEA has achieved significant performance improvement in the entity alignment task.

[0009] The present invention adopts the following technical solutions to achieve the invention purpose:

[0010] A semi-supervised multi-modal entity alignment method, characterized by comprising the following steps:

[0011] S1: Symbolic definition of the multi-modal entity alignment task;

[0012] S2: Construct a semi-supervised multi-modal entity alignment model;

[0013] S21: The multi-modal knowledge feature embedding module extracts heterogeneous features through heterogeneous encoders;

[0014] S22: The cross-modal fusion module models the complementarity and correlation between modalities using the cross-attention mechanism to generate joint representations;

[0015] S23: The pseudo-label generation and momentum contrast learning module filters high-confidence pseudo-labels based on graph label propagation and enhances the stability of contrast learning by combining a momentum queue.

[0016] As a further limitation of this technical solution, the multi-modal knowledge graph is defined as: , where: respectively represent entities, relationships, attributes, and visuals;

[0017] Given two multi-modal knowledge graphs: the source knowledge graph and the target knowledge graph , the alignment seed set across the two multi-modal knowledge graphs is defined as: , where: represents and are equivalent entities referring to the same real-world object.

[0018] As a further limitation of this technical solution, the specific steps of S21 are as follows:

[0019] S211: Structure embedding;

[0020] Use a graph attention network to model the structural information of and , effectively extracting local and global features. Specifically, use a two-layer graph attention network to aggregate multi-hop neighbor information, and the output of the last layer is used as the final structure embedding :

[0021] (1)

[0022] where: and are learnable parameters;

[0023] represents the embedding obtained through GAT;

[0024] represents the local structure subgraph of entity in the knowledge graph;

[0025] S212: Relationship and attribute embedding;

[0026] Use the bag-of-words model to convert the text representation of the relationship into a feature vector, and then input these feature vectors into a feed-forward layer to obtain the embedding of the relationship:

[0027] (2);

[0028] where: represents a learnable parameter;

[0029] FN m represents a modality m a specific feed-forward neural network;

[0030] r and a represent the relationship and attribute modalities respectively;

[0031] BOW ( m ) represents the relationship or attribute features obtained through the BOW model;

[0032] S213: Visual embedding;

[0033] For each entity the corresponding image , input it into the visual model, extract the output features of the model, and further process these features through the feed-forward layer to obtain refined visual embedding :

[0034] (3);

[0035] Among them: represents learnable parameters;

[0036] FN v represents a feed-forward neural network;

[0037] represents the visual features obtained through the VGG16 model.

[0038] As a further limitation of this technical solution, the specific steps of S22 are as follows:

[0039] S221: Cross-modal complementary modeling;

[0040] The core challenge of cross-modal feature fusion is to effectively capture the complementary information between modalities. To solve this problem, a multi-round cross-attention mechanism is proposed, and the specific process is as follows:

[0041] For each modality feature of entity perform L2 norm normalization, as shown in Equation (4):

[0042] (4);

[0043] Among them: m represents a modality embedding vector;

[0044] represents the normalized vector;

[0045] In each round of cross-attention calculation, one modality is selected m as the query, while the remaining modalities are used as the key and value in turn. The detailed calculation process is as follows: The query, key, and value vectors are generated through shared parameter matrices , and as shown in Equations (5)-(7):

[0046] (5);

[0047] (6);

[0048] (7);

[0049] where: is the normalized representation of modality n ;

[0050] Softmax normalization is applied to calculate the attention scores, and based on these scores, the value vectors of each modality are weighted and aggregated as shown in Equation (8):

[0051] (8);

[0052] where: Softmax() is a commonly used normalization method;

[0053] is the dimension of the key;

[0054] The outputs of all modalities are concatenated to form the final complementary embedding matrix :

[0055] (9);

[0056] S222: Introduce a modality correlation matrix, where each element is calculated using cosine similarity as shown in Equation (10). During the fusion process, enhanced correlation information is introduced: After introducing cross-modal correlation modeling, Equation (9) is further updated to Equation (11):

[0057] (10);

[0058] where: S mn represents the similarity between modality m and modality n;

[0059] (11);

[0060] where: is a learnable scaling factor.

[0061] As a further limitation of this technical solution, the specific steps of S23 are as follows:

[0062] S231: Pseudo-label generation;

[0063] Adopt a label propagation method based on graph structure;

[0064] Screen the candidate pair initial pseudo-label set based on the cosine similarity of entity embeddings:

[0065] (12);

[0066] Where: and are the embedding representations of entities and respectively;

[0067] sim(⋅) represents cosine similarity;

[0068] is the similarity threshold;

[0069] For each candidate pseudo-label , check the alignment consistency of its k-hop neighborhood neighbors to verify the reliability of this pseudo-label:

[0070] (13);

[0071] Where: represents the set of k-hop neighbors of entity e ;

[0072] | represents a conditional separator, and the conditions to be satisfied follow;

[0073] is the neighbor alignment ratio threshold;

[0074] For the candidate pseudo-label respectively extract its k-hop neighborhood subgraph: , where and represent the neighbor entity sets in the two graphs respectively, represents its relationship set, and cross-graph relationship semantic alignment is performed through the relationship mapping function to filter out irrelevant neighborhood edges, enhance the accuracy of structure matching, and then calculate the structural similarity:

[0075] (14);

[0076] S232: Momentum contrastive learning

[0077] For thei An entity , whose positive sample set is defined as , where are the aligned entity pairs and high-confidence pseudo-labels, and the negative sample sets are respectively the in the source graph and the in the target graph, which are dynamically expanded during training;

[0078] Based on momentum contrast learning, construct an asymmetric forward path: the online encoder generates online entity representations, while the momentum encoder generates stable target representations, and its parameters are updated according to Equation (15):

[0079] (15);

[0080] where: and represent two different model parameters respectively;

[0081] Define the loss function in two aspects: First is the intra-modal contrast loss:

[0082] (16);

[0083] where: denotes summation over each entity in batch B;

[0084] represents the alignment probability distribution under each modality;

[0085] Use the KL divergence between the joint embedding and the single-modal embedding to represent the inter-modal alignment loss:

[0086] (17);

[0087] where: and represent the alignment probability distributions under the joint embedding and the single-modal embedding respectively;

[0088] The KL() divergence measures the difference between the joint embedding and the single-modal embedding ;

[0089] || indicates that the KL divergence between these two distributions is calculated, that is, it measures their difference in the alignment task;

[0090] The total loss of the final model is set as:

[0091] (18);

[0092] where: denotes Loss applied to joint embedding;

[0093] 、 For balancing and relative weights.

[0094] Compared with the prior art, the advantages and positive effects of the present invention are:

[0095] 1. Heterogeneous encoders extract multimodal features: Different encoders are used to extract features of neighborhood structure, relationships, attributes, and images to capture information of different modalities. Cross-modal feature fusion: In the fusion stage, a cross-attention mechanism is adopted to extract complementary information of modalities, and the dependence between different modalities is explicitly modeled through a modality correlation matrix to enhance the feature expression ability. Pseudo-label optimization based on momentum contrast learning: A label propagation method based on graph structure is designed, combined with momentum contrast learning, to make full use of labeled and unlabeled data, improve alignment robustness, and enable the model to achieve better performance in low-resource scenarios. Experiments were conducted on two multimodal knowledge alignment benchmark datasets, and the results show that CMMEA has achieved significant performance improvement in entity alignment tasks.

[0096] 2. The present invention proposes a semi-supervised multimodal entity alignment framework CMMEA, which uses multiple embedding methods to obtain entity representations of different modalities. Different from traditional direct interaction and fusion strategies, we adopt cross-modal complementarity modeling and correlation modeling to deeply explore the mutual relationship between modalities, so as to obtain more expressive joint feature embeddings. In addition, to improve the quality of pseudo-labels and enhance the effect of contrast learning, we combine the pseudo-label generation mechanism with momentum-based contrast learning, thereby optimizing the selection of alignment samples and further improving the performance of entity alignment. The model can effectively alleviate the problem of modality missing and make full use of the interaction information between multimodals. A large number of experimental results show that on two public datasets, CMMEA has achieved significant alignment effects, verifying the effectiveness of its method. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 It is the overall framework diagram of the CMMEA model of the present invention.

[0098] Figure 2 It is the momentum coefficient of the present invention and temperature coefficient influence. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0099] The following combines the drawings to describe in detail a specific embodiment of the present invention, but it should be understood that the protection scope of the present invention is not limited by the specific embodiment.

[0100] The present invention includes the following steps:

[0101] S1: Symbolic definition of the multi-modal entity alignment task;

[0102] The specific steps of S1 are as follows:

[0103] The multi-modal knowledge graph is defined as: , where: respectively represent entities, relationships, attributes, and vision;

[0104] Given two multi-modal knowledge graphs: the source knowledge graph and the target knowledge graph , the alignment seed set across the two multi-modal knowledge graphs is defined as: , where: represents and are equivalent entities referring to the same real-world object.

[0105] The goal of multi-modal entity alignment is to identify and match semantically equivalent entities from different multi-modal knowledge graphs and utilize multi-modal information to improve the accuracy of alignment.

[0106] S2: Construct a semi-supervised multi-modal entity alignment model (Cross-modal Feature Fusion and Momentum Contrastive Learning for Semi-supervised Multi-modal Entity Alignment, CMMEA).

[0107] S21: The multi-modal knowledge feature embedding module extracts heterogeneous features through a heterogeneous encoder.

[0108] The specific steps of S21 are as follows:

[0109] In a multi-modal knowledge graph (MMKG), entities usually contain heterogeneous multi-modal information (such as structure, attributes, relationships, and vision). Due to significant differences in the representation forms of different modalities, a heterogeneous encoder is used to extract features of each modality and map them to a unified vector space to ensure that the multi-modal fusion module can effectively model the interactions between modalities, thereby improving the quality of entity representation;

[0110] S211: Structure embedding;

[0111] The Graph Attention Network (GAT) is a neural network that operates on graph-structured data. By using an attention mechanism to calculate the relationship weights between nodes, it can more accurately capture the correlations in graph-structured data. Therefore, the GAT is used to and model the structural information, effectively extracting local and global features. Specifically, a two-layer graph attention network is used to aggregate multi-hop neighbor information, and the output of the last layer is used as the final structural embedding :

[0112] (i);

[0113] where: and are learnable parameters;

[0114] represents the embedding obtained through GAT;

[0115] represents the local structural subgraph of entity in the knowledge graph;

[0116] S212: Relationship and attribute embedding;

[0117] In relationship and attribute embedding, considering the simplicity, intuitiveness of the Bag of Words (BOW) model and its effective ability to extract text features, the BOW model is selected to obtain the embeddings of relationships and attributes. Specifically, the text representation of a relationship (or attribute) is converted into a feature vector using the BOW model, and then these feature vectors are input into a feedforward layer to obtain the embedding of the relationship (or attribute):

[0118] (ii);

[0119] where: represents a learnable parameter;

[0120] FN m represents a modality m specific feedforward neural network for non-linearly transforming and enhancing the bag-of-words features;

[0121] r and a represent the relationship and attribute modalities respectively;

[0122] BOW( m ) represents the relationship or attribute features obtained through the BOW model;

[0123] S213: Visual Embedding;

[0124] To effectively capture the visual features of entities, we adopt a pre-trained visual model (PVM) - VGG16. This model extracts the visual information of images through convolutional layers and generates embedded features. For each entity the corresponding image , its input into the visual model, extracts the output features of the model, and further processes these features through a feedforward layer to obtain refined visual embeddings :

[0125] (3);

[0126] Where: represents learnable parameters;

[0127] FN v represents a Feedforward Network, which is used to perform further non-linear transformation and mapping on the image features extracted by VGG16;

[0128] represents the visual features obtained through the VGG16 model.

[0129] S22: The cross-modal fusion module uses the cross-attention mechanism to model the complementarity and correlation between modalities and generate joint representations.

[0130] The specific steps of S22 are as follows:

[0131] In multi-modal learning, the information provided by different modalities often has different characteristics and perspectives. Traditional fusion methods usually rely on simple concatenation or weighting strategies. Although these methods can integrate multi-modal information, they often fail to reveal the deep relationships between modalities, thus limiting the model's ability to fully utilize multi-modal information. Such fusion methods may lead to information redundancy or loss of key features, ultimately affecting the learning effect and overall performance of the model. To solve these problems, a fusion strategy based on modeling the complementarity and correlation between modalities is proposed. This strategy uses cross-modal feature fusion to deeply explore the potential relationships between different modalities, enabling the model to more accurately capture the essence of each modality's information and ultimately improving the performance of the MMEA task.

[0132] S221: Cross-modal Complementarity Modeling;

[0133] The core challenge of cross-modal feature fusion is to effectively capture the complementary information between modalities. To solve this problem, a multi-round cross-attention mechanism is proposed. The specific process is as follows:

[0134] First, for each modal feature of the entity (where represent the structure, relationship, attribute, and visual modalities respectively), perform L2 normalization (Euclidean normalization, a mathematical tool used to measure the "length" or "size" of a vector), as shown in Equation (4):

[0135] (4);

[0136] Where: m represents a modal embedding vector, such as the original representation under the structure, relationship, attribute, or visual modality of the entity;

[0137] represents the normalized vector with a unit norm (unit vector);

[0138] In each round of cross-attention calculation, select one modality m as the query, and the remaining modalities , are used as keys and values in turn. The detailed calculation process is as follows: The query, key, and value vectors are generated through shared parameter matrices , and as shown in Equations (5)-(7):

[0139] (5);

[0140] (6);

[0141] (7);

[0142] Where: is the normalized representation of modality n ;

[0143] Next, apply softmax (Short for "Soft Maximum", Softmax) normalization to calculate the attention scores, and based on these scores, perform weighted aggregation on the value vectors of each modality, as shown in Equation (8):

[0144] (8);

[0145] Where: Softmax() is a commonly used normalization method for mapping a set of real numbers to a probability distribution;

[0146] is the dimension of the key, used to scale to prevent excessive gradients;

[0147] Finally, the outputs of all modalities are concatenated to form the final complementary embedding matrix :

[0148] (9);

[0149] S222: In order to further capture the potential information shared between modalities, a modality correlation matrix is introduced, in which each element is calculated using cosine similarity, as shown in formula (10). In the fusion process, enhanced related information is introduced: After introducing cross-modal correlation modeling, formula (9) is further updated to formula (11):

[0150] (10);

[0151] Where: S mn represents the similarity between modality m and modality n;

[0152] (11);

[0153] in: is a learnable scaling factor that balances the contributions of complementarity and correlation.

[0154] S23: The pseudo-label generation and momentum contrastive learning module selects high-confidence pseudo-labels based on graph label propagation, and combines momentum queues to enhance contrastive learning stability.

[0155] The specific steps of S23 are:

[0156] A semi-supervised framework with pseudo-label enhanced contrastive learning is proposed. The method consists of two core components: (1) Graph-based label propagation (GLP) is used to generate high-confidence pseudo-labels to expand the training set and effectively mitigate the error propagation caused by noisy labels. (2) A momentum-based contrastive learning framework can more effectively enhance alignment by attracting positive pairs and repelling negative pairs in the embedding space.

[0157] S231: pseudo label generation;

[0158] To improve the performance of multimodal entity alignment and reduce noise in the pseudo-label generation process, we use a graph-based label propagation (GLP) method. This method fully utilizes the structural information in the knowledge graph and combines neighbor consistency verification and cross-graph subgraph matching to ensure high pseudo-label quality.

[0159] First, the candidate pairs of initial pseudo-label sets are screened based on the cosine similarity of entity embeddings:

[0160] (12);

[0161] Wherein: and are the embedding representations of entities and respectively;

[0162] sim(⋅) represents the cosine similarity;

[0163] is the similarity threshold;

[0164] Then, for each candidate pseudo-label check the alignment consistency of its k-hop neighborhood neighbors to verify the reliability of the pseudo-label:

[0165] (13);

[0166] Wherein: represents the set of k-hop neighbors of entity e ;

[0167] | represents the conditional separator, followed by the satisfied condition;

[0168] is the neighbor alignment ratio threshold;

[0169] On the pseudo-labels that pass the neighbor consistency verification, further use topological similarity to measure its structural matching degree. The specific steps are as follows: For the candidate pseudo-label extract its k-hop neighborhood subgraphs respectively: where and represent the neighbor entity sets in the two graphs respectively, represents its relationship set, and through the relationship mapping function perform cross-graph relationship semantic alignment, filter out irrelevant neighborhood edges, enhance the accuracy of structural matching, and then calculate the structural similarity:

[0170] (14);

[0171] In the initial stage of training, set a higher threshold for strict screening; in the middle stage of training, linearly decay to to introduce medium-consistency pseudo-labels; in the later stage of training, adaptively adjust to allow .

[0172] Finally, on the FB15K-DB15K and FB15K-YAGO15K datasets, determine the hyperparameters through cross-validation to optimize the quality and quantity of pseudo-labels;

[0173] S232: Momentum Contrastive Learning

[0174] Inspired by the recent contrastive learning, momentum contrastive learning, this paper proposes a semi-supervised framework that combines pseudo-label generation with momentum-based contrastive learning to optimize entity alignment performance. For the i th entity in the mini-batch B, its positive sample set is defined as , where is the set of aligned entity pairs and high-confidence pseudo-labels, and the negative sample sets are in the source graph and in the target graph respectively, which are dynamically expanded during training;

[0175] Based on momentum contrastive learning, an asymmetric forward path is constructed: the online encoder generates online entity representations, while the momentum encoder generates stable target representations, and its parameters are updated according to Equation (15):

[0176] (15);

[0177] where: and represent two different model parameters for the update between online learning and the target model (momentum model) respectively;

[0178] The loss function is defined in two aspects: First, the intra-modal contrast loss:

[0179] (16);

[0180] where: denotes the sum over each entity in the batch B;

[0181] represents the alignment probability distribution under each modality;

[0182] The inter-modal alignment loss is represented by the KL (Kullback-Leibler Divergence, KLDivergence) between the joint embedding and the single-modal embedding:

[0183] (17);

[0184] where: and represent the alignment probability distributions under the joint embedding and the single-modal embedding respectively;

[0185] The KL() divergence measures the difference between the joint embedding and the single-modal embedding ;

[0186] || indicates that the KL divergence between these two distributions is calculated, which measures their differences in the alignment task;

[0187] The total loss of the final model is set as:

[0188] (18);

[0189] Where: represents the loss applied to the joint embedding; is applied to the loss on the joint embedding;

[0190] , is used to balance the and The relative weights of. , are set as learnable parameters and automatically updated during model training using homoscedastic uncertainty to balance the total loss of the model.

[0191] Through end-to-end optimization, CMMEA significantly improves the alignment robustness in the noisy label scenario.

[0192] 3. Experiments

[0193] 3.1 Experimental settings

[0194] 3.1.1 Datasets

[0195] This invention conducts experiments on two widely used multi-modal entity alignment (MMEA) benchmark datasets: FB15K-DB15K and FB15K-YAGO15K. Among them, FB15K is derived from the Freebase knowledge base, and DB15K and YAGO15K come from the DBPedia and YAGO knowledge bases respectively. These datasets have become the mainstream evaluation benchmarks for the MMEA task due to their rich multi-modal information (text, structure, vision) and standardized alignment annotations.

[0196] 3.1.2 Evaluation metrics

[0197] The experiments adopt Hits@1, Hits@10, and mean reciprocal rank (MRR) as evaluation metrics: Hits@n represents the proportion of correct entities appearing in the top n predicted results, and MRR measures the overall alignment quality by calculating the average reciprocal of the correct entity rankings. The higher the metric value, the better the model performance.

[0198] 3.1.3 Baselines

[0199] The comparison baseline covers two types of methods: traditional entity alignment methods (TransE, IPTransE, and GCN-Align) and multi-modal entity alignment methods (PoE, MMEA, EVA, HMEA, MCLEA, and MEAformer). Traditional methods mainly rely on unimodal structural information, while multi-modal methods fuse cross-modal features through attention mechanisms, contrastive learning, or hybrid models.

[0200] 3.1.4 Implementation Details

[0201] The experiments were implemented based on the PyTorch framework, and the model parameters were set as follows: GAT adopted a 2-layer structure with a hidden layer dimension of 300, while the embedding size of other modalities was 100. The AdamW optimizer was used with a learning rate of 5e-4 and a weight decay of 1e-2. The batch size was set to 512, the number of training epochs was 500, and an early stopping strategy (patience value of 10 epochs) was adopted. In momentum contrastive learning, the momentum coefficient was set to 0.99, and the temperature parameter τ t was set to 0.1.

[0202] 3.2 Results

[0203] Table 1 Experimental results on datasets with two different seed ratios

[0204]

[0205] The best results in Table 1 are shown in bold, and the second results are underlined.

[0206] To verify the effectiveness of CMMEA, Table 1 reports the comparison of CMMEA with other baselines when the alignment seeds are 20%, 50%, and 80% respectively.

[0207] From Table 1, it can be observed that: 1) On both datasets with all seed ratios, CMMEA outperforms all baseline models in terms of Hits@1, Hits@10, and MRR metrics, indicating the generality of the model. Specifically, when the seeds are 50% and 80%, the Hits@1 of the model on the dataset increases by 2.1%, 3.7%, 2.6%, and 2.3% respectively. In other cases, CMMEA also achieves corresponding improvements. Especially when the seeds are at other ratios and under low resources, it can still maintain a large improvement, indicating the effectiveness of the proposed pseudo-label generation module. 2) Compared with traditional MMEA, the model has achieved corresponding improvements, indicating that a more reasonable fusion method can better utilize existing information. 3) Compared with traditional EA, the significant improvement achieved by MMEA fully demonstrates that using multi-modal information is more effective than using single-modal information. The above results also indicate the effectiveness of the model.

[0208] 3.3 Ablation Experiments

[0209] Table 2 Variant experiments on 20% seeds of two datasets, w / o means removing the corresponding module

[0210]

[0211] To verify the role of each module, variant experiments were conducted, and the results are shown in Table 2. From Table 2, it can be observed that: 1) After removing the structural and visual information, the performance of the model on the two datasets decreased significantly. This is because the structural information provides the context of entities, which helps to distinguish entities at different levels; while the visual information provides semantic information such as the shape and color of entities. Especially in the case of multiple entities with the same name, the visual information plays an important role in helping to distinguish and align entities. 2) When removing L ICL the performance also decreased significantly, which is because L ICL directly clusters similar entities together, while other entities transfer information between different patterns. 3) After w / o GLP, the performance decreased, indicating the importance of pseudo-labels in semi-supervised entity alignment.

[0212] In summary, when any module is removed, the performance decreases, from which it can be concluded that each module plays an important role in the model.

[0213] 3.4 Parameter Analysis

[0214] Hyperparameter studies were conducted using FB15K-DB15K, and the results are as Figure 2 shown. For the momentum coefficient, a moderately large value can bring better stability and accuracy in Hits@1, which indicates that momentum-based contrast learning is more effective than just contrast learning. Selecting = 0.99 is a comprehensive balance considering the model convergence stability, computational efficiency, and risk controllability on the premise that the experimental performance is close to the optimal. Different values have a significant impact on CMMEA, especially in terms of Hits@1 and MRR, which is because the penalty intensity for negative samples is controlled, and an appropriate value is beneficial to learning to distinguish entity embeddings.

[0215] The specific embodiments disclosed above are only for the present invention. However, the present invention is not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A semi-supervised multi-modal entity alignment method, characterized in that Including the following steps: S1: Symbol definition of the multi-modal entity alignment task; S2: Construct a semi-supervised multi-modal entity alignment model; S21: The multi-modal knowledge feature embedding module extracts heterogeneous features through a heterogeneous encoder; S22: The cross-modal fusion module uses the cross-attention mechanism to model the complementarity and correlation between modalities and generates a joint representation; S23: The pseudo-label generation and momentum contrast learning module filters high-confidence pseudo-labels based on graph label propagation and combines the momentum queue to enhance the stability of contrast learning; The specific steps of S1 are as follows: A multimodal knowledge graph is defined as: , where: represent entity, relationship, attribute, and vision respectively; Given two multimodal knowledge graphs: the source knowledge graph and the target knowledge graph , the alignment seed set across the two multimodal knowledge graphs is defined as: , where: denotes and are equivalent entities referring to the same real-world object; The specific steps of S21 are as follows: S211: Structure embedding; Using a graph attention network for and to model the structural information, effectively extracting local and global features. Specifically, a two-layer graph attention network is used to aggregate multi-hop neighbor information, and the output of the last layer is used as the final structural embedding. : （1） Wherein: and are learnable parameters; Denote the embeddings obtained by GAT; Represents an entity The local structure subgraph in the knowledge graph; S212: Relationship and attribute embedding; Use the bag-of-words model to convert the text representation of the relationship into a feature vector, and then input these feature vectors into a feed-forward layer to obtain the embedding of the relationship: （2）； Wherein: represents learnable parameters; FN m represents a modality m a specific feedforward neural network; r and a respectively represent a relationship and an attribute modality; BOW ( m ) represents the relationship or attribute features obtained through the BOW model; S213: Visual embedding; For each entity the corresponding image , input it into the visual model, extract the output features of the model, and further process these features through the feed-forward layer to obtain refined visual embeddings : （3）； Wherein: represents learnable parameters; FN v represents a feedforward neural network; Represents the visual features obtained through the VGG16 model.

2. The semi-supervised multi-modal entity alignment method according to claim 1, wherein: The specific steps of S22 are as follows: S221: Cross-modal complementarity modeling; The core challenge of cross-modal feature fusion is to effectively capture the complementary information between modalities. To solve this problem, a multi-round cross-attention mechanism is proposed. The specific process is as follows: For an entity each modal feature of is normalized by L2 norm as shown in Equation (4): （4）； Wherein: m represents a modal embedding vector; Indicates the normalized vector; In each round of cross-attention calculation, one modality is selected m as the query, while the remaining modalities are used as the key and value in turn. The detailed calculation process is as follows: The query, key, and value vectors are generated by sharing parameter matrices , and as shown in equations (5)-(7): （5）； （6）； （7）； Wherein: is the normalized representation of the modality n ; Apply softmax normalization to calculate the attention scores, and based on these scores, perform weighted aggregation on the value vectors of each modality, as shown in Equation (8): （8）； Where: Softmax() is a commonly used normalization method; is the dimension of the key; Connect the outputs of all modalities to form the final complementary embedding matrix : （9）； S222: Introduce a modality correlation matrix, where each element is calculated using cosine similarity, as shown in Equation (10). During the fusion process, enhanced correlation information is introduced: After introducing cross-modal correlation modeling, Equation (9) is further updated to Equation (11): （10）； Where: S mn represents the similarity between mode m and mode n; （11）； Wherein: is a learnable scaling factor.

3. The semi-supervised multi-modal entity alignment method according to claim 2, wherein: The specific steps of S23 are as follows: S231: Pseudo-label generation; Adopt a label propagation method based on the graph structure; Filter the initial pseudo-label set of candidate pairs based on the cosine similarity of entity embeddings: （12）； Wherein: and are respectively the embedding representations of entities and ; sim(⋅) represents cosine similarity; is the similarity threshold; For each candidate pseudo-label , check the alignment consistency of its k-hop neighborhood neighbors to verify the reliability of the pseudo-label: （13）； Wherein: represents the set of k-hop neighbors of the entity e ; | represents the conditional separator, and the following is the satisfied condition; is the neighbor alignment ratio threshold; For candidate pseudo-labels Extract their k-hop neighborhood subgraphs respectively: , where and represent the neighbor entity sets in the two graphs respectively, represents its relation set, and cross-graph relation semantic alignment is performed through the relation mapping function to filter out irrelevant neighborhood edges, enhance the accuracy of structure matching, and then calculate the structural similarity: （14）； S232: Momentum contrast learning For the i th entity in the small batch B , its positive sample set is defined as , where are the aligned entity pairs and high-confidence pseudo-labels, and the negative sample sets are respectively in the source graph and in the target graph, which are dynamically expanded during training; Based on momentum contrast learning, construct an asymmetric forward path: The online encoder generates online entity representations, while the momentum encoder generates stable target representations, and its parameters are updated according to Equation (15): （15）； Wherein: and respectively represent two different model parameters; Define the loss function in two aspects: First is the intra-modal contrast loss: （16）； Wherein: represents the sum for each entity in batch B to be carried out; Represents the alignment probability distribution in each modality; Use the KL divergence between the joint embedding and the single-modal embedding to represent the inter-modal alignment loss: （17）； Wherein: and represent the alignment probability distributions under joint embedding and unimodal embedding, respectively; The KL() divergence measures the difference between the joint embedding and the unimodal embedding ; || indicates that the KL divergence between these two distributions is calculated, that is, it measures their differences in the alignment task; The total loss of the final model is set as: （18）； Wherein: denotes the loss applied to the joint embedding; ​ , for balancing and the relative weights.

Images