Transform-based multi-modal knowledge graph dynamic adaptive alignment method

Abstract

The application discloses a kind of based on the dynamic self-adaptive alignment method of multi-modal knowledge graph of Transformer, comprising: two multi-modal knowledge graphs are decomposed according to modality feature dimension and then encoded mapping to obtain multi-modal entity feature representation;Align aggregation is carried out based on the multi-modal entity feature representation projected and mapped after global adaptive weight;Fusion vector is obtained by self-defined modality fusion based on attention mechanism through Transformer;Based on alignment aggregation and fusion vector, and introducing modal adaptive contrast learning framework, one-stage training loss is constructed;Based on the imagination module of cyclic missing modality, two-stage training loss is constructed, and learning of all modules is based on loss function, to realize modal knowledge graph dynamic self-adaptive alignment.

Description

Technical Field

[0001] This invention belongs to the field of entity alignment technology, specifically relating to a dynamic adaptive alignment method for multimodal knowledge graphs based on Transformer. Background Technology

[0002] Entity alignment (EA) aims to discover equivalent entities across different knowledge graphs (KGs) to facilitate knowledge fusion. Early EA systems (Jiménez-Ruiz, Ernesto, and Bernardo Cuenca Grau. "Logmap: Logic-based and scalable ontology matching." The Semantic Web – ISWC 2011: 10th International Semantic Web Conference, Bonn, Germany, October 23-27, 2011, Proceedings, Part I 10. Springer Berlin Heidelberg, 2011.) utilized techniques such as logical reasoning and lexical matching for entity mapping construction, heavily relying on (specific) heuristics. In recent years, embedding-based EA methods have alleviated the heterogeneity problem by learning an embedding space to represent the KGs to be aligned, where similar entities are placed together, while dissimilar entities are separated.

[0003] Since the development of multimodal learning, incorporating visual modalities into entity alignment (EA) in the construction of multimodal knowledge graphs (MMKG) has gradually attracted the attention of the community. Multimodal entity alignment (MMEA) mixes the knowledge representations of modalities and then minimizes the distance between the overall embeddings of aligned entities. EVA (Liu, Fangyu, et al. "Visual pivoting for (unsupervised) entity alignment." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 5. 2021.) employs a variable-weight attention mechanism to assign different importance to each modality. The multimodal synchronous entity alignment method MSNEA (Chen, Liyi, et al. "Multi-modalsiamese network for entity alignment." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.) integrates visual features to guide relation feature learning while assigning valuable attribute weights to the alignment. The MCLEA model (Lin Z, Zhang Z, Wang M, et al. Multi-modal Contrastive Representation Learning for Entity Alignment[C] / / Proceedings of the 29th International Conference on Computational Linguistics.2022:2572-2584.) further enhances the learning within entities through contrastive learning, and applies KL divergence between joint and unimodal embeddings to reduce the gap between modalities.

[0004] However, all of the aforementioned multimodal alignment methods neglect the dynamic modal effects of each entity when aligning multimodal data. This is significant in real-world EA scenarios because errors and noise are inevitable in internet or professional domain KGs (especially MMKGs), such as the inclusion of unrecognizable images. Furthermore, KGs often contain feature differences between modalities (e.g., node degree) and source biases (e.g., missing modalities, imbalances, or uncertainties). Summary of the Invention

[0005] In view of the above, the purpose of this invention is to provide a dynamic adaptive alignment method for multimodal knowledge graphs based on Transformer, which has better robustness, stronger ability to resist noise, and higher accuracy.

[0006] To achieve the above-mentioned objectives, this invention provides a dynamic adaptive alignment method for multimodal knowledge graphs based on Transformer, comprising the following steps:

[0007] Two given multimodal knowledge graphs are decomposed into multiple dimensions according to modal features, and the content features of each dimension are encoded to obtain the multimodal entity feature representation corresponding to each entity. The multimodal entity feature representation is then projected onto the same space.

[0008] A global modality aggregation module is constructed, which performs alignment and aggregation based on the multimodal entity feature representation after global adaptive weight projection mapping;

[0009] Construct a modal alignment module at the entity level, and obtain a fusion vector by performing custom modal fusion based on an attention mechanism using Transformer;

[0010] Based on alignment aggregation and fusion vectors, and introducing a modality-adaptive contrastive learning framework, a one-stage training loss is constructed.

[0011] Construct a recurrent missing modality imagination module, and build a two-stage training loss based on the recurrent missing modality imagination module;

[0012] Based on the first-stage training loss and the second-stage training loss, and using a modality-aware hard negative sample replay strategy to learn all modules, dynamic adaptive alignment of the modality knowledge graph is achieved.

[0013] Preferably, the dimensions include graph structure dimension, relationship dimension, attribute dimension, and visual image dimension;

[0014] For the attribute and relation dimensions, a bag-of-words model is used to encode the content features corresponding to the attributes and relations; for the graph structure dimension, a graph attention model is used to encode the content features corresponding to the graph structure; for the visual image dimension, an image pre-trained encoder is used to encode the content features corresponding to the visual image.

[0015] A modally adaptive fully connected layer is used to perform cospatial projection mapping on the entity feature representations corresponding to each dimension.

[0016] Preferably, in the global modality aggregation module, the following formula is used to perform the first alignment aggregation of the projected multimodal entity feature representations based on global adaptive weights:

[0017]

[0018] in, w represents the feature representation obtained by alignment aggregation. m This represents the global adaptive weight of modality m in two multimodal knowledge graphs. This represents the entity feature representation after projection mapping corresponding to mode m. This represents the modality set, or the dimension set. This indicates a weighted concatenation operation, where the subscript i represents the entity concept number.

[0019] Preferably, the entity-level modal alignment module includes a dynamic cross-modal weighting submodule;

[0020] In the dynamic cross-modal weighted submodule, modal weights are adaptively scored and modal fusion is performed based on the Transformer's attention mechanism, specifically as follows:

[0021] The entity feature representation h after each modal projection mapping m Based on transformation function Mapping to independent The matrix is ​​used to calculate the attention weight β for each attention head i. mj and attention vector And by spatial mapping and concatenation, the vector MHCA(h) is obtained. m ), expressed by the formula:

[0022]

[0023]

[0024]

[0025] Where, β mj This represents the mutual attention weights of modalities m and j at each attention head. Representing the Query matrix The transpose of K j and K l Let K and d represent the key matrices for modes j and l, respectively. h =d / N h This represents the dimension of the vector in each attention head. Represents a set of modes. V represents the attention weights between modal m and j calculated based on the i-th attention head. j (i) V represents the Value matrix calculated based on mode j. W represents the concatenation operation. o N represents the transformation matrix. hIndicates the number of attention heads;

[0026] Introducing LayerNorm and residuals for stable model training to apply vector MHCA(h) m The first-stage fusion vector is obtained by calculation. Expressed as a formula:

[0027]

[0028] Adding a feedforward neural network for two-stage fusion yields a two-stage fusion vector. Expressed as a formula:

[0029]

[0030]

[0031] Where W1 and W2 represent learnable transformation matrices, and b1 and b2 represent learnable biases.

[0032] Preferably, the method further includes: extracting attention weights β from the dynamic cross-modal weighted submodule. mj After accumulation, the meta-weights of each mode m are obtained through the SoftMax operation. Expressed as a formula:

[0033]

[0034] in, Indicates the number of modes.

[0035] Preferably, the construction of the one-stage training loss based on alignment aggregation and fusion vectors, and incorporating a modality-adaptive contrastive learning framework, includes:

[0036] The learning objective of the contrastive learning framework is defined as:

[0037]

[0038] in, This indicates a positive sample pair, which is a pair of entities that have been pre-aligned from two multimodal knowledge graphs. express The entity feature representation corresponding to the modality m dimension. Let e ​​represent the set of negative samples. j Indicates originating from And with To form negative sample pairs, τ represents the temperature coefficient hyperparameter, and i represents the index of the original entity concept. Indicates positive sample pairs Alignment probability;

[0039] Considering that the alignment process is bidirectional, a bidirectional alignment training objective is defined. for:

[0040]

[0041] When the training target is aligned bidirectionally. When adapting to global modality alignment, construct the loss function. for:

[0042]

[0043] in, use according to τ), we can obtain This refers to what was mentioned earlier. The corresponding result is the alignment and aggregation of the multimodal entity feature representations after projection mapping based on global adaptive weights;

[0044] A confidence-enhancing intra-modal alignment strategy is introduced to achieve independent alignment of each modality. The loss function is constructed as follows:

[0045]

[0046]

[0047] in, for One variant, namely the introduction Functions are used to cancel out potential noise interference, achieving robustness against relevant missing modes / modal noise. According to the formula Take from entity and Corresponding meta weights The minimum value (i.e., the minimum confidence value) is taken as the weight; for Another variant, China adopts in, express The two-stage fusion vector corresponding to the modality m dimension;

[0048] Ultimately, the training loss in the first phase is:

[0049]

[0050] Preferably, the two-stage training loss constructed based on the recurrent missing modality imagination module includes:

[0051] The cyclic missing modality imagination module employs a variational multimodal autoencoder containing an encoder and a decoder, using trimodal hybrid features. As input, where as well as These represent entity feature representations for three modalities: relation, attribute, and graph structure, respectively, and use the output of the hidden layer between the encoder and decoder as imagined pseudo-visual features. Variational multimodal autoencoder for trimodal hybrid features The calculation process can be expressed by the formula:

[0052]

[0053]

[0054]

[0055] Among them, MLP Enc (·) indicates an encoding operation. Indicates decoding operation, μ i and σ i These represent the mean and variance of a normal distribution. ⊙ represents the concatenation operation, ⊙ represents the dot product operation, and z represents the normal distribution. This indicates the decoding result;

[0056] Define the reconstructed training loss and To minimize respectively and in, Representing real image features, the formula is expressed as:

[0057]

[0058]

[0059] Introducing KL training loss The output of the guided encoding approximates a Gaussian distribution, which can be expressed by the formula:

[0060]

[0061] in, It is an entity with a complete image in a minimum batch, where i represents the entity index;

[0062] By analyzing trimodal mixture features The overall fusion vector similarity matrix is ​​obtained, and the similarity matrix is ​​distilled to virtual image features. In the similarity matrix, to preserve known information, construct... The loss function is expressed as:

[0063]

[0064] in, and They are represented as follows:

[0065]

[0066]

[0067]

[0068] This indicates a positive sample pair, which is a pair of entities that have been pre-aligned from two multimodal knowledge graphs. express The corresponding trimodal mixture characteristics, express The corresponding pseudo-visual feature, τ represents the temperature coefficient hyperparameter;

[0069] Finally, the two-stage loss function is:

[0070]

[0071] Preferably, the learning of all modules is divided into two stages. In the first stage, a one-stage loss function is used to optimize the parameters of the main body only. In the second stage, the one-stage loss function and the two-stage loss function are used for joint training. The specific process is as follows: the main body is frozen, and the parameters of the recurrent missing modality imagination module are optimized using the two-stage loss function. Then, the recurrent missing modality imagination module is frozen, and the parameters of the main body are optimized using the one-stage loss function. The main body includes the network of the feature encoding process, the network of the mapping process, the global modality aggregation module, and the modality alignment module at the entity level.

[0072] Preferably, the modality-aware hard negative sample replay strategy is as follows: During training, the entity misaligned in each round is defined as a hard entity, and the hard entity is added to the negative samples to participate in contrastive learning training. Specifically, the intra-batch contrastive learning is extended to search-based out-of-batch contrastive learning, using a strong negative sample mapping matrix M. neg To achieve negative sample retrieval and localization, the strong negative sample mapping matrix M... neg The system is dynamically updated in each batch during the training process, and in each round, mismatched entity pairs are stored in the strong negative sample mapping matrix as dictionary key-value pairs.

[0073] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0074] This paper proposes a novel and superior entity alignment strategy, which generates reasonable instance-level multimodal entity hybrid features to better address intramodal differences and intermodal preferences for each entity. Compared with previous methods, this invention places greater emphasis on dynamism. Through an entity-level modality alignment module, it adaptively predicts relative modality weights for each entity, achieving cross-modal correction and further modality adjustment. Furthermore, this invention employs a modality-adaptive contrastive learning objective to separate modal information, further enhancing this dynamism. A modality-aware hard entity replay strategy is used to further strengthen the model's robustness to blurred entity details. Compared with existing methods, this invention exhibits superior robustness, stronger noise resistance, and higher accuracy. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 This is a flowchart of the Transformer-based dynamic adaptive alignment method for multimodal knowledge graphs provided in the embodiment;

[0077] Figure 2 This is a schematic diagram of the model structure for applying the alignment method provided in the embodiment;

[0078] Figure 3 These are comparison images of the effects provided in the embodiments. The left image shows the data problems that may be encountered in current actual scenarios, and the right image shows the final alignment effect of the method of the present invention. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0080] like Figure 1 and Figure 2 As shown in the figure, the multimodal knowledge graph dynamic adaptive alignment method based on Transformer provided in this embodiment of the invention includes the following steps:

[0081] Step 1: Decompose the two given multimodal knowledge graphs into multiple dimensions according to modal features, encode the content features of each dimension to obtain the multimodal entity feature representation corresponding to each entity, and perform isospace projection mapping on the multimodal entity feature representation.

[0082] Two given multimodal knowledge graphs are decomposed into multiple dimensions based on modal features, specifically including graph structure dimension, relation dimension, attribute dimension, and visual image dimension. During the feature dimension decomposition process, modality loss is tolerated; that is, partial coupling exists between different modalities, and partial modality loss (such as image modality loss) can be replaced by random noise as input.

[0083] After decomposition, for the content features of each dimension, a relevant encoder is used for feature encoding, resulting in four different multimodal entity feature representations for each entity. Specifically, for the attribute and relation dimensions, a bag-of-words model is used to encode the content features corresponding to attributes and relations. This is a multi-hot encoding method where, with a fixed bag-of-words size, relation / attribute features for each entity are pre-sampled and defined after being sorted by frequency of occurrence. Features with lower frequency and ranking below the bag-of-words size are discarded, increasing the flexibility and transferability of the method and effectively improving computational speed. For the visual image dimension, an image pre-trained encoder is used to encode the content features corresponding to the visual images. This image pre-trained encoder is frozen to reduce training parameters and alleviate overfitting. For the graph structure dimension, a graph attention model (GAT) is used to encode the content features corresponding to the graph structure. The GAT is a trainable structure and participates in training along with randomly initialized feature vectors for each node.

[0084] After obtaining the feature representation of each modality entity, the features of the three modalities r, a, and v are mapped to the feature space through a fully connected layer to obtain the mapped entity feature representation h. m This can be expressed as a formula:

[0085] h m =FC m (x m ),m∈{r,a,v}

[0086] In this way, a fully connected network with modality adaptation can project modality vectors from different feature spaces into the same space.

[0087] Step 2: Construct a global modality aggregation module and perform alignment and aggregation based on the multimodal entity feature representation after global adaptive weight projection mapping.

[0088] A Global Modality Integration (GMI) module is constructed to associate each pair of multimodal entities through global alignment. The multimodal feature representations of each entity are concatenated, and then aligned using learnable global weights. This allows the model to adaptively learn the global relative importance of each modality in the two MMKGs. Through globally adaptive weighting, the GMI module can utilize high-quality modal information while minimizing the impact of low-quality modalities (such as redundant information in attributes or relationships and noise in the image). Furthermore, it ensures that valuable information is preserved to a certain extent, thereby improving the ability to perceive feature differences between unaligned entities.

[0089] In the GMI module, the following formula is used to perform the first alignment aggregation of the projected multimodal entity feature representations based on global adaptive weights:

[0090]

[0091] in, w represents the feature representation obtained from the first alignment aggregation. m This represents the global adaptive weight of modality m in two multimodal knowledge graphs. This represents the entity feature representation after projection mapping corresponding to mode m. This represents the modality set, or the dimension set. This indicates a weighted concatenation operation, where the subscript 'i' represents the entity concept's ID. For example, the entity concept 'Thor' has ID 'i', and it can appear in both multimodal knowledge graphs simultaneously. This represents the pre-aligned entity pairs for the Thor entity from two multimodal knowledge graphs. There may be different names, such as "Thor" and "Thunder God", or languages, such as "Thunder God" and "Thor".

[0092] Step 3: Construct a modal alignment module at the entity level, and obtain a fusion vector by performing custom modal fusion based on an attention mechanism using Transformer.

[0093] A modal alignment module is constructed at the design entity level. The modal alignment module includes a dynamic cross-modal weighting (DCW) submodule, in which the DCW submodule is used to obtain a fusion vector by performing custom modal fusion based on an attention mechanism through Transformer.

[0094] Specifically, based on multimodal entity feature representations within the same semantic space, a DCW submodule is designed using the Transformer architecture to perform intramodal interaction and fusion. The Transformer includes a Multi-Head Cross-Modal Attention (MHCA) layer and a Feedforward Neural Network (FFN). The MHCA layer performs adaptive scoring of modal weights based on an attention mechanism for modal fusion. The loss function (objective function) in the entity alignment stage is used to enhance the accuracy of the weights in MHCA. The specific process is as follows:

[0095] The entity feature representation h after projection mapping of each modality (m) m Based on transformation function Mapping to independent A matrix, expressed by the formula:

[0096]

[0097] The attention weight β for each attention head i is calculated based on a matrix. mj and attention vector And by spatial mapping and concatenation, the vector MHCA(h) is obtained. m ), expressed by the formula:

[0098]

[0099]

[0100]

[0101] Where, β mj This represents the mutual attention weights of modalities m and j at each attention head. Representing the Query matrix The transpose of K j and K l Let K and d represent the key matrices for modes j and l, respectively. h =d / N h This represents the dimension of the vector in each attention head. Represents a set of modes. V represents the attention weights between modal m and j calculated based on the i-th attention head. j (i) V represents the Value matrix calculated based on mode j. W represents the concatenation operation. o N represents the transformation matrix. h Indicates the number of attention heads;

[0102] Introducing LayerNorm and residuals for stable model training to apply vector MHCA(h) mThe first-stage fusion vector is obtained by calculation. Expressed as a formula:

[0103]

[0104] FFN is used for two-stage fusion to obtain the two-stage fusion vector. Expressed as a formula:

[0105]

[0106]

[0107] Where W1 and W2 represent learnable transformation matrices, and b1 and b2 represent learnable biases.

[0108] Specifically, attention weight β is also extracted from the DCW submodule. mj After accumulation, the meta-weights of each mode m are obtained through the SoftMax operation. Expressed as a formula:

[0109]

[0110] in, This represents the number of modes, and the element weight. It includes key cross-modal information and can adaptively adjust the model's modal preference for each entity.

[0111] Step 4: Based on the alignment aggregation and fusion vectors, and by introducing a modality-adaptive contrastive learning framework, construct a one-stage training loss.

[0112] To facilitate learning these dynamic confidence levels and incorporating them into the training process, a modality-adaptive contrastive learning framework was designed to reinforce the modality alignment signal. A training entity is sampled from both the source and target multimodal knowledge graphs, with all entities in the batch forming a set N. i For pre-aligned entity pairs Let each entity be a positive sample to the others, and let all other entities in the set be negative samples. The learning framework participates in the comparative learning process, where ε1 and ε2 represent entity sets from two multimodal knowledge graphs. This framework simultaneously applies to both the multimodal fusion vector and the independent modality vector of each entity. The learning objective is defined as:

[0113]

[0114] in, This indicates a positive sample pair, which is a pair of entities that have been pre-aligned from two multimodal knowledge graphs. express In the entity feature representation corresponding to the modality m dimension, e j Indicates originating from And with To form negative sample pairs, τ represents the temperature coefficient hyperparameter, and i represents the index of the original entity concept. Indicates positive sample pairs Alignment probability;

[0115] Considering that the alignment process is bidirectional, a bidirectional alignment training objective is defined. for:

[0116]

[0117] When the training target is aligned bidirectionally. When adapting to global modality alignment, construct the loss function. for:

[0118]

[0119] in, use according to achievable This refers to what was mentioned earlier. The corresponding result is the alignment and aggregation of the multimodal entity feature representations after projection mapping based on global adaptive weights;

[0120] An Explicit Confidence-Augmented Intra-Modal Alignment (ECIA) strategy is introduced to achieve independent alignment of each modality. The loss function is constructed as follows:

[0121]

[0122]

[0123] in, for One variant, namely the introduction Functions are used to cancel out potential noise interference, achieving robustness against relevant missing modes / modal noise. According to the formula Take from entity and Corresponding meta weights The minimum value (i.e., the minimum confidence value) is taken as the weight; for Another variant, China adopts An implicit inter-modal refinement (IIR) training objective was achieved, further enhancing the learning of dynamic weights.

[0124] in, express The two-stage fusion vector corresponding to the modality m dimension;

[0125] Ultimately, the training loss in the first phase is:

[0126]

[0127] Step 5: Construct a recurrent missing modality imagination module and build a two-stage training loss based on the recurrent missing modality imagination module.

[0128] The designed Circularly Missing Modality Imagination (CMMI) module emphasizes the generation of new models and their domain transferability, actively completing missing modality information. Specifically, it employs a variational multimodal autoencoder containing an encoder and a decoder, using trimodal hybrid features. As input, where as well as These represent entity feature representations for three modalities: relation, attribute, and graph structure, respectively, and use the output of the hidden layer between the encoder and decoder as imagined pseudo-visual features. Variational multimodal autoencoder for trimodal hybrid features The calculation process can be expressed by the formula:

[0129]

[0130]

[0131]

[0132] Among them, MLP Enc (·) indicates an encoding operation. Indicates decoding operation, μ i and σ i These represent the mean and variance of a normal distribution. ⊙ represents the concatenation operation, ⊙ represents the dot product operation, and z represents the normal distribution. This indicates the decoding result;

[0133] Define the reconstructed training loss and To minimize respectively and in, Representing real image features, the formula is expressed as:

[0134]

[0135]

[0136] Introducing KL (Kullback–Leibler) training loss The output of the guided encoding approximates a Gaussian distribution, which can be expressed by the formula:

[0137]

[0138] in, It is an entity with a complete image in a minimum batch, where i represents the entity index;

[0139] By analyzing trimodal mixture features The overall fusion vector similarity matrix is ​​obtained, and the similarity matrix is ​​distilled to virtual image features. In the similarity matrix, to preserve known information, construct... The loss function is expressed as:

[0140]

[0141] in, and They are represented as follows:

[0142]

[0143]

[0144]

[0145] This indicates a positive sample pair, which is a pair of entities that have been pre-aligned from two multimodal knowledge graphs. express The corresponding trimodal mixture characteristics, express The corresponding pseudo-visual feature, τ represents the temperature coefficient hyperparameter;

[0146] Finally, the two-stage loss function is:

[0147]

[0148] Step 6: Based on the first-stage training loss and the second-stage training loss, and using a modality-aware hard negative sample replay strategy, all modules are learned to achieve dynamic adaptive alignment of the modality knowledge graph.

[0149] In the embodiment, based on the first-stage training loss and the losses from the second phase of training The learning process for all modules is divided into two stages. Stage 1 uses... Only the main body of the parameters is optimized; Stage 2 adopts... and The joint training process involves: the main body being frozen, and then... Optimize CMMI module parameters, then freeze the CMMI module. The parameters of the main body are optimized. The main body includes the network for the feature encoding process, the network for the mapping process, the global modality aggregation module, and the modality alignment module at the entity level.

[0150] In this embodiment, a modality-aware hard negative sample replay strategy is also introduced. This involves defining the entities misaligned in each round of training as hard entities and adding them to the negative samples for contrastive learning training, thereby enhancing the difference in the contrastive signals. Specifically, this process can extend in-batch contrastive learning to search-based out-of-batch contrastive learning, using a strong negative sample mapping matrix. Achieve fast negative sample retrieval and localization. This strong negative sample mapping matrix M neg The system is dynamically updated in each batch during training. In each round, mismatched entity pairs are stored in the strong negative sample mapping matrix in a dictionary-like key-value pair manner, using the entity ID as the retrieval index. Matrix size... Sufficient to encompass the source knowledge graph (number of entities) ) and target knowledge graph (number of entities) All entities of ).

[0151] In the entity alignment process of two multimodal KGs, entities on one side may have missing images or differences in the emphasis of image content, which is detrimental to alignment. However, the method of this invention, through adaptive weights and multi-level alignment, can effectively mitigate these noises, resulting in the following: Figure 3 shown.

[0152] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic adaptive alignment method for multimodal knowledge graphs based on Transformer, characterized in that, Includes the following steps: Two given multimodal knowledge graphs are decomposed into multiple dimensions according to modal features, and the content features of each dimension are encoded to obtain the multimodal entity feature representation corresponding to each entity. The multimodal entity feature representation is then projected onto the same space. The dimensions include graph structure dimension, relationship dimension, attribute dimension, and visual image dimension. For the attribute and relation dimensions, a bag-of-words model is used to encode the content features corresponding to the attributes and relations; for the graph structure dimension, a graph attention model is used to encode the content features corresponding to the graph structure; for the visual image dimension, an image pre-trained encoder is used to encode the content features corresponding to the visual image; and a modality-adaptive fully connected layer is used to perform cospatial projection mapping on the entity feature representations corresponding to each dimension. A global modality aggregation module is constructed, which performs alignment and aggregation based on the multimodal entity feature representation after global adaptive weight projection mapping; Construct a modal alignment module at the entity level, and obtain a fusion vector by performing custom modal fusion based on an attention mechanism using Transformer; Based on alignment aggregation and fusion vectors, and introducing a modality-adaptive contrastive learning framework, a one-stage training loss is constructed. Construct a recurrent missing modality imagination module, and build a two-stage training loss based on the recurrent missing modality imagination module; Based on the first-stage training loss and the second-stage training loss, and using a modality-aware hard negative sample replay strategy to learn all modules, dynamic adaptive alignment of the modality knowledge graph is achieved.

2. The Transformer-based dynamic adaptive alignment method for multimodal knowledge graphs according to claim 1, characterized in that, In the global modality aggregation module, the following formula is used to perform the first alignment aggregation of the projected multimodal entity feature representations based on global adaptive weights: in, This represents the feature representation obtained from alignment aggregation. Representing modes m Global adaptive weights in two multimodal knowledge graphs Representing modes m The corresponding entity feature representation after projection mapping, This represents the modality set, or the dimension set. Indicates a weighted concatenation operation, subscript i The number representing the entity concept.

3. The method for dynamic adaptive alignment of multimodal knowledge graphs based on Transformer according to claim 1, characterized in that, The entity-level modal alignment module includes a dynamic cross-modal weighting submodule; In the dynamic cross-modal weighted submodule, modal weights are adaptively scored and modal fusion is performed based on the Transformer's attention mechanism, specifically as follows: Entity feature representation after each modal projection mapping Based on transformation function Mapping to independent Matrix, and calculate each attention head based on the matrix. attention weights and attention vector And the vector is obtained by concatenating the vectors after spatial mapping. This can be expressed as a formula: in, Representative mode m and j The mutual attention weights at each attention head, Representing the Query matrix transpose, and Representing modes j and modality l Key matrix , This represents the dimension of the vector in each attention head. Represents a set of modes. Indicates based on the first i Modalities of attention head computation m and j Attention weights between them Indicates modality j The calculated Value matrix V, This indicates a splicing operation. Represents the transformation matrix. Indicates the number of attention heads; Introducing methods for stabilizing model training and residuals to vector Calculations are performed to obtain the first-stage fusion vector. This can be expressed as a formula: Adding a feedforward neural network for two-stage fusion yields a two-stage fusion vector. This can be expressed as a formula: in, and Represents the learnable transformation matrix. and This represents a learnable bias.

4. The Transformer-based dynamic adaptive alignment method for multimodal knowledge graphs according to claim 3, characterized in that, It also includes: extracting attention weights from the dynamic cross-modal weighted submodule. After accumulation, each modality is obtained through SoftMax operation. Meta weights This can be expressed as a formula: in, Indicates the number of modes.

5. The Transformer-based dynamic adaptive alignment method for multimodal knowledge graphs according to claim 4, characterized in that, The first-stage training loss is constructed using a contrastive learning framework based on alignment aggregation and fusion vectors, and incorporating modality adaptation. This framework includes: The learning objective of the contrastive learning framework is defined as: in, , This indicates a positive sample pair, which is a pair of entities that have been pre-aligned from two multimodal knowledge graphs. express In modality m Entity feature representation corresponding to the dimension Represents the set of negative samples. Indicates originating from And with Form negative sample pairs, This indicates the temperature coefficient hyperparameter. i Index representing the original entity concept, Indicates positive sample pairs Alignment probability; Considering that the alignment process is bidirectional, a bidirectional alignment training objective is defined. for: When bidirectional alignment training target When adapting to global modality alignment, construct the loss function. for: in, use ,according to , can be obtained , This refers to what was mentioned earlier. The corresponding result is the alignment and aggregation of the multimodal entity feature representations after projection mapping based on global adaptive weights; A confidence-enhancing intra-modal alignment strategy is introduced to achieve independent alignment of each modality. The loss function is constructed as follows: = in, for One variant, namely the introduction Functions are used to cancel out potential noise interference, achieving robustness against relevant missing modes / modal noise. According to the formula Take from entity and Corresponding meta weights The minimum value among them is used as the weight; for Another variant, China adopts , ,in, express In modality m The two-stage fusion vector corresponding to the dimension; Ultimately, the training loss in the first phase is:

6. The Transformer-based dynamic adaptive alignment method for multimodal knowledge graphs according to claim 1, characterized in that, The two-stage training loss constructed based on the recurrent missing mode imagination module includes: The cyclic missing modality imagination module employs a variational multimodal autoencoder containing an encoder and a decoder, using trimodal hybrid features. As input, where , as well as These represent entity feature representations for three modalities: relation, attribute, and graph structure, respectively, and use the output of the hidden layer between the encoder and decoder as imagined pseudo-visual features. Variational multimodal autoencoders for trimodal mixed features The calculation process can be expressed by the formula: in, Indicates encoding operation. Indicates a decoding operation. and These represent the mean and variance of a normal distribution. This indicates a splicing operation. This represents the dot product operation. Indicates that it follows a normal distribution , This indicates the decoding result; Define the reconstructed training loss and To minimize respectively and ,in, Representing real image features, the formula is expressed as: = Introducing KL training loss The output of the guided encoding approximates a Gaussian distribution, which can be expressed by the formula: in, It is an entity with a complete image in the smallest batch. i Represents an entity index; By analyzing trimodal mixture features The overall fusion vector similarity matrix is ​​obtained, and the similarity matrix is ​​distilled to virtual image features. In the similarity matrix, to preserve known information, construct... The loss function is expressed as: in, and They are represented as follows: , , This indicates a positive sample pair, which is a pair of entities that have been pre-aligned from two multimodal knowledge graphs. express The corresponding trimodal mixture characteristics, express The corresponding pseudo-visual features, Indicates the temperature coefficient hyperparameter; Finally, the two-stage loss function is: 。 7. The Transformer-based dynamic adaptive alignment method for multimodal knowledge graphs according to claim 1, characterized in that, The learning process for all modules is divided into two stages. In the first stage, a one-stage loss function is used to optimize the parameters of the main body only. In the second stage, the one-stage loss function and the two-stage loss function are used for joint training. Specifically, the main body is frozen, and the parameters of the recurrent missing modality imagination module are optimized using the two-stage loss function. Then, the recurrent missing modality imagination module is frozen, and the parameters of the main body are optimized using the one-stage loss function. The main body includes the network for feature encoding, the network for mapping, the global modality aggregation module, and the modality alignment module at the entity level.

8. The Transformer-based dynamic adaptive alignment method for multimodal knowledge graphs according to claim 1, characterized in that, The modality-aware hard negative sample replay strategy is as follows: During training, the entity misaligned in each round is defined as a hard entity, and this hard entity is added to the negative samples to participate in contrastive learning training. Specifically, the intra-batch contrastive learning is extended to a search-based out-of-batch contrastive learning, using a strong negative sample mapping matrix. To achieve negative sample retrieval and localization, this strong negative sample mapping matrix... The system is dynamically updated in each batch during the training process, and in each round, mismatched entity pairs are stored in the strong negative sample mapping matrix as dictionary key-value pairs.

Images