Multi-modal entity feature alignment and fusion method and device, and electronic device

Abstract

The application discloses a kind of multimodal entity alignment methods, specifically is related to a kind of multimodal entity feature alignment fusion method, device and electronic equipment, method includes: the knowledge graph to be aligned is input into multimodal embedding module and is encoded, obtains multimodal embedding, the multimodal embedding is input into cross-modal attention fusion module and is extracted, obtains cross-modal attention fusion embedding, and the multimodal embedding is input into multimodal adaptive feature fusion module and is fused, obtains adaptive feature fusion embedding;Based on multimodal early fusion embedding, cross-modal attention fusion embedding and adaptive feature fusion embedding, the entity in the knowledge graph to be aligned is aligned and fused, wherein, multimodal early fusion embedding is obtained by cascading fusion of multimodal embedding in embedding layer. This method can solve the problem of weak modal fusion imbalance and semantic ambiguity when aligning multimodal entity features.

Description

Technical Field

[0001] This invention relates to a multimodal entity alignment method, specifically to an alignment and fusion method, apparatus, and electronic device for multimodal entity features. Background Technology

[0002] A knowledge graph (KG) is composed of relational facts and entities, and is essentially a large-scale semantic web. Traditional knowledge graphs are mainly composed of text; however, text-based unimodal knowledge graphs are often incomplete, lacking joint representations of heterogeneous knowledge such as images, videos, and audio. In recent years, a significant trend has been to construct multimodal knowledge graphs (MMKGs) by integrating multimodal data, such as visual data, into knowledge graphs.

[0003] The real world contains a vast amount of knowledge, but most MMKGs are typically constructed from a single data source or in a single language. This results in a scarcity of triple data for some entities, or even the absence of modalities. These entities with sparse relationships and missing modalities are called long-tail entities. MMKGs constructed from different data sources mean that entities with the same semantic meaning may exist in different MMKGs. If entities with the same semantic meaning can be merged, it can effectively supplement long-tail entity data and enhance the semantic representation capabilities of MMKGs. To achieve this, a common strategy is to integrate the modal information of different MMKGs into a unified MMKG to achieve entity matching, thereby expanding the coverage of knowledge. This technique of using visual, attribute, and numerical modalities to match equivalent entities of the same object in the real world across different MMKGs is called Multi-modal Entity Alignment (MMEA).

[0004] MMEA is a complex technique, and current MMEA methods mainly focus on designing a cross-modal fusion paradigm. However, these methods concentrate on directly merging single-modal features after using different modal embeddings, without considering cross-modal information interaction during multimodal fusion. This ignores the intramodal differences and entity sparsity of MMKG, leading to imbalance and semantic ambiguity problems in the fusion of weak modalities such as vision. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the above-mentioned methods commonly used in the prior art directly merge single-modal features after using different modal embeddings, without considering cross-modal information interaction during multimodal fusion. This ignores the intramodal differences and entity sparsity of MMKG, resulting in imbalance and semantic ambiguity in the fusion of weak modalities such as vision. In order to solve the above problems, the present invention provides a method, apparatus and electronic device for aligning and fusing multimodal entity features.

[0006] The content of this invention includes:

[0007] In a first aspect, embodiments of the present invention provide a method for aligning and fusing multimodal entity features, including:

[0008] The knowledge graph to be aligned is input into the multimodal embedding module for feature encoding to obtain the multimodal embedding. The knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding and visual embedding.

[0009] The multimodal embedding is input into the cross-modal attention fusion module for feature fusion and extraction to obtain the cross-modal attention fusion embedding, and the multimodal embedding is input into the multimodal adaptive feature fusion module for feature fusion to obtain the adaptive feature fusion embedding.

[0010] The entities in the knowledge graph to be aligned are aligned and fused based on the multimodal early fusion embedding, the cross-modal attention fusion embedding, and the adaptive feature fusion embedding, wherein the multimodal early fusion embedding is obtained by cascading and fusing the multimodal embeddings in the embedding layer.

[0011] Optionally, the multimodal embedding includes the structural embedding, the multimodal embedding module includes a graph attention network, and the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0012] The graph structure modal data in the knowledge graph to be aligned is input into the graph attention network to extract graph structure features and obtain the structure embedding.

[0013] Optionally, the multimodal embedding includes the text embedding, the multimodal embedding module includes a fully connected feature extraction network, and the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0014] The text modal data in the knowledge graph to be aligned is input into the corresponding fully connected feature extraction network to extract text features, thereby obtaining the text embedding.

[0015] Optionally, the multimodal embedding includes the visual embedding, the multimodal embedding module includes RESNET-15, and the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0016] The visual modal data in the knowledge graph to be aligned is input into the RESNET-152 for visual feature extraction to obtain the visual embedding.

[0017] Optionally, the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0018] The knowledge graph to be aligned is input into the multimodal embedding module for feature encoding to obtain the multimodal embedding, thus obtaining the initial multimodal embedding.

[0019] Gaussian mask noise is added to the initial multimodal embedding to obtain the multimodal embedding.

[0020] Optionally, the method further includes:

[0021] The feature-balanced multimodal entity alignment model is iteratively trained, wherein the feature-balanced multimodal entity alignment model includes the multimodal embedding module, the cross-modal attention fusion module, and the multimodal adaptive feature fusion module;

[0022] The loss function for iterative training includes a multimodal joint contrastive loss and a multimodal joint embedding alignment loss, wherein the multimodal joint contrastive loss satisfies:

[0023]

[0024] in, The contrastive loss, α, is used to characterize the multimodal fusion embedding. m For hyperparameters, M = {g, r, a, n, v} is used to characterize the single-modal contrastive loss and to characterize the single-modal embedding.

[0025] The multimodal joint embedding alignment loss satisfies:

[0026]

[0027] in, The alignment loss, β, is used to characterize the multimodal fusion embedding. m For hyperparameters, Used to characterize single-modal alignment loss.

[0028] Optionally, the total loss function of the iterative training satisfies:

[0029]

[0030] Secondly, embodiments of the present invention provide an alignment and fusion apparatus for multimodal entity features, comprising:

[0031] The feature encoding module is used to input the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain multimodal embedding. The knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding and visual embedding.

[0032] The feature fusion module is used to input the multimodal embedding into the cross-modal attention fusion module for feature fusion and extraction to obtain the cross-modal attention fusion embedding, and to input the multimodal embedding into the multimodal adaptive feature fusion module for feature fusion to obtain the adaptive feature fusion embedding;

[0033] The alignment and fusion module is used to align and fuse entities in the knowledge graph to be aligned based on the multimodal early fusion embedding, the cross-modal attention fusion embedding, and the adaptive feature fusion embedding, wherein the multimodal early fusion embedding is obtained by cascading and fusing the multimodal embeddings in the embedding layer.

[0034] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps in the multimodal entity feature alignment and fusion method as described in the first aspect.

[0035] Fourthly, embodiments of the present invention provide a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the multimodal entity feature alignment and fusion method described in the first aspect.

[0036] In this embodiment of the invention, the knowledge graph to be aligned is input into a multimodal embedding module for feature encoding to obtain a multimodal embedding. The knowledge graph to be aligned includes multimodal data. The multimodal embedding is input into a cross-modal attention fusion module for feature fusion and extraction to obtain a cross-modal attention fusion embedding. The multimodal embedding is then input into a multimodal adaptive feature fusion module for feature fusion to obtain an adaptive feature fusion embedding. Entities in the knowledge graph to be aligned are aligned and fused based on the early multimodal fusion embedding, cross-modal attention fusion embedding, and adaptive feature fusion embedding. The beneficial effects of this invention are that it fully utilizes the complementarity between different modalities for step-by-step fusion, thereby enhancing the robustness of entity alignment, mitigating the impact of long-tail entity data sparsity, enabling the model to better adapt to the data characteristics of different modalities, and solving problems such as insufficient cross-modal effects and unbalanced cross-modal fusion. Attached Figure Description

[0037] Appendix Figure 1 A flowchart illustrating the alignment and fusion method for multimodal entity features provided in an embodiment of the present invention;

[0038] Appendix Figure 2a This is a schematic diagram of the overall structure of the FBMEA model provided in an embodiment of the present invention;

[0039] Appendix Figure 2b for Figure 2aOne of the magnified schematic diagrams of a localized FBMEA model;

[0040] Appendix Figure 2c for Figure 2a Part 2 of the enlarged schematic diagram of the FBMEA model in China;

[0041] Appendix Figure 3 Example diagram of multimodal entity alignment;

[0042] Appendix Figure 4a One of the schematic diagrams of the ablation experimental results of FBMEA on the DBP15KZH-EN dataset;

[0043] Appendix Figure 4b The second illustration shows the ablation experimental results of FBMEA on the DBP15KZH-EN dataset;

[0044] Appendix Figure 4c One of the schematic diagrams showing the ablation experimental results of FBMEA on the DBP15KJA-EN dataset;

[0045] Appendix Figure 4d The second illustration shows the ablation experimental results of FBMEA on the DBP15KJA-EN dataset;

[0046] Appendix Figure 5 A schematic diagram of an alignment and fusion device for multimodal entity features provided in an embodiment of the present invention;

[0047] Appendix Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] In this application's embodiments, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. In this application's embodiments, the term "multiple" refers to two or more, and other quantifiers are similar.

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0050] Please see Figure 1 , Figure 1This is a flowchart illustrating the alignment and fusion method for multimodal entity features provided in an embodiment of the present invention. The method specifically includes the following steps:

[0051] Step 101: Input the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain multimodal embedding. The knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding and visual embedding.

[0052] Step 102: Input the multimodal embedding into the cross-modal attention fusion module for feature fusion and extraction to obtain the cross-modal attention fusion embedding; and input the multimodal embedding into the multimodal adaptive feature fusion module for feature fusion to obtain the adaptive feature fusion embedding.

[0053] Step 103: Align and fuse entities in the knowledge graph to be aligned based on the multimodal early fusion embedding, the cross-modal attention fusion embedding, and the adaptive feature fusion embedding, wherein the multimodal early fusion embedding is obtained by cascading and fusing the multimodal embeddings in the embedding layer.

[0054] To facilitate understanding, the symbol parameters involved in the embodiments of this application will be explained below. A Knowledge Graph (KG) is composed of relational facts and entities, and is essentially a large-scale semantic network. KG uses entities and concepts as nodes, and the semantic relationships between nodes as edges, represented as G = (E, R, T), where E, R, and T represent the entity, relation, and triple set, respectively. The triple T = {(h, r, t) | h, t ∈ E, r ∈ R}, where h represents the head entity, t represents the tail entity, and r represents the relation between the head and tail entities.

[0055] Multimodal knowledge graphs are formalized as directed graphs MMKG = {E, R, A, V, T}, where E represents the entity set, R represents the relation set, A represents the attribute set, V represents the image set, and T represents the triple set. Two multimodal knowledge graphs to be aligned can be represented as MMKG1 = {E1, R1, A1, V1, T1} and MMKG2 = {E2, R2, A2, V2, T2}.

[0056] During model training, the pre-aligned seed entity pair set Seed is used as training data, represented as Seed={(e s1 e s2 )|e s1 ∈E1,e s2 ∈E2,e s1 ~e s2The goal of multimodal entity alignment is to design a multimodal alignment framework based on the proportion of the seed entity set in the total entity set and the characteristics of entity pairs, and to find the set of entity pairs in MMKG1 and MMKG2 that represent the same objective object in the real world: Align = {(e1, e2) | e1∈E1, e2∈E2}. ， The expression `e1~e2}` represents an equivalence relation.

[0057] The multimodal entity feature alignment and fusion method provided in this invention can also be called the Feature Balance Method for Multi-modal Entity Alignment (FBMEA) method. This invention also provides an FFBMEA model, specifically as follows: Figures 2a-2c As shown, solid lines represent the relationships between entities, while dashed lines represent the attribute information of entities across different graphs.

[0058] It should be understood that there must be at least two knowledge graphs to be aligned, and the data modalities contained in these two knowledge graphs can be the same or different. For details, please refer to [link / reference needed]. Figure 3 In the multimodal embedding module, feature encoding is performed on the multimodal data of the knowledge graph to be aligned, resulting in multimodal embeddings. The feature extraction method varies depending on the data modality.

[0059] Optionally, in some embodiments, the multimodal embedding includes the structural embedding, the multimodal embedding module includes a graph attention network, and the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0060] The graph structure modal data in the knowledge graph to be aligned is input into the graph attention network to extract graph structure features and obtain the structure embedding.

[0061] The purpose of graph embedding is to map high-dimensional dense matrix knowledge graph data into low-dimensional dense vectors. In this embodiment, a graph attention network (GAT) is used to extract graph structure features from the knowledge graph.

[0062] Specifically, entity e i The aggregate hidden layer of a one-hop neighbor is represented as (d represents the hidden layer dimension), the self-attention coefficient of each node. To represent the importance of node j to node i, attention is introduced into the graph structure, and the set of one-hop neighbors N of node i is assigned. i Regularization can be expressed as:

[0063]

[0064] Attention mechanism α ij It is a single-layer feedforward neural network. LeakyReLU is added to the output layer to obtain a complete attention mechanism.

[0065]

[0066] in, This indicates a join operation, where T represents transpose. Represents the weight matrix. This represents a learnable vector. To stabilize the self-attention learning process, the attention mechanism is extended to multi-head attention, employing a linear transformation of the weight matrix W. k Replacing the join operation, the final embedded structure representation is as follows:

[0067]

[0068] In the formula, The normalized attention mechanism for the k-th attention computation typically employs a two-layer GAT model to aggregate the domain structure, with the output of the last GAT layer used as the structural embedding.

[0069] Optionally, in some embodiments, the multimodal embedding includes the visual embedding, the multimodal embedding module includes RESNET-15, and the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0070] The visual modal data in the knowledge graph to be aligned is input into the RESNET-152 for visual feature extraction to obtain the visual embedding.

[0071] In this embodiment, the pre-trained model RESNET-152 is used instead of the VGG16 in the dataset to extract visual features. Compared to VGG16, RESNET-152 has fewer parameters, delivers better results, accelerates neural network training, and significantly improves model accuracy. RESNET-152 performs a forward pass for each image, using the final layer output before logits as image features. These features are then sent through a trainable feedforward layer, resulting in a visual embedding that satisfies the following conditions:

[0072]

[0073] Among them, Ing i This represents the i-th image as input. and These are learnable hyperparameters.

[0074] Optionally, in some embodiments, the multimodal embedding includes the text embedding, the multimodal embedding module includes a fully connected feature extraction network, and the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0075] The text modal data in the knowledge graph to be aligned is input into the corresponding fully connected feature extraction network to extract text features, thereby obtaining the text embedding.

[0076] It should be understood that, in order to alleviate the problem of mixed text feature relationships, this embodiment sets up a separate fully connected feature extraction network for each text modality. In this embodiment, the cross-language entity alignment method of bag of words is followed for feature representation, and the features are input into the feedforward neural network to establish the relationship between the modalities in the low-dimensional space.

[0077] The text embedding obtained by the above method satisfies:

[0078]

[0079] Where m∈{r, a, n} represents the relation, attribute, and name (surface form) modality. and For learnable hyperparameters, F m This represents the bag-of-words features of each modality.

[0080] MMEA suffers from widespread issues of modality loss and visual-text blurring, leading to overfitting in the multimodal fusion framework. Optionally, in some embodiments, the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0081] The knowledge graph to be aligned is input into the multimodal embedding module for feature encoding to obtain the multimodal embedding, thus obtaining the initial multimodal embedding.

[0082] Gaussian mask noise is added to the initial multimodal embedding to obtain the multimodal embedding.

[0083] In this embodiment, introducing irrelevant document noise can improve cross-modal alignment capability. Specifically, in this embodiment, Gaussian mask noise is added to the multimodal embedding to enhance modal feature representation, better adapt to noise and uncertainty existing in the real world, and enhance generalization capability.

[0084] The resulting multimodal embedding satisfies:

[0085]

[0086] Where ε is the noise ratio, {r, a, v} represents the relationship, attribute, and visual modality, and γ represents the relative intensity of noise and modal features. m With μ m represents the standard deviation and mean of the noise data for a specific modality, respectively. n The samples are extracted from a Gaussian distribution N(0,1), ensuring that the introduced noise is consistent with the inherent statistical characteristics of each mode. In subsequent embodiments, It is considered an embedding with added noise.

[0087] In this embodiment, Gaussian noise is added to the MMKG fusion training to learn the features that are insensitive to noisy data, thereby better adapting to changes in different modal data and enhancing the generalization performance of the model.

[0088] The multimodal embedding is input into the multimodal adaptive feature fusion module for feature fusion, and the specific method for obtaining the adaptive feature fusion embedding is described below:

[0089] Knowledge graphs contain a large number of long-tail entities with scarce structural information. Therefore, semantic information weights need to be increased when aligning long-tail entities. To fully realize the value of each modality and balance the impact of low-quality multimodal data, this application employs a structural and semantic similarity matrix for dynamic feature fusion at this stage, utilizing cosine similarity measurement of different modal embeddings in the multimodal knowledge graph. Semantic similarity.

[0090]

[0091] in, and These represent embedded structural information and embedded unstructured information, respectively. This represents the structural similarity between the i-th entity in the source knowledge graph and the j-th entity in the target knowledge graph. For unstructured semantic information embedding... The calculation using the cascade method is as follows:

[0092]

[0093] Among them, w m These are the trainable weights for modality m. Before weighted concatenation, L2 normalization is performed on each input feature matrix to obtain unstructured semantic embeddings. The structural similarity matrix is ​​then used as the basis for... The semantic similarity matrix was calculated in the same way. Fusion and The final similarity score M is obtained. ij :

[0094]

[0095] Where wg represents the weight of structural information and w′s represents the weight of semantic information. To better adjust the contribution of structural and semantic information to entity alignment, this embodiment employs a degree-aware entity alignment method, dynamically assigning weights to different features, specifically as follows:

[0096]

[0097] w g =1-w s .

[0098] Where K, b, and α are hyperparameters, degree is the degree of the entity, and N hop N represents the degree of association between an entity and its seed entity. hop The calculation is as follows:

[0099] N hop =n 1-hop ×w1+lg(n 2-hop ×w2)

[0100] Where, n 1-ho p and n 2-hop w1 and w2 represent the number of entities within 1 hop and 2 hops of the seed entity, respectively, and are hyperparameters. The method provided in this application can effectively balance semantic embedding and structural embedding, adaptively calculating the structural embedding weights w. g With semantic embedding weight w s Thus, entity e is calculated. i Adaptive feature fusion embedding:

[0101]

[0102] Finally, based on the similarity score M ij Candidate entity pairs are ranked by similarity, and entities with high modal similarity are selected as aligned entity pairs to comprehensively capture the semantic information of entities.

[0103] It should be understood that the similarity score here is prior information for entity alignment. This part uses adaptive feature fusion to obtain multimodal fusion embedding, while the similarity score is obtained through structural modality and semantic modality similarity. In this embodiment, entity pairs with high similarity scores between different entities are aligned.

[0104] In the adaptive feature fusion module, similarity scores are used to guide the entity alignment task, selecting only candidate entity pairs with high similarity for alignment. Simultaneously, the text also utilizes an iterative learning strategy, adding high-confidence entity pairs to seed entity pairs for training, thus addressing the issues of insufficient seed entity information and high costs associated with manual annotation.

[0105] The specific process of inputting the multimodal embedding into the cross-modal attention fusion module for feature fusion and extraction to obtain the cross-modal attention fusion embedding is described below:

[0106] In this embodiment, dynamic cross-modal weighting of MEAformer and UMAEA is introduced. This network can dynamically adjust the meta-weights of each modality entity. After single-modal embedding, multi-head convolutional attention (MHCA) and a fully connected feedforward neural network (FFN) are added.

[0107] MHCA in N h The attention function runs on each of the i-th parallel heads, and the i-th head is parameterized by the modality sharing matrix. (where d is the input modality dimension, d h =d / N n (where the key vector dimension is used to scale the dot product to stabilize the softmax function), and the multimodal input h is... m Convert to modality-aware query key Sum Then, by combining the inputs from all heads through a linear transformation, the modality feature output is as follows:

[0108]

[0109] Among them, W o ∈R d×d , The attention weights of an entity between modal m and modal j can be expressed as:

[0110]

[0111] Where, d h =d / N h The training is stabilized using Layer Normalization (LN) and Residual Connection (RC). FFN consists of two linear transformation layers and a ReLU activation function, which are applied to LN and RC as follows:

[0112]

[0113] in, d in Let b1 and b2 represent the cross-modal attention feature dimensions, where b1 and b2 are learnable parameters. The meta-weight w represents the output weight for each modality. m Represented as:

[0114]

[0115] Here, M represents the set of available modalities. It should be understood that |M| is used to account for the similarity between different modalities, and negative values ​​are avoided during the calculation.

[0116] Entity e i The joint embedding is represented as:

[0117]

[0118] This is for entity e in the FBMEA specification. i The cross-modal attention fusion output (i.e., fusing different multimodal information) is obtained. In the cross-modal attention fusion module, the cross-modal joint embedding (which is shown in the framework diagram in Figure 2, part of which is obtained by the meta-weights of the cross-modal attention output and the feature output of the Transformer, and the other part is obtained by the meta-weights of the cross-modal attention output and the multimodal embedding features) is then aligned and discriminated.

[0119] Optionally, the method further includes:

[0120] The feature-balanced multimodal entity alignment model is iteratively trained, wherein the feature-balanced multimodal entity alignment model includes the multimodal embedding module, the cross-modal attention fusion module, and the multimodal adaptive feature fusion module;

[0121] The loss function for iterative training includes multimodal joint contrastive loss and multimodal joint embedding alignment loss.

[0122] This application employs multimodal contrastive representation learning to construct an adaptive contrastive loss L within the modal feature channels. icl This loss function leverages the similarity of the original embedding space to identify the same entities in different KGs. During entity representation, weak modalities may more easily exert an excessive influence on the overall model. To achieve enhanced fusion between modalities and avoid imbalanced feature fusion, this application employs a multi-angle design of a multi-modal contrastive learning module, combining overall modal representation with joint representation of single modalities, to fully exploit the information contained in each modality within a finite seed pair.

[0123] Each pair of seed entities in the model Entities that can be considered positive samples are considered positive samples, while other entity pairs without relational links are considered negative samples. Entities follow a 1-to-1 alignment constraint. Define the i-th entity in the minimum batch S. Positive samples can be represented as:

[0124]

[0125] Negative samples consist of two parts: the unaligned portion within the KG and the unaligned portion outside the KG, which can be represented as:

[0126]

[0127] By limiting the range of N samplings to a small batch, the probability of alignment is defined as:

[0128]

[0129] in, Let be the encoder for mode m, τ represent the temperature hyperparameter, and T represent the transpose. Before calculating the inner product, the embedded input features are L2 normalized. The single-modal contrastive loss can be expressed as:

[0130]

[0131] Single-modal contrastive loss is performed during contrastive supervised learning, applying it separately to each modal channel. A single-modal contrastive loss is used to make the boundaries of each modality in the data embedding space more discriminative. Simultaneously, a multi-modal fusion embedding h is performed by weighted fusion of modalities such as visual, attribute, and name (M = {g, r, a, n, v}). joint (Multimodal early fusion embedding h) i Adaptive feature fusion embedding Embedded with cross-modal attention )

[0132] It should be understood that the multimodal fusion embeddings of different embedding methods here are all obtained by weighting the single-modal embeddings. The weights corresponding to each modality can be obtained through the methods described above. All three embedding fusion methods use graph structure, relations, attributes, names, and vision for joint representation.

[0133] For multimodal fusion embedding, contrastive loss is used, and a hyperparameter α is added. m The obtained multimodal joint embedding contrastive loss L icl It can be represented as:

[0134]

[0135] in, The contrastive loss, α, is used to characterize the multimodal fusion embedding. m For hyperparameters, The contrast loss is used to characterize unimodal embeddings, where M = {g, r, a, n, v} represents the unimodal embeddings of structure, relation, attribute, name, and vision. The contrast loss of the multimodal joint embedding is determined by the contrast loss of the multimodal fusion embedding and the contrast loss of the unimodal embedding.

[0136] It should be understood that the joint embedding here refers to the previous multimodal feature fusion. In particular, it should be noted that the cross-modal attention fusion consists of two parts: one part is obtained by combining the meta-weights of the cross-modal attention output with the feature output of the Transformer, and the other part is obtained by combining the meta-weights of the cross-modal attention output with the initial multimodal embedding features (see Figure 2 for details).

[0137] In classification tasks, Neighborhood Component Analysis (NCA) is used to design the NCA loss. It uses local and global statistical measures of sample importance and employs a soft-weighting scheme to penalize hard negations to mitigate embedding space shifts. The NCA loss uses cosine similarity to calculate the alignment loss based on entity similarity, attribute similarity, and neighbor dissimilarity. Multi-view evaluation assesses the consistency of entity alignment, and the single-modal alignment loss is calculated as follows:

[0138]

[0139] in, Indicates alignment entity Cosine similarity of unimodal m-or joint embeddings, where α and β represent hyperparameters, and N represents the minimum batch size. Comparison with adaptive contrastive loss L. icl Similarly, this embodiment applies the loss to other modal alignment channels and calculates the multimodal joint characterization loss.

[0140] Add hyperparameter β m The multimodal joint embedding alignment loss is calculated as follows:

[0141]

[0142] in, The alignment loss, β, is used to characterize the multimodal fusion embedding. m For hyperparameters, This is used to characterize the alignment loss of a single modality. Here, it represents the alignment loss between entities after multimodal fusion.

[0143] FFBMEA is viewed as a multi-task learning model, treating each type of loss as a task. It learns by sharing feature representations and leverages the correlation and information interaction between tasks to improve the model's generalization performance and efficiency. α m β m There are two types of balancing losses, but manually tuning these parameters is expensive and time-consuming. This application treats FFBMEA as a multi-task learning paradigm, utilizing variance uncertainty to automatically adjust the weights between various losses during model training. The multi-task loss function is derived by maximizing the Gaussian likelihood under task uncertainty.

[0144] Optionally, in some embodiments, the loss function of the iterative training satisfies:

[0145]

[0146] In this embodiment, an optimization scheme is designed that utilizes intra-state loss from multimodal contrastive learning and inter-state loss from multimodal alignment to optimize the objective. This optimization scheme decouples different modes, making the model more independent and controllable in the process of learning different modes.

[0147] As shown in Figure 2, this method encodes the features of each modality independently through a multimodal embedding channel in noisy multimodal data. Then, the multimodal embeddings are input into a Transformer for cross-modal attention fusion and extraction, dynamically adjusting modality preferences to achieve finer-grained entity alignment. Simultaneously, the multimodal embeddings are input into an adaptive feature fusion module to adjust the fusion of structural and semantic embeddings, increasing the weight of semantic information and balancing the problems of sparse structural embeddings and low graph coverage. Finally, a joint embedding is learned using contrastive loss and alignment loss. This method combines adaptive feature fusion with a multi-head attention mechanism, using multi-level fusion to mitigate the imbalance problem in feature fusion and adjusting structural information embedding to reduce the influence of long-tail entities, providing richer semantic representations for MMKG.

[0148] The beneficial effects of the method provided in this application will be verified through experiments below.

[0149] (1) Dataset

[0150] Experiments were conducted using five multimodal entity alignment public datasets, including three bilingual datasets DBP15K (ZH-EN, JA-EN, FR-EN) and two cross-knowledge graph datasets FB15K-DB15K and FB15K-YAGO15K. To compare with baseline performance, 20% of the entities provided in the DBP15K dataset were used as pre-aligned seed entity pairs. For the cross-knowledge graph datasets, 20%, 50%, and 80% of the seed entity pairs were used for the experiments.

[0151] (2) Baseline

[0152] The FFBEA method provided in this embodiment of the invention is compared with nine current mainstream baseline models. They can be mainly divided into two categories: entity alignment methods with graph structure as the mainstay and entity methods with multimodal fusion paradigm. The former only uses knowledge graph structure, while the latter uses other modalities to assist in the alignment task.

[0153] (3) Implementation details

[0154] To achieve cross-modal information interaction and learning, and to ensure that each modality is fully preserved in space, we will modify the hidden layer h of the GAT graph attention network.g With other modes h m The embedding layer was set to size 300, and the entity similarity threshold θ = 0.6. Similar to MCLEA, the AdamW optimizer was used with a 2×10⁻⁶ optimization. -5 The learning rate is used to update the parameters, and 1000 epoc training cycles are set, with a batch size of 1024. To be consistent with the current baseline method, we adopt the name translation and word vectors provided by Xu et al.

[30] . The visual embedding uses the visual preprocessing model of ResNet-152 to extract features with a dimension of 2048, which are then mapped to a 300-dimensional feature vector. Features are extracted across the graph dataset using VGG-16 with a dimension of 4096. For the adaptive feature fusion strategy, we use the dimension d g The parameters are set to 300, with hyperparameters b = 1.5, α = 1, w1 = 0.8, w2 = 0.1, and K depending on the seed set size. γ is set to 0.8. Similar to current mainstream MMEA, FBMEA employs an iterative learning strategy, adding each pair of intersecting entities that are each other's nearest neighbors to the candidate list. If the extracted entities are still each other's nearest neighbors in K rounds, they are added to the training set.

[0155] (4) Evaluation indicators

[0156] The model was evaluated using Hit@N (N=1, 10) (the ratio of hits in the top k recommendations to the total number of test samples) and MRR (mean reciprocal ranking), both common metrics in multimodal entity alignment tasks. The best results are indicated in bold, and the second-best results are indicated by underscores.

[0157] Please refer to Table 1, which shows the alignment performance of FBMEA on the bilingual dataset DBP15K using surface form and iterative training. Currently, for the bilingual dataset, bold text indicates the optimal model, underlined text indicates the suboptimal model, and * indicates the model's replication result under equivalent computational conditions. The baseline model is already close to ideal performance, and FBMEA still achieves a certain degree of improvement. Compared to the baseline MEAformer, FBMEA improves performance by 1.3% and 2.4% on the ZH-EN and JA-EN datasets His@1, respectively, but the improvement is insufficient on FR-EN. The MMR metric improves by 0.01% to 0.1%, indicating that FBMEA can approach perfect performance on bilingual datasets. To test the model's robustness, we further validate it on cross-graph datasets.

[0158] Table 1 shows the entity alignment results of the bilingual dataset with a seed training data ratio of 20%, using surface information and iterative methods.

[0159]

[0160] Table 2 shows the entity alignment performance of FFMEA on the cross-graph datasets FB15K-DB15K and FB15K-YAGO15K with different seed entity pair ratios. In Table 2, X% represents the percentage of aligned entity pairs used for training in the dataset, and * indicates the model's reproducibility under the same computational conditions.

[0161] Under various conditions, FBMEA achieved the best experimental results in almost all scenarios. When given 20% entity pairs, FBMEA's Hits@1 metric improved by 0.7% and 5.6% compared to the best comparative model, showing the most significant improvement, indicating that FBMEA can adapt well to low-resource environments. When given 50% and 80%, the performance improvements in Hits@1 and MRR remained significant. Furthermore, FBMEA showed even greater improvements on the FB15K-YAGO15K dataset, with the most significant MRR increase (8.4%, 3.6%, and 1.7%), indicating that FBMEA can significantly improve the link prediction capability and entity alignment accuracy of MMKG, alleviating the sparsity of MMKG. On the FB15K-DB15K dataset, the MRR metric continued to improve. Of particular note, we found that the Hits@10 on the FB15K-DB15K dataset is slightly lower than that of MEAformer. This indicates that while FFMEA can accurately retrieve results, its performance is insufficient for a broader query range. We speculate that this is because adaptive feature fusion weakens the structured information of long-tail entities, resulting in a reduced Hits@10 candidate set. Bilingual datasets and cross-graph datasets differ significantly in attributes and relationships, exhibiting considerable heterogeneity. For example, the EVA model is insufficient in cross-graph datasets, providing direct evidence for the generalization performance and model stability of FFMEA.

[0162] Table 2 shows the alignment results across the map dataset under iterative conditions.

[0163]

[0164] The ablation experiment will now be conducted.

[0165] We conducted component ablation experiments on a bilingual dataset to demonstrate the Hits@1 / MRR gains brought by different components. The results are as follows: Figures 4a-4dAs shown in the diagram, we first explored the impact of removing each modality on the results. We found that removing any modality leads to a performance degradation, with surface forms (entity names, etc.) being the most significant influencing modality. Removing surface forms resulted in the most severe degradation, followed by visual modalities, a result consistent with previous studies. Furthermore, we examined different optimization schemes and entity alignment fusion frameworks, including AFF (Adaptive Feature Fusion) and CAF (Cross-Modal Attention Fusion). We found that the CAF entity alignment scheme exhibited better performance, and joint learning combined with AFF adaptive fusion effectively improved alignment accuracy, thus validating the effectiveness of FBMEA. ICL and NCA showed similar performance. ICL's contrastive learning optimization focuses more on intra-modal similarity alignment, while NCA measures sample importance globally and locally, using soft weighting to strengthen inter-modal attention. Both achieved superior performance through joint learning using Gaussian likelihood maximization. Finally, adding noise to attributes, relationships, and images also improves entity alignment performance, replicating the modal feature distribution and thus more effectively enhancing the model's robustness.

[0166] In reality, only 50% to 70% of entities in MMKG have images, meaning more than two-thirds of entities lack valid images. FBMEA mitigates the impact of weak modalities through adaptive feature fusion, but it doesn't address the issue from a dataset perspective. Therefore, subsequent entity alignment work could supplement external image libraries to enhance entity alignment accuracy.

[0167] As described above, the method provided in this application introduces a Gaussian noise mask into the initial modality embedding and utilizes the Transformer's multi-head self-attention mechanism and adaptive feature fusion method to perform multi-level fusion of multimodal data. Furthermore, by jointly evaluating the multimodal embedding using contrastive learning loss and cross-modal alignment loss, efficient alignment of equivalent entities across different multimodal knowledge graphs can be achieved. Experimental results show that this method achieves optimal performance on multiple datasets.

[0168] like Figure 5 As shown, this embodiment of the invention provides an alignment and fusion device 500 for multimodal entity features, comprising:

[0169] The feature encoding module 501 is used to input the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain multimodal embedding. The knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding and visual embedding.

[0170] The feature fusion module 502 is used to input the multimodal embedding into the cross-modal attention fusion module for feature fusion and extraction to obtain the cross-modal attention fusion embedding, and to input the multimodal embedding into the multimodal adaptive feature fusion module for feature fusion to obtain the adaptive feature fusion embedding.

[0171] The alignment and fusion module 503 is used to align and fuse entities in the knowledge graph to be aligned based on the multimodal early fusion embedding, the cross-modal attention fusion embedding, and the adaptive feature fusion embedding, wherein the multimodal early fusion embedding is obtained by cascading and fusing the multimodal embeddings in the embedding layer.

[0172] Optionally, the multimodal embedding includes the structural embedding, the multimodal embedding module includes a graph attention network, and the feature encoding module 501 includes:

[0173] The graph structure feature extraction unit is used to input the graph structure modal data in the knowledge graph to be aligned into the graph attention network to extract graph structure features and obtain the structure embedding.

[0174] Optionally, the multimodal embedding includes the text embedding, the multimodal embedding module includes a fully connected feature extraction network, and the feature encoding module 501 includes:

[0175] The text feature extraction unit is used to input the text modal data in the knowledge graph to be aligned into the corresponding fully connected feature extraction network to extract text features and obtain the text embedding.

[0176] Optionally, the multimodal embedding includes the visual embedding, the multimodal embedding module includes RESNET-15, and the feature encoding module 501 includes:

[0177] The visual feature extraction unit is used to input the visual modal data in the knowledge graph to be aligned into the RESNET-152 for visual feature extraction to obtain the visual embedding.

[0178] Optionally, the feature encoding module 501 includes:

[0179] The feature encoding unit is used to input the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding and the initial multimodal embedding.

[0180] An addition unit is used to add Gaussian mask noise to the initial multimodal embedding to obtain the multimodal embedding.

[0181] Optionally, the alignment and fusion device 500 for multimodal entity features further includes:

[0182] An iterative training module is used to iteratively train a feature-balanced multimodal entity alignment model, wherein the feature-balanced multimodal entity alignment model includes the multimodal embedding module, the cross-modal attention fusion module, and the multimodal adaptive feature fusion module;

[0183] The loss function for iterative training includes a multimodal joint contrastive loss and a multimodal joint embedding alignment loss, wherein the multimodal joint contrastive loss satisfies:

[0184]

[0185] in, The contrastive loss, α, is used to characterize the multimodal fusion embedding. m For hyperparameters, M = {g, r, a, n, v} is used to characterize the unimodal contrastive loss, and is used to characterize the embedding of structure, relation, attribute, name and vision in unimodal contexts.

[0186] The multimodal joint embedding alignment loss satisfies:

[0187]

[0188] in, The alignment loss, β, is used to characterize the multimodal fusion embedding. m For hyperparameters, Used to characterize single-modal alignment loss.

[0189] Optionally, the loss function of the iterative training satisfies:

[0190]

[0191] The multimodal entity feature alignment and fusion device 500 provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0192] It should be noted that the division of units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0193] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0194] like Figure 6 As shown, this application provides an electronic device 600, including: a memory 602, a processor 601, and a program stored in the memory 602 and executable on the processor 601; the processor 601 is used to read the program in the memory 602 to implement the steps in the multimodal entity feature alignment and fusion method as described above.

[0195] This application also provides a readable storage medium storing a program that, when executed by a processor, implements the various processes of the above-described multimodal entity feature alignment and fusion method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here. The readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (e.g., compact disks (CDs), digital video discs (DVDs), Blu-ray discs (BD), high-definition versatile discs (HVD), etc.), and semiconductor storage (e.g., read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), non-volatile memory (NAND FLASH), solid-state disks (SSDs), etc.).

[0196] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0198] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for aligning and fusing multimodal entity features, characterized in that, include: The knowledge graph to be aligned is input into the multimodal embedding module for feature encoding to obtain the multimodal embedding. The knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding and visual embedding. The multimodal embedding is input into the cross-modal attention fusion module for feature fusion and extraction to obtain the cross-modal attention fusion embedding, and the multimodal embedding is input into the multimodal adaptive feature fusion module for feature fusion to obtain the adaptive feature fusion embedding. The entities in the knowledge graph to be aligned are aligned and fused based on the multimodal early fusion embedding, the cross-modal attention fusion embedding, and the adaptive feature fusion embedding, wherein the multimodal early fusion embedding is obtained by cascading and fusing the multimodal embeddings in the embedding layer; The method further includes: The feature-balanced multimodal entity alignment model is iteratively trained, wherein the feature-balanced multimodal entity alignment model includes the multimodal embedding module, the cross-modal attention fusion module, and the multimodal adaptive feature fusion module; The loss function for iterative training includes a multimodal joint contrastive loss and a multimodal joint embedding alignment loss, wherein the multimodal joint contrastive loss satisfies: ； in, The contrastive loss used to characterize multimodal fusion embeddings For hyperparameters, Used to characterize single-modal contrast loss. Single-modal embeddings used to represent structure, relationships, attributes, names, and visuals; The multimodal joint embedding alignment loss satisfies: ； in, Alignment loss used to characterize multimodal fusion embedding. For hyperparameters, Used to characterize single-modal alignment loss.

2. The method as described in claim 1, characterized in that, The multimodal embedding includes the structural embedding, the multimodal embedding module includes a graph attention network, and the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes: The graph structure modal data in the knowledge graph to be aligned is input into the graph attention network to extract graph structure features and obtain the structure embedding.

3. The method as described in claim 1, characterized in that, The multimodal embedding includes the text embedding, and the multimodal embedding module includes a fully connected feature extraction network. The step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes: The text modal data in the knowledge graph to be aligned is input into the corresponding fully connected feature extraction network to extract text features, thereby obtaining the text embedding.

4. The method as described in claim 1, characterized in that, The multimodal embedding includes the visual embedding, and the multimodal embedding module includes RESNET-15. The step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes: The visual modal data in the knowledge graph to be aligned is input into the RESNET-152 for visual feature extraction to obtain the visual embedding.

5. The method according to any one of claims 1-4, characterized in that, The step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes: The knowledge graph to be aligned is input into the multimodal embedding module for feature encoding to obtain the multimodal embedding, thus obtaining the initial multimodal embedding. Gaussian mask noise is added to the initial multimodal embedding to obtain the multimodal embedding.

6. The method as described in claim 1, characterized in that, The loss function for the iterative training satisfies: 。 7. A multimodal entity feature alignment and fusion apparatus, characterized in that, include: The feature encoding module is used to input the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain multimodal embedding. The knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding and visual embedding. The feature fusion module is used to input the multimodal embedding into the cross-modal attention fusion module for feature fusion and extraction to obtain the cross-modal attention fusion embedding, and to input the multimodal embedding into the multimodal adaptive feature fusion module for feature fusion to obtain the adaptive feature fusion embedding; The alignment and fusion module is used to align and fuse entities in the knowledge graph to be aligned based on the multimodal early fusion embedding, the cross-modal attention fusion embedding, and the adaptive feature fusion embedding, wherein the multimodal early fusion embedding is obtained by cascading and fusing the multimodal embeddings in the embedding layer; The alignment and fusion device for multimodal entity features further includes: An iterative training module is used to iteratively train a feature-balanced multimodal entity alignment model, wherein the feature-balanced multimodal entity alignment model includes the multimodal embedding module, the cross-modal attention fusion module, and the multimodal adaptive feature fusion module; The loss function for iterative training includes a multimodal joint contrastive loss and a multimodal joint embedding alignment loss, wherein the multimodal joint contrastive loss satisfies: ； in, The contrastive loss used to characterize multimodal fusion embeddings For hyperparameters, Used to characterize single-modal contrast loss. Single-modal embeddings used to represent structure, relationships, attributes, names, and visuals; The multimodal joint embedding alignment loss satisfies: ； in, Alignment loss used to characterize multimodal fusion embedding. For hyperparameters, Used to characterize single-modal alignment loss.

8. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program from the memory to implement the steps in the method for aligning and fusing multimodal entity features as claimed in any one of claims 1 to 6.

9. A readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps in the alignment and fusion method for multimodal entity features as described in any one of claims 1 to 6.

Images