Multi-modal entity feature alignment fusion method and device and electronic equipment

Abstract

The invention discloses a multi-modal entity alignment method, and particularly relates to a multi-modal entity feature alignment fusion method and device and electronic equipment, and the method comprises the steps: inputting a to-be-aligned knowledge graph into a multi-modal embedding module for feature coding, and obtaining multi-modal embedding, inputting the multi-modal embedding into a cross-modal attention fusion module for feature fusion and extraction to obtain cross-modal attention fusion embedding, and inputting the multi-modal embedding into a multi-modal adaptive feature fusion module for feature fusion to obtain adaptive feature fusion embedding; the entities in the knowledge graph to be aligned are subjected to alignment fusion based on multi-modal early fusion embedding, cross-modal attention fusion embedding and self-adaptive feature fusion embedding, and the multi-modal early fusion embedding is obtained by performing cascade fusion on multi-modal embedding in an embedding layer. According to the method, the problems of visual weak modal fusion imbalance and semantic fuzziness during multi-modal entity feature alignment can be solved.

Description

Technical Field

[0001] The present invention relates to a multimodal entity alignment method, and more particularly to a multimodal entity feature alignment and fusion method, device, and electronic device. Background Art

[0002] A knowledge graph (KG) is composed of relationships between facts and entities and is essentially a large-scale semantic web. Traditional knowledge graphs are primarily composed of text. However, text-based, unimodal knowledge graphs are often incomplete, lacking the joint representation of heterogeneous knowledge such as images, videos, and speech. In recent years, a prominent trend has been to build multimodal knowledge graphs (MMKGs) by integrating multimodal data, such as visual data, into knowledge graphs.

[0003] The amount of real-world knowledge is enormous, but most MMKGs are typically constructed from a single data source or in a single language, which results in very little triple data for some entities, or even modality missing. These entities with sparse relationships and modality missing are called long-tail entities. MMKGs constructed from different data mean that the same semantic entities may exist in different MMKGs. If entities with the same semantics can be fused, it can effectively supplement the long-tail entity data and enhance the semantic representation capabilities of the MMKG. To implement this technology, a common strategy is to integrate the various modal information of different MMKGs into a unified MMKG to achieve entity matching, thereby expanding the coverage of knowledge. This technology of matching equivalent entities of the same object in the real world across different MMKGs using modalities such as vision, attributes, and numerical values ​​is called multi-modal entity alignment (MMEA).

[0004] MMEA is a complex technology, and current MMEA methods primarily focus on designing a cross-modal fusion paradigm. However, these methods focus on directly merging unimodal features after embedding different modalities, without considering cross-modal information interaction during multimodal fusion. This ignores the intra-modal differences and entity sparsity of MMKG, leading to imbalanced fusion and semantic ambiguity in weak modalities such as vision. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the above-mentioned method commonly used in the prior art directly merges single-modal features after using different modal embeddings, without considering the cross-modal information interaction in the multimodal fusion process. This ignores the characteristics of intra-modal differences and entity sparsity of MMKG, resulting in imbalanced fusion of weak modalities such as vision and semantic ambiguity. In order to solve the above problems, the present invention provides a method, device and electronic device for aligning and fusing multimodal entity features.

[0006] The content of the present invention includes:

[0007] In a first aspect, an embodiment of the present invention provides a method for aligning and fusing multimodal entity features, comprising:

[0008] Inputting the knowledge graph to be aligned into a multimodal embedding module for feature encoding to obtain a multimodal embedding, wherein the knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding, and visual embedding;

[0009] Inputting the multimodal embedding into a cross-modal attention fusion module for feature fusion and extraction to obtain a cross-modal attention fusion embedding, and inputting the multimodal embedding into a multimodal adaptive feature fusion module for feature fusion to obtain an adaptive feature fusion embedding;

[0010] The entities in the knowledge graph to be aligned are aligned and fused based on the multimodal early fusion embedding, the cross-modal attention fusion embedding and the adaptive feature fusion embedding, wherein the multimodal early fusion embedding is obtained by cascading the multimodal embedding in the embedding layer.

[0011] Optionally, the multimodal embedding includes the structural embedding, the multimodal embedding module includes a graph attention network, and the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0012] The graph structure modal data in the knowledge graph to be aligned is input into the graph attention network to extract graph structure features to obtain the structure embedding.

[0013] Optionally, the multimodal embedding includes the text embedding, the multimodal embedding module includes a fully connected feature extraction network, and the inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0014] The text modality data in the knowledge graph to be aligned is input into the corresponding fully connected feature extraction network to perform text feature extraction to obtain the text embedding.

[0015] Optionally, the multimodal embedding includes the visual embedding, the multimodal embedding module includes RESNET-15, and the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0016] The visual modality data in the knowledge graph to be aligned is input into the RESNET-152 to extract visual features and obtain the visual embedding.

[0017] Optionally, inputting the knowledge graph to be aligned into a multimodal embedding module for feature encoding to obtain a multimodal embedding includes:

[0018] Inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain a multimodal embedding and an initial multimodal embedding;

[0019] Gaussian masked noise is added to the initial multimodal embedding to obtain the multimodal embedding.

[0020] Optionally, the method further includes:

[0021] Iteratively training a feature-balanced multimodal entity alignment model, wherein the feature-balanced multimodal entity alignment model includes the multimodal embedding module, the cross-modal attention fusion module, and the multimodal adaptive feature fusion module;

[0022] The loss function of the iterative training includes a multimodal joint contrast loss and a multimodal joint embedding alignment loss, and the multimodal joint contrast loss satisfies:

[0023]

[0024] in, Contrastive loss for characterizing multimodal fusion embedding, α m is a hyperparameter, Used to represent unimodal contrast loss, M = {g, r, a, n, v} is used to represent unimodal embedding;

[0025] The multimodal joint embedding alignment loss satisfies:

[0026]

[0027] in, Alignment loss for characterizing multimodal fusion embedding, β m is a hyperparameter, Used to characterize unimodal alignment loss.

[0028] Optionally, the total loss function of the iterative training satisfies:

[0029]

[0030] In a second aspect, an embodiment of the present invention provides a device for aligning and fusing multimodal entity features, comprising:

[0031] A feature encoding module is configured to input the knowledge graph to be aligned into a multimodal embedding module for feature encoding to obtain a multimodal embedding, wherein the knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding, and visual embedding;

[0032] A feature fusion module is used to input the multimodal embedding into a cross-modal attention fusion module for feature fusion and extraction to obtain a cross-modal attention fusion embedding, and to input the multimodal embedding into a multimodal adaptive feature fusion module for feature fusion to obtain an adaptive feature fusion embedding;

[0033] An alignment fusion module is used to align and fuse the entities in the knowledge graph to be aligned based on the multimodal early fusion embedding, the cross-modal attention fusion embedding and the adaptive feature fusion embedding, wherein the multimodal early fusion embedding is obtained by cascading the multimodal embedding in the embedding layer.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device comprising: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps in the method for aligning and fusing multimodal entity features as described in the first aspect.

[0035] In a fourth aspect, an embodiment of the present invention provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the method for aligning and fusing multimodal entity features as described in the first aspect.

[0036] In an embodiment of the present invention, the knowledge graph to be aligned is input into a multimodal embedding module for feature encoding to obtain a multimodal embedding, the knowledge graph to be aligned includes multimodal data, the multimodal embedding is input into a cross-modal attention fusion module for feature fusion and extraction to obtain a cross-modal attention fusion embedding, and the multimodal embedding is input into a multimodal adaptive feature fusion module for feature fusion to obtain an adaptive feature fusion embedding; based on multimodal early fusion embedding, cross-modal attention fusion embedding and adaptive feature fusion embedding, the entities in the knowledge graph to be aligned are aligned and fused. The beneficial effect of the present invention is to make full use of the complementarity between different modalities for step-by-step fusion to enhance the robustness of entity alignment, mitigate the impact of sparse long-tail entity data, and enable the model to better adapt to the data characteristics of different modalities, thereby solving the problems of insufficient cross-modal effect and unbalanced cross-modal fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Attachment Figure 1 A flowchart of a method for aligning and fusing multimodal entity features provided by an embodiment of the present invention;

[0038] Attachment Figure 2a A schematic diagram of the overall structure of the FBMEA model provided in an embodiment of the present invention;

[0039] Attachment Figure 2b for Figure 2aOne of the partially enlarged schematic diagrams of the FBMEA model;

[0040] Attachment Figure 2c for Figure 2a The second partially enlarged schematic diagram of the FBMEA model;

[0041] Attachment Figure 3 This is an example diagram of multimodal entity alignment;

[0042] Attachment Figure 4a This is a schematic diagram of the ablation experiment results of FBMEA on the DBP15KZH-EN dataset;

[0043] Attachment Figure 4b This is the second schematic diagram of the ablation experiment results of FBMEA on the DBP15KZH-EN dataset;

[0044] Attachment Figure 4c This is a schematic diagram of the ablation experiment results of FBMEA on the DBP15KJA-EN dataset;

[0045] Attachment Figure 4d This is the second schematic diagram of the ablation experiment results of FBMEA on the DBP15KJA-EN dataset;

[0046] Attachment Figure 5 A schematic diagram of a device for aligning and fusing multimodal entity features provided by an embodiment of the present invention;

[0047] Attachment Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In the embodiments of this application, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship. In the embodiments of this application, the term "plurality" refers to two or more, and other quantifiers are similar.

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] See Figure 1 , Figure 1A flowchart of a method for aligning and fusing multimodal entity features provided in an embodiment of the present invention is provided. The method specifically includes the following steps:

[0051] Step 101: Input the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain a multimodal embedding, wherein the knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding, and visual embedding.

[0052] In step 102, the multimodal embedding is input into a cross-modal attention fusion module for feature fusion and extraction to obtain a cross-modal attention fusion embedding, and the multimodal embedding is input into a multimodal adaptive feature fusion module for feature fusion to obtain an adaptive feature fusion embedding.

[0053] Step 103: align and fuse the entities in the knowledge graph to be aligned based on the multimodal early fusion embedding, the cross-modal attention fusion embedding, and the adaptive feature fusion embedding, wherein the multimodal early fusion embedding is obtained by cascading the multimodal embedding in the embedding layer.

[0054] For ease of understanding, the symbolic parameters involved in the embodiments of the present application are first explained below. The Knowledge Graph (KG) is composed of relational facts and entities, and is essentially a large-scale semantic network. KG uses entities and concepts as nodes, and the semantic relationships between nodes as edges, expressed as G = (E, R, T), where E, R, and T represent entities, relationships, and triple sets, respectively. Triple T = {(h, r, t) | h, t∈E, r∈R}, where h represents the head entity, t represents the tail entity, and r represents the relationship between the head and tail entities.

[0055] A multimodal knowledge graph is formalized as a directed graph MMKG = {E, R, A, V, T}, where E represents the set of entities, R represents the set of relationships, A represents the set of attributes, V represents the set of images, and T represents the set of triples. The two multimodal knowledge graphs to be aligned can be represented as MMKG1 = {E1, R1, A1, V1, T1} and MMKG2 = {E2, R2, A2, V2, T2}.

[0056] During the model training process, the pre-aligned seed entity pair set Seed is used as training data, which is expressed as Seed = {(e s1 , e s2 )|e s1 ∈E1,e s2 ∈E2,e s1 ~e s2The goal of multimodal entity alignment is to design a multimodal alignment framework based on the proportion of the seed entity set in the total entity set and the characteristics of the entity pairs, and to find the entity pair set Align = {(e1, e2) | e1∈E1, e2∈E2 in MMKG1 and MMKG2 that represents the same objective object in the real world. ， e1~e2}, where ~ represents an equivalence relation,

[0057] The method for aligning and fusing multi-modal entity features provided in the embodiment of the present invention can also be called a feature-balanced multi-modal entity alignment (FBMEA) method. The embodiment of the present invention also provides an FBMEA model, specifically Figure 2a-2c As shown in the figure, the solid lines represent the relationship links between entities, and the dotted lines represent the attribute information of entities between different graphs.

[0058] It should be understood that the number of knowledge graphs to be aligned is at least two, and the data modalities contained in the at least two knowledge graphs to be aligned can be the same or different. For details, see Figure 3 In the multimodal embedding module, the multimodal data of the knowledge graph to be aligned are feature encoded to obtain multimodal embedding. The feature extraction method varies depending on the data modality.

[0059] Optionally, in some embodiments, the multimodal embedding includes the structural embedding, the multimodal embedding module includes a graph attention network, and inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0060] The graph structure modal data in the knowledge graph to be aligned is input into the graph attention network to extract graph structure features to obtain the structure embedding.

[0061] The purpose of graph embedding is to map the knowledge graph data of high-dimensional dense matrices into low-dimensional dense vectors. In this embodiment, a graph attention network (GAT) is used to extract graph structure features in the knowledge graph.

[0062] Specifically, entity e i The aggregated hidden layer of one-hop neighbors is represented as (d represents the hidden layer dimension), the self-attention coefficient of each node Indicates the importance of node j to node i, introduces attention into the graph structure, and assigns it to the set of one-hop neighbor nodes N of node i i Regularization can be expressed as:

[0063]

[0064] Attention mechanism α ij It is a single-layer feedforward neural network. LeakyReLU is added to the output layer to obtain a complete attention mechanism.

[0065]

[0066] in, represents the connection operation, T represents the transposition, represents the weight matrix, Represents a learnable vector. In order to stabilize the self-attention learning process, the attention mechanism is extended with multi-head attention and the linear transformation weight matrix W is used. k Instead of the connection operation, the final structure embedding is expressed as follows:

[0067]

[0068] Where, The normalized attention mechanism representing the k-th attention calculation usually adopts a two-layer GAT model to aggregate the domain structure and uses the last layer of GAT output as the structure embedding.

[0069] Optionally, in some embodiments, the multimodal embedding includes the visual embedding, the multimodal embedding module includes RESNET-15, and inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0070] The visual modality data in the knowledge graph to be aligned is input into the RESNET-152 to extract visual features and obtain the visual embedding.

[0071] In this example, the pre-trained model RESNET-152 is used to extract visual features instead of the VGG16 in the dataset. In comparison, RESNET-152 has fewer parameters and more prominent effects, which can accelerate neural network training and significantly improve model accuracy. RESNET-152 forward passes each image, uses the output of the final layer before logits as image features, and sends the features through a trainable feedforward layer. The resulting visual embedding satisfies:

[0072]

[0073] Among them, Ing i represents the i-th image input, and is a learnable hyperparameter.

[0074] Optionally, in some embodiments, the multimodal embedding includes the text embedding, the multimodal embedding module includes a fully connected feature extraction network, and inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0075] The text modality data in the knowledge graph to be aligned is input into the corresponding fully connected feature extraction network to perform text feature extraction to obtain the text embedding.

[0076] It should be understood that in order to alleviate the problem of mixed text feature relationships, this embodiment sets up a separate fully connected feature extraction network for each text modality. In this embodiment, the cross-language entity alignment method of the bag of words is followed for feature representation, and the features are input into the feedforward neural network to establish connections between the modalities in a low-dimensional space.

[0077] Through the above method, the obtained text embedding satisfies:

[0078]

[0079] where m∈{r, a, n} represents the relation, attribute, and name (surface form) modalities, and is a learnable hyperparameter, F m Represents the bag-of-words features of each modality.

[0080] The problems of modality loss and visual text ambiguity are common in MMEA, which leads to overfitting of the multimodal fusion framework. Optionally, in some embodiments, the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes:

[0081] Inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain a multimodal embedding and an initial multimodal embedding;

[0082] Gaussian masked noise is added to the initial multimodal embedding to obtain the multimodal embedding.

[0083] In this embodiment, the introduction of irrelevant contextual noise can improve cross-modal alignment capabilities. Specifically, in this embodiment, Gaussian masked noise is added to the multimodal embedding to enhance the representation of modal features, better adapt to the noise and uncertainty existing in the real world, and enhance generalization capabilities.

[0084] The resulting multimodal embedding satisfies:

[0085]

[0086] Where ε is the noise ratio, and {r, a, v} represents the relationship, attribute, and visual modality, which indicates the relative strength of noise and modality features. m With μ m Represent the standard deviation and mean value of the noise data of a specific mode, respectively. n is a sample extracted from the Gaussian distribution N(0,1), ensuring that the introduced noise is consistent with the inherent statistical characteristics of each mode. is considered as an embedding with added noise.

[0087] In this embodiment, Gaussian noise is added to the MMKG fusion training to learn features that are insensitive to the noised data, thereby better adapting to changes in data of different modalities and enhancing the generalization performance of the model.

[0088] The multimodal embedding is input into the multimodal adaptive feature fusion module for feature fusion to obtain the adaptive feature fusion embedding. The specific method is as follows:

[0089] There are a large number of long-tail entities in the knowledge graph, and the structural information of these entities is very scarce. Therefore, when aligning long-tail entities, it is necessary to increase the weight of semantic information. In order to fully release the value of each modality and balance the impact of low-quality multimodal data, the embodiment of this application uses the structural and semantic similarity matrix to perform dynamic feature fusion at this stage, and uses the cosine similarity measurement of different modal embeddings of the multimodal knowledge graph to measure the cosine similarity of the two modal embeddings. semantic similarity.

[0090]

[0091] in, and Represent structural information embedding and non-structural information embedding, Indicates the structural similarity between the i-th entity of the source knowledge graph and the j-th entity of the target knowledge graph. For non-structural semantic information embedding The calculation using the cascade method is:

[0092]

[0093] Among them, w m is the trainable weight of modality m. Before weighted concatenation, each input feature matrix is ​​L2 normalized to obtain the unstructured semantic embedding. The semantic similarity matrix is ​​calculated in the same way Fusion and Get the final similarity score M ij :

[0094]

[0095] Where wg represents the weight of structural information, and w′s represents the weight of semantic information. In order to better adjust the contribution of structural information and semantic information entity alignment, this embodiment adopts a degree-aware entity alignment method to dynamically assign weights to different features, which can be specifically expressed as:

[0096]

[0097] w g =1-w s .

[0098] Among them, K, b, α are hyper parameters, degree is the degree of the entity, N hop is the degree of association between the entity and the seed entity, N hop The calculation is as follows:

[0099] N hop =n 1-hop ×w1+lg(n 2-hop ×w2)

[0100] Among them, n 1-ho p and n 2-hop Represents the number of entities 1 hop and 2 hops away from the seed entity, respectively. w1 and w2 are hyperparameters. The method provided by this application can fully balance semantic embedding and structural embedding, and adaptively calculate the structural embedding weight w g and semantic embedding weight w s , thereby calculating the entity e i Adaptive feature fusion embedding of:

[0101]

[0102] Finally, based on the similarity score M ij The candidate entity pairs are sorted by similarity, and entities with high modal similarity are selected as aligned entity pairs to fully capture the entity semantic information.

[0103] It should be understood that the similarity score here is the prior information for entity alignment. This part uses adaptive feature fusion to obtain multimodal fusion embedding, and the similarity score is obtained through the similarity of structural modality and semantic modality. In this embodiment, entity pairs with high similarity scores between different entities are aligned.

[0104] In the adaptive feature fusion module, similarity scores are used to guide the entity alignment task, selecting only candidate entity pairs with high similarity for alignment. Furthermore, the text also utilizes an iterative learning strategy, adding high-confidence entity pairs to the seed entity pairs for training to address the lack of seed entity information and the high cost of manual annotation.

[0105] The multimodal embedding is input into the cross-modal attention fusion module for feature fusion and extraction to obtain the cross-modal attention fusion embedding. The specific process is described as follows:

[0106] In this embodiment, dynamic cross-modal weighting of MEAformer and UMAEA is introduced. The network can dynamically adjust the entity-level meta-weight of each modality, and add multi-head convolutional attention (MHCA) and a fully connected feedforward neural network (FFN) after single-modal embedding.

[0107] MHCA in N h The attention function is run on parallel heads, and the i-th head is parameterized by the modality sharing matrix as (where d is the input modality dimension, d h =d / N n is the key vector dimension, used to scale the dot product to stabilize the softmax function), and the multimodal input h m Convert to modality-aware query key Sum Afterwards, the inputs from all heads are combined through linear transformation, and the modal feature output is as follows:

[0108]

[0109] Among them, W o ∈R d×d , is the attention weight of the entity between modality m and modality j, which can be expressed as:

[0110]

[0111] Among them, d h =d / N h , Layer Normalization (LN) and Residual Connection (RC) are used to stabilize the training. FFN consists of two linear transformation layers and ReLU activation function, which is applied to LN and RC and is expressed as:

[0112]

[0113] in, d in represents the cross-modal attention feature dimension, b1 and b2 are learnable parameters. Each modality output element weight w m Expressed as:

[0114]

[0115] Where M represents the set of available modalities. It should be understood that |M| is used to consider the similarity between different modalities, and no negative values ​​appear in the calculation process.

[0116] Entity e i The joint embedding of is expressed as:

[0117]

[0118] Here is FBMEA for entity e i The cross-modal attention fusion output (i.e., fusing different multimodal information) is obtained in the cross-modal attention fusion module. The cross-modal joint embedding is obtained (as shown in the framework diagram in Figure 2, part of which is obtained by combining the meta-weights of the cross-modal attention output with the feature output of the Transformer, and the other part is obtained by combining the meta-weights of the cross-modal attention output with the multimodal embedding features) and then alignment discrimination is performed.

[0119] Optionally, the method further includes:

[0120] Iteratively training a feature-balanced multimodal entity alignment model, wherein the feature-balanced multimodal entity alignment model includes the multimodal embedding module, the cross-modal attention fusion module, and the multimodal adaptive feature fusion module;

[0121] The loss function of the iterative training includes multimodal joint contrast loss and multimodal joint embedding alignment loss.

[0122] This application uses multimodal contrast representation learning to construct an adaptive contrast loss L in the modal feature channel icl This loss uses the similarity of the original embedding space to identify the same entities in different KGs. In the process of entity representation, weak modalities may more easily have an excessive impact on the overall model. In order to achieve enhanced fusion between modalities and avoid feature fusion imbalance, this application adopts holistic modality representation and single modality joint representation to design a multimodal comparative learning module from multiple angles to fully explore the information contained in each modality in the limited seed pairs.

[0123] Each pair of seed entities in the model It can be regarded as a positive sample, and other entity pairs without relationship links are regarded as negative samples. The entities follow the 1-to-1 alignment constraint principle. Define the i-th entity of the minimum batch S The positive sample can be expressed as:

[0124]

[0125] Negative samples consist of two parts: the unaligned part within the KG and the unaligned part outside the KG, which can be expressed as:

[0126]

[0127] Limiting the range of N sampling to small batches, the probability of alignment is defined as:

[0128]

[0129] in, is the encoder of modality m, τ represents the temperature hyperparameter, and T represents the transpose. Before calculating the inner product, the embedded input features are L2 normalized, and the unimodal contrast loss can be expressed as:

[0130]

[0131] The unimodal contrast loss is performed in contrastive supervised learning, and each modality channel is applied separately. The single-modal contrast loss is composed to make the boundary of each modality in the data embedding space more distinguishable. At the same time, the multi-modal fusion embedding h is also weighted fusion of visual, attribute, name and other modalities (M = {g, r, a, n, v}) joint (Multimodal early fusion embedding h i , adaptive feature fusion embedding Fusion embedding with cross-modal attention )

[0132] It should be understood that the multimodal fusion embeddings of different embedding methods here are all derived by weighting the single-modal embeddings. The weights corresponding to each modality can be obtained through the above method. All three embedding fusion methods use graph structure, relationships, attributes, names, and vision for joint representation.

[0133] .Use contrast loss for multimodal fusion embedding and add hyperparameter α m , the obtained multimodal joint embedding contrast loss L icl It can be expressed as:

[0134]

[0135] in, Contrastive loss for characterizing multimodal fusion embedding, α m is a hyperparameter, The contrastive loss is used to represent the unimodal embeddings, M = {g, r, a, n, v}, which is used to represent the unimodal embeddings of structure, relationship, attribute, name, and vision. The contrastive loss of the multimodal joint embedding is determined by the contrastive loss of the multimodal fusion embedding and the contrastive loss of the unimodal embedding.

[0136] It should be understood that the joint embedding here refers to the previous multimodal feature fusion. In particular, cross-modal attention fusion consists of two parts: one is obtained by combining the meta-weights of the cross-modal attention output with the feature output of the Transformer, and the other is obtained by combining the meta-weights of the cross-modal attention output with the initial multimodal embedding features (see Figure 2 for details).

[0137] Neighborhood Component Analysis (NCA) loss is designed for classification tasks. It uses local and global statistics to measure the importance of samples and a soft weighting scheme to penalize hard negatives to address the problem of embedding space shift. NCA loss uses cosine similarity to calculate the alignment loss for entity similarity, attribute similarity, and neighbor dissimilarity. The consistency of entity alignment is evaluated across multiple views, and the unimodal alignment loss is calculated as follows:

[0138]

[0139] in, Represents alignment entities Cosine similarity of single modality m or joint embedding, α, β represent hyperparameters, N represents the number of minimum batch training. And adaptive contrast loss L icl Similarly, this embodiment applies the loss to other modal alignment channels and calculates the multimodal joint representation loss.

[0140] Add hyperparameter β m , the multimodal joint embedding alignment loss is calculated as follows:

[0141]

[0142] in, Alignment loss for characterizing multimodal fusion embedding, β m is a hyperparameter, Used to represent the unimodal alignment loss. Here is the alignment loss between entities after multimodal fusion.

[0143] FBMEA is considered as a multi-task learning model, each type of loss is regarded as a task, learning is performed through shared feature representation, and the correlation and information interaction between tasks are used to improve the generalization performance and efficiency of the model. m , β m The two types of loss are hyperparameters that balance the two types of losses. However, manually tuning these parameters is expensive and time-consuming. This application considers FBMEA as a multi-task learning paradigm, leveraging variance uncertainty to automatically adjust the weights between the various losses during model training. The multi-task loss function is derived by maximizing the Gaussian likelihood under task uncertainty.

[0144] Optionally, in some embodiments, the loss function of the iterative training satisfies:

[0145]

[0146] In this embodiment, a target optimization scheme is designed using the intra-state loss of multimodal contrastive learning and the inter-state loss of multimodal alignment. This optimization scheme decouples different modalities, making the model more independent and controllable in the process of learning different modalities.

[0147] As shown in Figure 2, this method uses a multimodal embedding channel to independently encode features for each modality in the noisy multimodal data. Subsequently, the multimodal embedding is input into the Transformer for cross-modal attention fusion and extraction, dynamically adjusting the modal preference to achieve finer-grained entity alignment. At the same time, the multimodal embedding is input into the adaptive feature fusion module to adjust the fusion of structural embedding and semantic embedding, increase the weight of semantic information, balance the problems of sparse structural embedding and low graph coverage, and finally use contrast loss and alignment loss to learn joint embedding. This method combines adaptive feature fusion with a multi-head attention mechanism, combines multi-level fusion to alleviate the problem of feature fusion imbalance, and adjusts the structural information embedding to alleviate the impact of long-tail entities, providing MMKG with richer semantic representations.

[0148] The beneficial effects of the method provided in this application are verified through experiments below.

[0149] (1) Dataset

[0150] Experiments were conducted using five public multimodal entity alignment datasets, including three bilingual datasets (DBP15K, ZH-EN, JA-EN, and FR-EN) and two cross-knowledge graph datasets (FB15K-DB15K and FB15K-YAGO15K). To compare with the baseline, this experiment used 20% of the entities provided by DBP15K as pre-aligned seed entity pairs. For the cross-knowledge graph datasets, 20%, 50%, and 80% of the seed entity pairs were used, respectively.

[0151] (2) Baseline

[0152] The FBMEA method provided in an embodiment of the present invention is compared with nine current mainstream baseline models, which can be mainly divided into two categories: entity alignment methods based on graph structure and entity methods based on multimodal fusion paradigm. The former only uses the knowledge graph structure, while the latter uses other modalities to assist in the alignment task.

[0153] (3) Implementation details

[0154] In order to achieve cross-modal information interaction and learning and to preserve each modality completely in space, we transform the hidden layer h of the GAT graph attention network intog With the rest of the modes h m The embedding layer is set to size 300 and the entity similarity threshold θ = 0.6. Similar to MCLEA, the AdamW optimizer is used with a 2×10 -5 The learning rate is used to update the parameters, 1000 epoc training cycles are set, and the batch size is set to 1024. To be consistent with the current baseline method, we use the name translation and word vector provided by Xu et al.

[30] . The visual embedding uses the visual preprocessing model of ResNet-152 to extract features with a dimension of 2048, which are then mapped to a feature vector of 300 dimensions. The cross-atlas dataset uses VGG-16 to extract features with a dimension of 4096. For the adaptive feature fusion strategy, we use the dimension d g The hyperparameters are set to 300, with b = 1.5, α = 1, w1 = 0.8, and w2 = 0.1. The value of K depends on the size of the seed set, and γ is set to 0.8. Similar to the current mainstream MMEA, FBMEA adopts an iterative learning strategy, adding each pair of mutually nearest neighbors to the candidate list. If the extracted entities remain the closest neighbors for K rounds, they are added to the training set.

[0155] (4) Evaluation indicators

[0156] We use Hit@N (N=1, 10) (the ratio of the number of hits in the top k recommended results to the total number of test samples) and MRR (mean reciprocal ranking), common metrics in multimodal entity alignment tasks, to evaluate the model. The best results are bolded, and the suboptimal results are underlined.

[0157] Please refer to Table 1, which shows the alignment performance of FBMEA on the bilingual dataset DBP15K using surface form and iterative training. Currently for bilingual datasets. In Table 1, boldface indicates the optimal model, underlined indicates the suboptimal model, and * indicates the model's reproduction result under the same computing environment. The baseline model is close to the ideal performance, and FBMEA can still achieve some improvement. Compared with the baseline MEAformer, FBMEA improves His@1 by 1.3% and 2.4% on the ZH-EN and JA-EN datasets, respectively, but the improvement is insufficient on FR-EN. The MMR metric improves by 0.01% to 0.1%, indicating that FEMEA can achieve a near-perfect performance on bilingual datasets. To test the robustness of the model, we further validated it on a cross-graph dataset.

[0158] Table 1 Entity alignment results of bilingual datasets using surface information and iteration when seed training data accounts for 20%

[0159]

[0160] Table 2 shows the entity alignment performance of FBMEA on the cross-graph datasets FB15K-DB15K and FB15K-YAGO15K, using different seed entity pair ratios. In Table 2, X% indicates the percentage of aligned entity pairs used for training in the dataset, and * indicates the model's replication results under the same computing environment.

[0161] In various cases, FBMEA achieved the best experimental results in almost all cases. When given 20% entity pairs, FBMEA's Hits@1 index increased by 0.7% and 5.6% compared with the best comparison model, with the most significant improvement, indicating that FBMEA can better adapt to low-resource environments. When given 50% and 80%, Hits@1 and MRR performance improvements are still significant. In addition, FBMEA's improvement on the FB15K-YAGO15K dataset is even greater, with the most significant improvement in the MRR index (8.4%, 3.6%, and 1.7%), indicating that FBMEA can significantly improve the MMKG link prediction ability and entity alignment accuracy, and alleviate the sparsity of MMKG. On the FB15K-DB15K dataset, the MRR index is still improved. Of particular note, we found that the Hits@10 score on the FB15K-DB15K dataset was slightly lower than that of MEAformer. This suggests that while FBMEA can accurately generate query results, its performance is insufficient on a broader query range. We speculate that this is due to adaptive feature fusion weakening the structural information of long-tail entities, resulting in a smaller set of Hits@10 candidates. Bilingual datasets and cross-graph datasets differ significantly in attributes and relationships, and exhibit considerable heterogeneity. For example, the EVA model is insufficient on cross-graph datasets. This provides direct evidence for FBMEA's generalization performance and model stability.

[0162] Table 2 Alignment results across graph datasets under iterative conditions

[0163]

[0164] The ablation experiment is carried out below.

[0165] We conducted component ablation experiments on a bilingual dataset to demonstrate the Hits@1 / MRR gains brought by different components. The results are as follows: Figure 4a-4dAs shown in Figure 3, we first explored the impact of removing each modality on the results. We found that removing any modality resulted in performance degradation, and that surface form (such as entity names) was the most significant modality, with the most severe degradation after removing surface form, followed by the visual modality. This result is consistent with previous research. Furthermore, we investigated different optimization schemes and entity alignment fusion frameworks, including AFF (an adaptive feature fusion module) and CAF (a cross-modal attention fusion module). We found that the CAF entity alignment scheme performed well, and that joint learning with AFF adaptive fusion effectively improved alignment accuracy, thus validating the effectiveness of FBMEA. The ICL and NCA alignment methods showed similar performance. ICL's contrastive learning optimization prioritizes intra-modality similarity alignment, while NCA measures sample importance globally and locally, strengthening inter-modality attention through soft weighting. Joint learning of both approaches, maximizing Gaussian likelihood, achieved superior performance. Finally, adding noise to attributes, relations, and images also improved entity alignment performance by replicating the distribution of modal features, thereby more effectively enhancing model robustness.

[0166] In reality, only 50% to 70% of entities in MMKG have images, and over two-thirds of entities lack valid images. FBMEA mitigates the impact of weak modalities using adaptive feature fusion, but it does not address the issue from a dataset perspective. Therefore, subsequent entity alignment work can supplement external image libraries to enhance entity alignment accuracy.

[0167] According to the above content, the method provided in the embodiment of the present application introduces a Gaussian noise mask in the initial modal embedding, and uses the Transformer's multi-head self-attention mechanism and the adaptive feature fusion method to perform multi-level fusion of multimodal data. In addition, the use of contrastive learning loss and cross-modal alignment loss to jointly evaluate multimodal embeddings can achieve efficient alignment of equivalent entities in different multimodal knowledge graphs. Experimental results show that this method achieves optimal performance on multiple datasets.

[0168] like Figure 5 As shown, an embodiment of the present invention provides a multimodal entity feature alignment and fusion device 500, comprising:

[0169] A feature encoding module 501 is configured to input the knowledge graph to be aligned into a multimodal embedding module for feature encoding to obtain a multimodal embedding, wherein the knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding, and visual embedding;

[0170] A feature fusion module 502 is configured to input the multimodal embedding into a cross-modal attention fusion module for feature fusion and extraction to obtain a cross-modal attention fusion embedding, and input the multimodal embedding into a multimodal adaptive feature fusion module for feature fusion to obtain an adaptive feature fusion embedding;

[0171] The alignment fusion module 503 is used to align and fuse the entities in the knowledge graph to be aligned based on the multimodal early fusion embedding, the cross-modal attention fusion embedding and the adaptive feature fusion embedding, wherein the multimodal early fusion embedding is obtained by cascading the multimodal embedding in the embedding layer.

[0172] Optionally, the multimodal embedding includes the structural embedding, the multimodal embedding module includes a graph attention network, and the feature encoding module 501 includes:

[0173] A graph structure feature extraction unit is used to input the graph structure modal data in the knowledge graph to be aligned into the graph attention network to extract graph structure features and obtain the structure embedding.

[0174] Optionally, the multimodal embedding includes the text embedding, the multimodal embedding module includes a fully connected feature extraction network, and the feature encoding module 501 includes:

[0175] A text feature extraction unit is used to input the text modality data in the knowledge graph to be aligned into the corresponding fully connected feature extraction network to perform text feature extraction and obtain the text embedding.

[0176] Optionally, the multimodal embedding includes the visual embedding, the multimodal embedding module includes RESNET-15, and the feature encoding module 501 includes:

[0177] A visual feature extraction unit is used to input the visual modality data in the knowledge graph to be aligned into the RESNET-152 to extract visual features and obtain the visual embedding.

[0178] Optionally, the feature encoding module 501 includes:

[0179] A feature encoding unit, configured to input the knowledge graph to be aligned into a multimodal embedding module for feature encoding to obtain a multimodal embedding and an initial multimodal embedding;

[0180] An adding unit is configured to add Gaussian mask noise to the initial multimodal embedding to obtain the multimodal embedding.

[0181] Optionally, the multimodal entity feature alignment and fusion device 500 further includes:

[0182] an iterative training module, configured to iteratively train a feature-balanced multimodal entity alignment model, wherein the feature-balanced multimodal entity alignment model comprises the multimodal embedding module, the cross-modal attention fusion module, and the multimodal adaptive feature fusion module;

[0183] The loss function of the iterative training includes a multimodal joint contrast loss and a multimodal joint embedding alignment loss, and the multimodal joint contrast loss satisfies:

[0184]

[0185] in, Contrastive loss for characterizing multimodal fusion embedding, α m is a hyperparameter, Used to represent the unimodal contrast loss, M = {g, r, a, n, v} is used to represent the unimodal embedding of structure, relationship, attribute, name and vision;

[0186] The multimodal joint embedding alignment loss satisfies:

[0187]

[0188] in, Alignment loss for characterizing multimodal fusion embedding, β m is a hyperparameter, Used to characterize unimodal alignment loss.

[0189] Optionally, the loss function of the iterative training satisfies:

[0190]

[0191] The multimodal entity feature alignment and fusion device 500 provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effects are similar, which will not be repeated in this embodiment.

[0192] It should be noted that the division of units in the embodiments of the present application is schematic and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0194] like Figure 6 As shown, an embodiment of the present application provides an electronic device 600, comprising: a memory 602, a processor 601, and a program stored in the memory 602 and executable on the processor 601; the processor 601 is configured to read the program in the memory 602 to implement the steps in the method for aligning and fusing multimodal entity features as described above.

[0195] An embodiment of the present application also provides a readable storage medium, on which a program is stored. When the program is executed by a processor, the various processes of the above-mentioned multimodal entity feature alignment and fusion method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. Among them, the readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as compact disk (CD), digital video disc (DVD), Blu-ray disc (BD), high-definition versatile disc (HVD), etc.), and semiconductor memory (such as read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read only memory (EEPROM), non-volatile memory (NAND FLASH), solid state drive (SSD), etc.).

[0196] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0197] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD-ROM), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0198] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for aligning and fusing multimodal entity features, characterized in that: include: Inputting the knowledge graph to be aligned into a multimodal embedding module for feature encoding to obtain a multimodal embedding, wherein the knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding, and visual embedding; Inputting the multimodal embedding into a cross-modal attention fusion module for feature fusion and extraction to obtain a cross-modal attention fusion embedding, and inputting the multimodal embedding into a multimodal adaptive feature fusion module for feature fusion to obtain an adaptive feature fusion embedding; Based on the multimodal early fusion embedding, the cross-modal attention fusion embedding and the adaptive feature fusion embedding, the entities in the knowledge graph to be aligned are aligned and fused, wherein the multimodal early fusion embedding is obtained by cascading the multimodal embeddings in the embedding layer.

2. The method according to claim 1, characterized in that: The multimodal embedding includes the structural embedding, the multimodal embedding module includes a graph attention network, and the inputting of the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes: The graph structure modal data in the knowledge graph to be aligned is input into the graph attention network to extract graph structure features to obtain the structure embedding.

3. The method according to claim 1, characterized in that: The multimodal embedding includes the text embedding, the multimodal embedding module includes a fully connected feature extraction network, and the knowledge graph to be aligned is input into the multimodal embedding module for feature encoding to obtain the multimodal embedding, including: The text modality data in the knowledge graph to be aligned is input into the corresponding fully connected feature extraction network to perform text feature extraction to obtain the text embedding.

4. The method according to claim 1, characterized in that: The multimodal embedding includes the visual embedding, the multimodal embedding module includes RESNET-15, and the step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain the multimodal embedding includes: The visual modality data in the knowledge graph to be aligned is input into the RESNET-152 to extract visual features and obtain the visual embedding.

5. The method according to any one of claims 1 to 4, characterized in that: The step of inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain a multimodal embedding includes: Inputting the knowledge graph to be aligned into a multimodal embedding module for feature encoding to obtain a multimodal embedding, thereby obtaining an initial multimodal embedding; Gaussian masked noise is added to the initial multimodal embedding to obtain the multimodal embedding.

6. The method according to claim 1, characterized in that: The method further comprises: Iteratively training a feature-balanced multimodal entity alignment model, wherein the feature-balanced multimodal entity alignment model includes the multimodal embedding module, the cross-modal attention fusion module, and the multimodal adaptive feature fusion module; The loss function of the iterative training includes a multimodal joint contrast loss and a multimodal joint embedding alignment loss, and the multimodal joint contrast loss satisfies: in, The contrast loss used to characterize the multimodal fusion embedding, α m is a hyperparameter, Used to represent unimodal contrast loss, M = {g, r, a, n, v} is used to represent unimodal embeddings of structure, relationship, attribute, name and vision; The multimodal joint embedding alignment loss satisfies: in, Alignment loss used to characterize multimodal fusion embedding, β m is a hyperparameter, Used to characterize unimodal alignment loss.

7. The method according to claim 6, characterized in that: The loss function of the iterative training satisfies:

8. A device for aligning and fusing multimodal entity features, characterized in that: include: A feature encoding module, used for inputting the knowledge graph to be aligned into the multimodal embedding module for feature encoding to obtain a multimodal embedding, wherein the knowledge graph to be aligned includes multimodal data, and the multimodal embedding includes at least two of structural embedding, text embedding and visual embedding; A feature fusion module, used for inputting the multimodal embedding into a cross-modal attention fusion module for feature fusion and extraction to obtain a cross-modal attention fusion embedding, and inputting the multimodal embedding into a multimodal adaptive feature fusion module for feature fusion to obtain an adaptive feature fusion embedding; An alignment fusion module is used to align and fuse the entities in the knowledge graph to be aligned based on the multimodal early fusion embedding, the cross-modal attention fusion embedding and the adaptive feature fusion embedding, wherein the multimodal early fusion embedding is obtained by cascading the multimodal embedding in the embedding layer.

9. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the method for aligning and fusing multimodal entity features as described in any one of claims 1 to 7.

10. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps in the method for aligning and fusing multimodal entity features as described in any one of claims 1 to 7 are implemented.

Images