Entity alignment method based on multi-modal data fusion and neighborhood matching

Abstract

The invention discloses an entity alignment method based on multi-modal data fusion and neighborhood matching, and the method comprises the steps: firstly, capturing the multi-dimensional features of an entity through the fusion of a structural feature, an attribute feature and an image feature, and more accurately recognizing the entity representing the same real world object in different knowledge maps, thereby improving the entity alignment accuracy; secondly, based on dynamic feature fusion of an attention mechanism, the weight of each modal feature can be adjusted according to the characteristics of data, so that the model can dynamically fuse different features, and the fusion result is more accurate; and finally, by calculating neighborhood matching scores between entities and taking the scores as screening standards, entity pairs with the neighborhood matching scores lower than a preset threshold value are effectively identified and eliminated, so that mismatching items are filtered, and the reliability of an alignment result is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to an entity alignment method based on multimodal data fusion and neighborhood matching. Background Art

[0002] Due to the diversity of knowledge graph data sources, the same entity is often represented differently in different knowledge graphs, which increases the difficulty of data integration. In addition, in real application scenarios, the coverage of a single knowledge graph is limited, which to a certain extent limits the effective use of knowledge graphs by downstream applications. By adopting knowledge graph entity alignment technology, knowledge graphs from multiple different sources can be merged to obtain a larger knowledge graph with wider coverage.

[0003] At present, although there are some knowledge graph entity alignment methods, these methods have certain limitations. First, when performing entity alignment, existing methods mainly rely on text information, and often ignore the impact of image information on entity alignment. For example, "desktop" may refer to a physical desktop or a computer desktop in a text environment. In addition, existing methods often only make entity alignment judgments based on the similarity between entities, which can easily lead to incorrect entity alignment, which to some extent affects the accuracy of entity alignment.

[0004] The steps of the technical solution of "A Multimodal Entity Alignment Method Based on Adaptive Feature Fusion" by the National University of Defense Technology of the People's Liberation Army of China are as follows: first, obtain the data of two multimodal knowledge graphs; second, learn the structural features and visual features of the entities respectively; finally, combine the structural and visual features and perform entity alignment. However, this method mainly focuses on learning the structural features and image features of the knowledge graph, but does not mine the attribute information of the entity. In addition, this method directly aligns entities based on entity similarity, but fails to consider the important role of neighborhood structure differences in distinguishing misaligned entities.

[0005] Through the analysis of the existing technology, it is found that the main shortcomings of the existing technology include:

[0006] (1) Insufficient entity information integration: When performing entity alignment tasks, we often face the problem of insufficient entity information integration, ignoring the complementarity between different information sources. Therefore, when faced with complex and changeable entity data, it is often difficult to capture comprehensive entity features, which in turn affects the accuracy of the alignment results.

[0007] (2) Lack of dynamic vector fusion capability: When fusing the vector representation of multimodal data, fixed weights or simple splicing methods are usually used, which cannot dynamically adjust the importance of different modal vector features according to actual conditions. This static fusion method may ignore important information or introduce noise when processing diverse entity data.

[0008] (3) Insufficient handling of misalignment: When performing the entity alignment task, entity alignment is performed only based on the similarity between entities. There is a lack of effective mechanism to filter out misaligned entities, which leads to incorrect matches in the alignment results and reduces the accuracy of entity alignment. Summary of the invention

[0009] The present invention aims to solve the above problems existing in the prior art and provide an entity alignment method based on multimodal data fusion and neighborhood matching.

[0010] The technical solution of the present invention is: an entity alignment method based on multimodal data fusion and neighborhood matching, taking the knowledge graph to be aligned as input, and proceeding according to the following steps:

[0011] Step 1. Use graph convolutional networks to extract the structural features and attribute features of the knowledge graph. The structural features are represented by the structural vector H s , the attribute feature is represented by the attribute vector H a ;

[0012] Step 2: Use the residual network to extract entity image features, where the image features are represented by image vectors H i ;

[0013] Step 3. Based on the attention mechanism, the structural features, attribute features and image features are dynamically fused to obtain the entity joint vector representation H j ;

[0014] Step 4. Represent H based on entity joint vector j , calculate the Manhattan distance between entity joint vectors to determine entity similarity;

[0015] Step 5. Calculate the neighborhood matching scores between entities;

[0016] Step 6. Obtain preliminary entity alignment results based on entity similarity, predict unmatched entities based on neighborhood matching scores between entities, filter out unmatched entities from the preliminary entity alignment results, and obtain the final entity alignment results.

[0017] In step 1, the graph convolutional network aggregates the information of neighbor nodes through layer-by-layer convolution operations. The structure vector representation and attribute vector representation of the lth layer are respectively expressed as:

[0018]

[0019] Where A' is obtained by summing the adjacency matrix and the identity matrix; D' is the diagonal node degree matrix of A'; σ(·) is the ReLU activation function; and is the weight matrix.

[0020] The step 2 is to use the residual network to perform forward propagation for each input image, and map the image features into a vector representation through a trainable feedforward layer to obtain the image vector representation H i , the calculation formula is:

[0021] H i =W i ResNet(Ι)+b i

[0022] Among them, W i is the weight matrix, b i is the bias vector.

[0023] The step 3 is to use the attention mechanism to dynamically learn the structure vector representation H of the entity s , attribute vector representation H a and the image vector representation H i The importance of each modality is calculated through the attention mechanism, and then the structural vector H of the entity is represented s , attribute vector representation H a and the image vector representation H i Perform dynamic vector concatenation to obtain entity joint vector representation H j ; The calculation formula is:

[0024]

[0025] α m =softmax(o m )

[0026]

[0027] Among them, α m is the contribution fraction of mode m; H m is the vector representation of mode m; s, a, i represent structure, attribute and image modes respectively; || represents vector concatenation; W m and u m is an independent trainable matrix for each modality m; δ(·) is the LeakyReLU function.

[0028] Step 4 is based on the entity joint vector representation H j, calculate the Manhattan distance between the joint vectors of the two entities in the vector space to determine the entity similarity and obtain the preliminary entity alignment result. The calculation formula is:

[0029]

[0030] in, and are the joint vector representations of entities e1 and e2 respectively.

[0031] The step 5 is to approximate the relationship vector representation with the structural vector representation of the entity, specifically, taking the average of the structural vector representations of all head entities and tail entities connected by the relationship as the relationship vector representation; performing relationship-aware neighborhood matching, for two entities to be aligned, based on the relationships r1, r2 and n1, n2 they are linked to, the alignment probability of the two entities is calculated:

[0032] C(r1,r2,n1,n2)=C(r1,n1)·C(r2,n2)

[0033]

[0034] Among them, C(r1,n1) and C(r2,n2) represent the mapping probabilities of entities and relations; T1 and T2 represent the sets of relation triplets where r1,n1 and r2,n2 are located respectively;

[0035] Based on the alignment probability, the neighborhood matching score between the two entities is further calculated:

[0036]

[0037] in, is a matching set; and is the set of corresponding entity pairs.

[0038] The step 6 of predicting unmatched entities based on the neighborhood matching scores between entities is to regard the two entities as unmatched entities when the neighborhood matching score between the two entities is less than a set neighborhood matching score threshold.

[0039] The present invention also includes inputting seed data and training the model using a margin-based loss function, wherein the loss function is as follows:

[0040]

[0041] Among them, γ is the margin parameter, which represents the margin between positive examples and negative examples; L' represents the negative example set of L, and the negative samples are calculated by the nearest neighbor negative sampling method.

[0042] Compared with the prior art, the present invention can effectively improve the accuracy of entity alignment, as shown in the following:

[0043] (1) Comprehensiveness of multimodal data fusion: By fusing structural features, attribute features, and image features, the present invention can capture the multi-dimensional features of entities, more accurately identify entities representing the same real-world objects in different knowledge graphs, and thus improve the accuracy of entity alignment.

[0044] (2) Dynamic feature fusion based on attention mechanism: The attention mechanism can adjust the weights of each modal feature according to the characteristics of the data, so that the model can dynamically fuse different features to ensure that the fusion result is more accurate.

[0045] (3) Accurate filtering of neighborhood matching: By calculating the neighborhood matching score between entities and using this score as a screening criterion, we can effectively identify and eliminate entity pairs whose neighborhood matching scores are lower than the preset threshold, thereby filtering out false matches in the entity alignment results to ensure the reliability of the alignment results. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of an embodiment of the present invention.

[0047] Figure 2 It is a model structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to better understand the present invention, some relevant knowledge and its definitions used in the embodiments of the present invention are explained as follows:

[0049] (1) Knowledge graph: A knowledge graph is a semantic network that reveals the relationships between entities.

[0050] (2) Entity: A specific object or thing in the real world or conceptual world.

[0051] (3) Entity alignment: matching entities that come from different knowledge bases or data sources but represent the same real-world objects.

[0052] (4) Multimodal data: Multimodal data refers to a data set that contains multiple different modalities. Each modality can be a different type of data, such as text, images, etc.

[0053] (5) Data fusion: Integrate and process data from different data sources.

[0054] (6) Seed data: refers to pre-known aligned entity pairs, that is, entity pairs that have been confirmed in different knowledge bases or data sets to represent the same object in the real world, and are often used as training data.

[0055] (7) Graph Convolutional Network: Graph Convolutional Network (GCN) is a neural network used to process graph structured data. It combines the structural information of the graph with the node features and uses convolution operations to learn the node’s neighbor information.

[0056] (8) Residual Network: Residual Network (ResNet) solves the gradient vanishing problem in deep network training by introducing residual connections, thereby allowing the construction of deeper and better performing neural network structures.

[0057] (9) Loss function: The loss function is a function used to measure the difference or error between the model's predicted value and the true value.

[0058] The following is a detailed introduction to an entity alignment method based on multimodal data fusion and neighborhood matching provided by an embodiment of the present invention. The knowledge graph to be aligned is used as input. The specific process is as follows: Figure 1 As shown, follow the steps below:

[0059] Step 1. Use graph convolutional networks to extract the structural features and attribute features of the knowledge graph. The structural features are represented by the structural vector H s , the attribute feature is represented by the attribute vector H a ;

[0060] Step 2: Use the residual network to extract entity image features, where the image features are represented by image vectors H i ;

[0061] Step 3. Based on the attention mechanism, the structural features, attribute features and image features are dynamically fused to obtain the entity joint vector representation H j ;

[0062] Step 4. Represent H based on entity joint vector j , calculate the Manhattan distance between entity joint vectors to determine entity similarity;

[0063] Step 5. Calculate the neighborhood matching scores between entities;

[0064] Step 6. Obtain preliminary entity alignment results based on entity similarity, predict unmatched entities based on neighborhood matching scores between entities, filter out unmatched entities from the preliminary entity alignment results, and obtain the final entity alignment results.

[0065] The model structure of the embodiment of the present invention is as follows Figure 2 As shown, it includes a multimodal data fusion module, a neighborhood matching score calculation module and an entity alignment module.

[0066] The multimodal data fusion module completes the following steps:

[0067] Step 1. Use graph convolutional networks to extract the structural features and attribute features of the knowledge graph. The structural features are represented by the structural vector H s , the attribute feature is represented by the attribute vector H a ;

[0068] Specifically, the graph convolutional network aggregates the information of neighboring nodes through layer-by-layer convolution operations. The structure vector representation and attribute vector representation of the lth layer are respectively expressed as:

[0069]

[0070] Where A' is obtained by summing the adjacency matrix and the identity matrix; D' is the diagonal node degree matrix of A'; σ(·) is the ReLU activation function; and is the weight matrix;

[0071] Step 2: Use the residual network to extract entity image features, where the image features are represented by image vectors H i ;

[0072] Specifically, a residual network is used to perform forward propagation for each input image, and the image features are mapped to a vector representation through a trainable feedforward layer to obtain the image vector representation H i , the calculation formula is:

[0073] H i =W i ResNet(Ι)+b i

[0074] Among them, W i is the weight matrix, b i is the bias vector;

[0075] Step 3. Based on the attention mechanism, the structural features, attribute features and image features are dynamically fused to obtain the entity joint vector representation H j ;

[0076] Specifically, the attention mechanism is used to dynamically learn the structural vector representation H of the entity. s , attribute vector representation H a and the image vector representation H i The importance of each modality is calculated through the attention mechanism, and then the structural vector H of the entity is represented s , attribute vector representation H a and the image vector representation H i Perform dynamic vector concatenation to obtain entity joint vector representation H j ; The calculation formula is:

[0077]

[0078] α m =softmax(o m )

[0079]

[0080] Among them, α m is the contribution fraction of mode m; H m is the vector representation of mode m; s, a, i represent structure, attribute and image modes respectively; || represents vector concatenation; W m and u m is an independent trainable matrix for each modality m; δ(·) is the LeakyReLU function;

[0081] Step 4. Based on the entity joint vector representation, the Manhattan distance between the entity joint vectors is calculated to determine the entity similarity;

[0082] Specifically, based on the entity joint vector representation, the Manhattan distance between two entity joint vector representations in the vector space is calculated to determine the entity similarity. The smaller the distance, the greater the similarity. The calculation formula is:

[0083]

[0084] in, and are the joint vector representations of entities e1 and e2 respectively;

[0085] The module for calculating the neighborhood matching score completes step 5 above, as follows:

[0086] The structural vector representation of the entity is used to approximate the relationship vector representation. Specifically, the average of the structural vector representations of all the head entities and tail entities connected by the relationship is used as the relationship vector representation; considering the relationship mapping (such as one-to-one and one-to-many), and considering the matching of the first-order entities and relationships linked by the entity, the neighborhood matching score is calculated based on the relationship-aware neighborhood matching; for two entities to be aligned, the alignment probability of the two entities is calculated based on the relationships r1, r2 and n1, n2 they are linked to:

[0087] C(r1,r2,n1,n2)=C(r1,n1)·C(r2,n2)

[0088]

[0089] Among them, C(r1,n1) and C(r2,n2) represent the mapping probabilities of entities and relations; T1 and T2 represent the sets of relation triplets where r1,n1 and r2,n2 are located respectively;

[0090] Based on the alignment probability, the neighborhood matching score between the two entities is further calculated:

[0091]

[0092] in, is a matching set; and is the set of corresponding entity pairs;

[0093] The entity alignment module completes the above step 6, specifically obtaining a preliminary entity alignment result based on entity similarity, and predicting unmatched entities based on the neighborhood matching scores between entities. That is, when the neighborhood matching score between two entities is less than the set neighborhood matching score threshold, the two entities are regarded as unmatched entities, and the unmatched entities are filtered out from the preliminary entity alignment result to obtain the final entity alignment result.

[0094] The method also includes inputting seed data and training the model using a margin-based loss function, wherein the loss function is as follows:

[0095]

[0096] Among them, γ is the margin parameter, which represents the margin between positive examples and negative examples; L' represents the negative example set of L, and the negative samples are calculated by the nearest neighbor negative sampling method.

Claims

1. An entity alignment method based on multimodal data fusion and neighborhood matching, taking the knowledge graph to be aligned as input, characterized in that Follow these steps: Step 1. Use graph convolutional networks to extract the structural features and attribute features of the knowledge graph. The structural features are represented by the structural vector H s , the attribute feature is represented by the attribute vector H a ; Step 2: Use the residual network to extract entity image features, where the image features are represented by image vectors H i ; Step 3. Based on the attention mechanism, the structural features, attribute features and image features are dynamically fused to obtain the entity joint vector representation H j ; Step 4. Represent H based on entity joint vector j , calculate the Manhattan distance between entity joint vectors to determine entity similarity; Step 5. Calculate the neighborhood matching scores between entities; Step 6. Obtain preliminary entity alignment results based on entity similarity, predict unmatched entities based on neighborhood matching scores between entities, filter out unmatched entities from the preliminary entity alignment results, and obtain the final entity alignment results.

2. The entity alignment method based on multimodal data fusion and neighborhood matching according to claim 1, characterized in that: In step 1, the graph convolutional network aggregates the information of neighbor nodes through layer-by-layer convolution operations. The structure vector representation and attribute vector representation of the lth layer are respectively expressed as: Where A' is obtained by summing the adjacency matrix and the identity matrix; D' is the diagonal node degree matrix of A'; σ(·) is the ReLU activation function; and is the weight matrix.

3. The entity alignment method based on multimodal data fusion and neighborhood matching according to claim 2, characterized in that: The step 2 is to use the residual network to perform forward propagation for each input image, and map the image features into a vector representation through a trainable feedforward layer to obtain the image vector representation H i , the calculation formula is: H i =W i ·ResNet(I)+b i Among them, W i is the weight matrix, b i is the bias vector.

4. The entity alignment method based on multimodal data fusion and neighborhood matching according to claim 3, characterized in that: The step 3 is to use the attention mechanism to dynamically learn the structure vector representation H of the entity s , attribute vector representation H a and the image vector representation H i The importance of each modality is calculated through the attention mechanism, and then the structural vector H of the entity is represented s , attribute vector representation H a and the image vector representation H i Perform dynamic vector concatenation to obtain entity joint vector representation H j ; The calculation formula is: α m =softmax(o m ) Among them, α m is the contribution fraction of mode m; H m is the vector representation of mode m; s, a, i represent structure, attribute and image modes respectively; || represents vector concatenation; W m and u m is an independent trainable matrix for each modality m; δ(·) is the LeakyReLU function.

5. The entity alignment method based on multimodal data fusion and neighborhood matching according to claim 4, characterized in that: Step 4 is based on the entity joint vector representation H j , calculate the Manhattan distance between the joint vectors of the two entities in the vector space to determine the entity similarity and obtain the preliminary entity alignment result. The calculation formula is: in, and are the joint vector representations of entities e1 and e2 respectively.

6. The entity alignment method based on multimodal data fusion and neighborhood matching according to claim 5, characterized in that: The step 5 is to approximate the relationship vector representation with the structural vector representation of the entity, specifically, taking the average of the structural vector representations of all head entities and tail entities connected by the relationship as the relationship vector representation; performing relationship-aware neighborhood matching, for two entities to be aligned, based on the relationships r1, r2 and n1, n2 they are linked to, the alignment probability of the two entities is calculated: C(r1,r2,n1,n2)=C(r1,n1)·C(r2,n2) Among them, C(r1,n1) and C(r2,n2) represent the mapping probabilities of entities and relations; T1 and T2 represent the sets of relation triplets where r1,n1 and r2,n2 are located respectively; Based on the alignment probability, the neighborhood matching score between the two entities is further calculated: in, is a matching set; and is the set of corresponding entity pairs.

7. The entity alignment method based on multimodal data fusion and neighborhood matching according to claim 6, characterized in that: The step 6 of predicting unmatched entities based on the neighborhood matching scores between entities is to regard the two entities as unmatched entities when the neighborhood matching score between the two entities is less than a set neighborhood matching score threshold.

8. The entity alignment method based on multimodal data fusion and neighborhood matching according to claim 7, characterized in that: It also includes inputting seed data and training the model using a margin-based loss function, which is as follows: Among them, γ is the margin parameter, which represents the margin between positive examples and negative examples; L' represents the negative example set of L, and the negative samples are calculated by the nearest neighbor negative sampling method.

Images