A multi-modal entity alignment pseudo seed generation method, medium, device and product

Abstract

The application provides a multi-modal entity alignment pseudo seed generation method, medium, equipment and product, relates to the technical field of knowledge graph entity alignment, and comprises the following steps: acquiring multi-modal feature vectors of two knowledge graph entities and fusing the multi-modal feature vectors, selecting entity pairs from each clustering cluster of the entities, and acquiring a pseudo seed set in a first stage; calculating the similarity between all entity pairs according to enhanced fusion feature vectors, and selecting entity pairs not in the set as a new pseudo seed set according to the similarity; correcting the set according to the similarity matrix of the entities in the set to obtain a corrected pseudo seed set; selecting entity pairs whose neighbor entities have a similarity higher than a set threshold value η and the entity pairs not in the set to obtain a pseudo seed set, and further correcting the entity pairs in the set to obtain a final pseudo seed set. The application improves the coverage rate while ensuring the quality of the pseudo seeds.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge graph entity alignment, and in particular to a multi-modal entity alignment pseudo seed generation method, medium, device and product. BACKGROUND

[0002] In the knowledge graph (KG) alignment task, some "seed alignment" (i.e., entity pairs representing the same entity in two KGs) are usually needed as a supervision signal to train the model. However, obtaining high-quality real seed alignment usually requires expensive manual annotation. Pseudo seed generation is proposed to solve this problem. It aims to find high-confidence, potential entity alignment pairs from two KGs as "pseudo seeds" to replace or supplement real seed alignment, thereby reducing the dependence on manual annotation.

[0003] The process of generating pseudo seeds is usually as follows: first, learn embedding vectors for entities in two KGs, then calculate the cosine or Euclidean distance between entity pairs in a unified vector space, and finally select entity pairs with a distance less than a certain threshold as pseudo seeds.

[0004] Integrating multi-modal data from different sources (such as pictures, text, videos, audio, etc.) can provide strong support for multi-modal search, cross-modal reasoning, and cross-modal retrieval, and significantly improve information processing and analysis capabilities. Multi-modal entity alignment technology aims to eliminate data silos and achieve comprehensive integration and collaborative use of data by discovering equivalent entities across different data sources and modalities. To overcome the limitation of being difficult to obtain labeled seed pairs, recent research has shifted to an unsupervised paradigm using pseudo alignment seeds. However, there is still a research gap in unsupervised entity alignment in the multi-modal scenario, which is mainly due to: the automatic generation method inevitably produces errors. If the wrong pseudo seeds (noise) are introduced into the training set, the model will be "polluted", leading to a decrease in model performance, and the error will be amplified in the iteration process, forming a vicious cycle (error propagation); and multi-modal information integration often leads to the problem of uneven coverage of pseudo seeds in knowledge graphs. SUMMARY

[0005] The purpose of the present application is to solve the problems of introducing noise and uneven coverage of pseudo seeds in knowledge graph pseudo seed generation. A multi-modal entity alignment pseudo seed generation method is proposed, which includes the following steps:

[0006] S1, obtaining two multi-modal knowledge graphs to be matched and wherein and represent entities, and represent the relationship of the entity, and representing an attribute of an entity, and representing visual information of an entity;

[0007] S2, respectively encoding the multi-modal information of the two multi-modal knowledge graph entities to obtain multi-modal feature vectors of the two multi-modal knowledge graph entities;

[0008] S3, respectively fusing the multi-modal feature vectors of the two multi-modal knowledge graph entities to obtain respective fusion feature vectors of the two multi-modal knowledge graph entities;

[0009] S4, based on the respective fusion feature vectors of the two multi-modal knowledge graph entities, clustering the entities in the two multi-modal knowledge graphs, selecting m entity pairs from each cluster to obtain a pseudo seed set in the first stage ;

[0010] S5, respectively enhancing the multi-modal feature vectors of the two multi-modal knowledge graph entities to obtain enhanced multi-modal feature vectors of the two multi-modal knowledge graph entities;

[0011] S6, respectively fusing the enhanced multi-modal feature vectors of the two multi-modal knowledge graph entities to obtain respective enhanced fusion feature vectors of the two multi-modal knowledge graph entities;

[0012] S7, based on the enhanced fusion feature vectors of the two multi-modal knowledge graph entities, calculating the similarity between all entity pairs in the two multi-modal knowledge graphs, screening entity pairs not in the pseudo seed set , and selecting the n entity pairs with the highest similarity as a new pseudo seed set ;

[0013] S8, calculating a similarity matrix of the entities in the pseudo seed set , and according to the similarity matrix, correcting the pseudo seed set to obtain a corrected pseudo seed set ;

[0014] S9, calculating the similarity of neighbor entities of the entity pairs in the pseudo seed set , screening entity pairs whose neighbor entities have a similarity higher than a set threshold η and which are not in the set , to obtain a pseudo seed set , and further correcting the entity pairs in the pseudo seed set to obtain a final pseudo seed set.

[0015] Further, the multi-modal knowledge graph is represented as wherein E represents an entity, R represents a relationship of the entity, A represents an attribute of the entity, and V represents visual information of the entity;

[0016] The visual information of the entity is encoded by using a ResNet model, and is represented as:

[0017]

[0018] wherein, represents a feature vector of the i-th entity visual information, represents a ResNet model, represents the i-th entity visual information;

[0019] The relationship of the entity and the attribute of the entity are encoded by using a pre-trained BERT, and are represented as:

[0020]

[0021] wherein, represents a feature vector of the i-th entity attribute, represents a feature vector of the i-th entity relationship, AVE represents an average value operation, and BERT represents a pre-trained BERT encoding, represents all attributes of the i-th entity, represents all relationships of the i-th entity.

[0022] Further, the multi-modal feature vectors are fused by using a splicing manner, and are represented as follows:

[0023]

[0024] wherein, represents a final feature vector of the i-th entity, represents a feature vector of the i-th entity visual information, represents a feature vector of the i-th entity attribute, represents a feature vector of the i-th entity relationship, represents vector splicing.

[0025] Further, S4 is specifically:

[0026] By using a K-means clustering method, the entities in the two multi-modal knowledge graphs are divided into K clusters, the similarity of the entity pairs in each cluster is calculated, all entity pairs in each cluster are sorted according to the similarity values from high to low, and entity pairs are selected from each cluster, wherein the number of the selected entity pairs from each cluster satisfies the following formula:

[0027]

[0028] wherein, denotes the number of entity pairs selected from the jth cluster, n denotes the total number of required pseudo seeds, denotes the number of entities in the jth cluster, and denote the number of entities in and, respectively. and denote the number of entities in and, respectively.

[0029] Further, S5 is specifically:

[0030] After the feature vector of the entity visual information is mapped through an independent linear layer, the neighborhood information is aggregated by using a graph attention network GAT to obtain an enhanced feature vector of the entity visual information;

[0031] The feature vector of the entity attribute and the feature vector of the entity relationship are respectively converted by using an independent linear layer to obtain an enhanced feature vector of the entity attribute and an enhanced feature vector of the entity relationship;

[0032] The ICL loss is used when training the above feature enhancement process:

[0033]

[0034] wherein, loss denotes a total loss, denotes an ICL loss, denote enhanced feature vectors of entity visual information in two knowledge graphs, respectively, denote enhanced feature vectors of entity attributes in two knowledge graphs, respectively, denote enhanced feature vectors of entity relationships in two knowledge graphs, respectively.

[0035] Further, S8 is specifically:

[0036] The feature matrix of the entity pairs in the pseudo seed set and is and The similarity matrix of the entities in the pseudo seed set is calculated according to and If the value of the diagonal element in the similarity matrix is not the maximum value, the pseudo seeds and are removed from the pseudo seed set , wherein, is the column number in which the maximum similarity value of the kth row in the similarity matrix M is located.

[0037] Further, the similarity of the neighbor entities of the entity pair is calculated according to the following formula: ​​

[0038]

[0039] wherein, denotes and a similarity of, and denote a neighbor entity of entities and respectively, and denote a neighbor entity set of entities and respectively, denotes an i-th entity in, denotes a j-th entity in, denotes a fusion feature vector of, denotes an enhanced fusion feature vector of, denotes a fusion feature vector of, denotes an enhanced fusion feature vector of.

[0040] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the multi-modal entity alignment pseudo seed generation method.

[0041] The application further provides an electronic device, which comprises a processor and a memory, wherein the processor and the memory are connected to each other, the memory is used for storing a computer program, the computer program comprises computer readable instructions, and the processor is configured to call the computer readable instructions to execute the multi-modal entity alignment pseudo seed generation method.

[0042] The application further provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the steps of the multi-modal entity alignment pseudo seed generation method.

[0043] The application provides the technical scheme and the beneficial effects are as follows:

[0044] The multi-modal entity alignment unsupervised pseudo seed generation method provided by the application gradually expands the pseudo seed scale through the three-stage pseudo seed generation and optimization process of multi-modal fusion and clustering sampling, global sampling and verification, neighborhood expansion and verification, effectively reduces the introduction of false pseudo seeds, and ensures the overall precision of the pseudo seed set. The clustering and global sampling strategy is introduced when selecting the pseudo seed, which avoids the concentration of pseudo seeds in a small number of high-frequency entities or local areas, realizes the balanced coverage of pseudo seeds in the knowledge graph, improves the coverage rate while ensuring the quality of the pseudo seed, and realizes the dynamic balance of quantity and precision. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flowchart of a multi-modal entity alignment pseudo seed generation method of an embodiment of the application.

[0046] Figure 2 is a block diagram of an electronic device in an exemplary embodiment of the application. DETAILED DESCRIPTION

[0047] To make the purpose, technical scheme and advantages of the application clearer, the embodiments of the application will be further described below with reference to the drawings.

[0048] The flowchart of the multi-modal entity alignment pseudo seed generation method of the embodiment of the application is as shown in Figure 1 , and specifically includes the following steps:

[0049] S1, obtaining two multi-modal knowledge graphs to be matched and , wherein and represent entities, and represent the relationship of the entity, and represent the attribute of the entity, and represent the visual information of the entity.

[0050] S2, respectively encoding the multi-modal information of the entities of the two multi-modal knowledge graphs, and obtaining the multi-modal feature vectors of the entities of the two multi-modal knowledge graphs.

[0051] Specifically, the visual information of the entity is encoded using a ResNet model, and the image data generates a 2048-dimensional feature vector after being processed by the ResNet network, represented as:

[0052]

[0053] wherein, represents the i-th feature vector of the entity, ia feature vector of the i-th entity visual information, denotes a ResNet model, denotes the i-th entity visual information. i denotes the i-th entity visual information.

[0054] The relationship of the entity and the attribute of the entity are encoded by using a pre-training BERT to obtain a 768-dimensional feature vector, and are denoted as:

[0055]

[0056] wherein, denotes a feature vector of the i-th entity attribute, denotes a feature vector of the i-th entity relationship, AVE denotes an average value operation, and BERT denotes a pre-training BERT encoding, denotes all attributes of the i-th entity, i denotes all relationships of the i-th entity.

[0057] S3, the multi-modal feature vectors of the two multi-modal knowledge graph entities are fused respectively to obtain the fusion feature vectors of the two multi-modal knowledge graph entities respectively.

[0058] The multi-modal feature vectors are fused in a splicing manner, and are denoted as follows:

[0059]

[0060] wherein, denotes a fusion feature vector of the i-th entity, denotes a feature vector of the i-th entity visual information, denotes a feature vector of the i-th entity attribute, denotes a feature vector of the i-th entity relationship, denotes vector splicing.

[0061] S4, clustering sampling: based on the fusion feature vectors of the two multi-modal knowledge graph entities, the entities in the two multi-modal knowledge graphs are clustered, m entity pairs are selected from each clustering cluster to obtain a pseudo seed set in the first stage .

[0062] The K-means clustering method is used in the application to divide the entities in the two multi-modal knowledge graphs into K clusters, denoted as , the similarity of the entity pairs in each cluster is calculated, and all entity pairs in each cluster are sorted according to the similarity value from high to low, and entity pairs are selected from each cluster, wherein the number of entity pairs selected from each cluster satisfies the following formula:

[0063] ​

[0064] wherein, denotes the number of entity pairs selected from the jth cluster, and n denotes the total number of required pseudo seeds, denotes the number of entities in the jth cluster, and denote the number of entities in and respectively.

[0065] S5, global sampling: the multi-modal feature vectors of the two multi-modal knowledge graph entities are enhanced respectively to obtain enhanced multi-modal feature vectors of the two multi-modal knowledge graph entities. For the visual feature of the entity , first, mapping processing is performed through an independent linear layer, and then neighborhood information is aggregated by using a graph attention network GAT to enhance the expression of the visual feature. For the attribute and relationship features of the entity, an independent linear layer is respectively used to convert the attribute feature and the relationship feature to finally obtain new entity attribute features and relationship features. The specific implementation manner is as follows:

[0066]

[0067] wherein, , and denote the enhanced features of , and respectively, , and denote the feature vector of the visual information of the ith entity, the feature vector of the attribute, and the feature vector of the relationship respectively.

[0068] When training the above feature enhancement process, an ICL loss is used:

[0069]

[0070] wherein, loss denotes a total loss, denotes an ICL loss, denote the enhanced feature vectors of the entity visual information in the two knowledge graphs respectively, denote the enhanced feature vectors of the entity attributes in the two knowledge graphs respectively, denote the enhanced feature vectors of the entity relationships in the two knowledge graphs respectively.

[0071] S6, by means of vector splicing, the enhanced multi-modal feature vectors of the two multi-modal knowledge graph entities are fused respectively to obtain respective enhanced fusion feature vectors of the two multi-modal knowledge graph entities.

[0072] S7, based on the two multi-modal knowledge graph entity enhanced fusion feature vector, calculate the similarity between all entity pairs in the two multi-modal knowledge graph, according to the similarity score, filter out the entity pairs not in the pseudo seed set S7, based on the two multi-modal knowledge graph entity enhanced fusion feature vector, calculate the similarity between all entity pairs in the two multi-modal knowledge graph, according to the similarity score, filter out the entity pairs not in the pseudo seed set .

[0073] S8, multi-modal information error correction: calculate the similarity matrix of the entities in the pseudo seed set S8, multi-modal information error correction: calculate the similarity matrix of the entities in the pseudo seed set S8, multi-modal information error correction: calculate the similarity matrix of the entities in the pseudo seed set .

[0074] Specifically:

[0075] The feature matrix of the entity pair in the pseudo seed set and is and , wherein is the entity in , is the entity in . According to and , the similarity matrix of the entities in the pseudo seed set is calculated , if the value of the diagonal element in the similarity matrix is not the maximum value, remove the pseudo seeds and from the pseudo seed set , wherein is the column number where the maximum similarity value of the kth row in the similarity matrix M is located.

[0076] The similarity matrix M is as follows:

[0077]

[0078] wherein represents the similarity between entity and entity , represents the similarity between entity and entity , represents the similarity between entity and entity , represents the similarity between entity and entity .

[0079] S9, neighborhood expansion: by expanding the pseudo-aligned seed pairs of adjacent entities to increase the number of seeds, while supplementing the information of the scattered entities around the knowledge graph, so as to enhance the attention to the originally scattered entities in the gradient update process. Calculate the pseudo seed set The similarity of the neighbor entities of the entity pair in the middle, the calculation formula is as follows:

[0080]

[0081] Among them, The similarity of and , and respectively represent a neighbor entity of entity and , and respectively represent the neighbor entity set of entity and , represents the i-th entity in , represents the j-th entity in , represents the fusion feature vector of , represents the enhanced fusion feature vector of , represents the fusion feature vector of , represents the enhanced fusion feature vector of .

[0082] The similarity of the neighbor entity is higher than the set threshold η, and the entity pair is not in the set The entity pair is obtained from the pseudo seed set , and the entity pair in the pseudo seed set is further corrected, and the information correction method in S8 is adopted to obtain the final pseudo seed set.

[0083] To verify the effectiveness of the method of the present application, the method of generating pseudo seeds (PSQE) of the present application and different existing entity alignment methods are combined, and three public datasets are selected: DBP15K (Chinese and English alignment), DBP15K (Japanese and English alignment), and DBP15K (French and English alignment). The existing different methods include: RDGCN (relation-aware dual graph convolutional network), RPR-RHGT (from the paper: Entity Alignment with Reliable Path Reasoning and Relation-Aware Heterogeneous Graph Transformer), DESAlign (from the paper: Towards Semantic Consistency: Dirichlet Energy Driven Robust Multi-Modal Entity Alignment), EVA (from the paper: Visual pivoting for (unsupervised) entity alignment), EVA+PSQE, MCLEA (from the paper Multi-modal Contrastive Representation Learning for Entity Alignment), MCLEA+PSQE, MEAformer (from the paper MEAformer: Multi-modal entity alignment transformer for meta modality hybrid), MEAformer+PSQE, PCMEA (from the paper Pseudo-Label Calibration Semi-supervised Multi-Modal Entity Alignment), PCMEA+PSQE. The experimental results of the method of the present application combined with different entity alignment methods on the three public datasets are compared in reference table 1. The indicators used include MRR (Mean Reciprocal Ranking) and HITS@n.

[0084]

[0085] wherein S is a set of triples, |S| is the number of triples, represents the link prediction ranking of the i-th triple. The larger the MRR index is, the better.

[0086]

[0087] Wherein, II represents an indicator function, if the condition is true, the function value is 1, otherwise 0. The larger the HITS@n index is, the better.

[0088] Table 1

[0089]

[0090] The multi-modal entity alignment unsupervised pseudo seed generation method provided by the application significantly improves the quality and stability of pseudo seed generation through a multi-stage iterative optimization strategy. Relying on the precision control and distribution control mechanism, the application can effectively eliminate false pseudo seeds and avoid the concentration of pseudo seeds in local areas, thereby ensuring the balance of the generated results in accuracy and coverage. At the same time, combined with multi-modal feature fusion and expression enhancement technology, the discriminability and robustness of entity vector representation are improved, so that good alignment performance can be maintained even in the presence of noise and information imbalance. Overall, the precision and coverage of the pseudo seeds generated by this method under unsupervised conditions are better than those of existing methods, providing a solid foundation for cross-modal alignment and fusion of multi-modal knowledge graphs.

[0091] In an exemplary embodiment, a computer readable storage medium is included, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the multi-modal entity alignment pseudo seed generation method described above.

[0092] Please refer to Figure 2 In an exemplary embodiment, an electronic device is also included, which includes at least one processor, at least one memory, and at least one communication bus.

[0093] The memory has a computer program stored thereon, and the computer program includes computer readable instructions. The processor invokes the computer readable instructions stored in the memory through the communication bus to execute the multi-modal entity alignment pseudo seed generation method described above.

[0094] In an exemplary embodiment, a computer program product is provided, which includes computer programs / instructions. The computer programs / instructions are executed by a processor to implement the steps of the multi-modal entity alignment pseudo seed generation method described above.

[0095] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-modal entity alignment pseudo seed generation method, characterized in that, The method comprises the following steps: S1, acquiring two multi-modal knowledge graphs to be matched and ; S2, respectively, to and Encode the multi-modal information of the entity, obtain and Multi-modal feature vector of the entity; S3, respectively to and fuse the multi-modal feature vectors of the entities to obtain and the respective fused feature vectors of the entities; S4, based on and the respective fusion feature vectors of the entities, clustering the entities in the entity set, selecting m entity pairs from each cluster, and obtaining a first-stage pseudo seed set and ;​ S5、respectively to and enhancing the multi-modal feature vector of the entity to obtain and the enhanced multi-modal feature vector of the entity; S6、respectively to and fuse the enhanced multi-modal feature vectors of the entities to obtain and the respective enhanced fusion feature vectors of the entities; S7、based on and enhanced fusion feature vector of the entity, calculating and the similarity between all entity pairs in the set , selecting the top n entity pairs with the highest similarity as the new pseudo seed set ; S8、calculating the set a similarity matrix of the entities, and correcting the set according to the similarity matrix to obtain a corrected pseudo-seed set ; S9、computing the set the similarity of the neighbor entities of the entity pair in the set the entity pair in the set , and further refining the entity pair in the set , to obtain the final pseudo seed set.

2. The method of claim 1, wherein, Multi-modal knowledge graph representation is wherein E represents an entity, R represents a relationship of the entity, A represents an attribute of the entity, and V represents visual information of the entity. The visual information of the entity is encoded by using a ResNet model, and is represented as: in, The feature vector representing the visual information of the i-th entity. Represents the ResNet model. Indicates the first i Visual information of each entity; The relationship of the entity and the attribute of the entity are encoded by using a pre-trained BERT, and are represented as: wherein, denotes a feature vector of the i-th entity attribute, denotes a feature vector of the i-th entity relation, AVE denotes an averaging operation, and BERT denotes a pre-trained BERT encoding, denotes all attributes of the i-th entity, denotes all relations of the i-th entity.

3. The method of claim 1, wherein, The multi-modal feature vectors are fused by using a splicing manner, and are represented as: wherein, represents the final feature vector of the i-th entity, represents the feature vector of the i-th entity visual information, represents the feature vector of the i-th entity attribute, represents the feature vector of the i-th entity relationship, represents vector concatenation.

4. The method of claim 1, wherein, S4 is specifically: K-means clustering method is used to divide the entities in the two multi-modal knowledge graphs into K clusters, the similarity of the entity pairs in each cluster is calculated, all entity pairs in each cluster are sorted according to the similarity value from high to low, and entity pairs are selected from each cluster, wherein the number of entity pairs selected from each cluster satisfies the following formula: wherein, represents the number of entity pairs selected from the jth cluster, n represents the total number of required pseudo seeds, represents the number of entities in the jth cluster, and respectively represent the number of entities in and .

5. The method of claim 1, wherein, S5 is specifically: After the feature vector of the entity visual information is mapped through an independent linear layer, the neighborhood information is aggregated by using a graph attention network GAT to obtain an enhanced feature vector of the entity visual information; The feature vector of the entity attribute and the feature vector of the entity relationship are respectively converted by using independent linear layers to obtain an enhanced feature vector of the entity attribute and an enhanced feature vector of the entity relationship; The ICL loss is used when training the above feature enhancement process: wherein loss denotes the total loss, denotes the ICL loss, denote the enhanced feature vectors of the entity visual information in the two knowledge graphs, respectively, denote the enhanced feature vectors of the entity attributes in the two knowledge graphs, respectively, denote the enhanced feature vectors of the entity relationships in the two knowledge graphs, respectively.

6. The method of claim 1, wherein, S8 is specifically: Pseudo-seed set Middle entity pair And The feature matrix of And , according to And Calculate the pseudo-seed set The similarity matrix of the middle entity The diagonal element in the similarity matrix If the value of Remove the pseudo-seed And , wherein The column number where the kth row similarity maximum value in the similarity matrix M is located.

7. The method of claim 1, wherein, The similarity of the neighbor entities of the entity pair is calculated according to the following formula: wherein, denotes and a similarity of and denote a neighbor entity of entities and respectively, and denote a set of neighbor entities of entities and respectively, denotes the i-th entity in denotes the j-th entity in denotes the i-th entity in denotes the j-th entity in denotes a fused feature vector of denotes an enhanced fused feature vector of denotes a fused feature vector of denotes an enhanced fused feature vector of denotes a fused feature vector of denotes an enhanced fused feature vector of denotes a fused feature vector of denotes an enhanced fused feature vector of 8. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to realize the method of any one of claims 1-7.

9. An electronic device, comprising: The computer program is executed by the processor to realize the method of any one of claims 1-7.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps of the method of any one of claims 1-7.

Images