Multi-modal entity alignment pseudo seed generation method, medium, equipment and product

Abstract

The invention provides a multi-modal entity alignment pseudo seed generation method, a medium, equipment and a product, and relates to the technical field of knowledge graph entity alignment, and the method comprises the steps: obtaining multi-modal feature vectors of two knowledge graph entities, carrying out the fusion, selecting an entity pair from each cluster of the entities, and obtaining a pseudo seed set of a first stage; calculating the similarity among all entity pairs according to the enhanced fusion feature vector, and screening the entity pairs which are not in the set as a new pseudo seed set according to the similarity; correcting the set according to the similarity matrix of the entities in the set to obtain a corrected pseudo seed set; the similarity of the neighbor entities of the entity pairs is higher than a set threshold eta, and the entity pairs are not in the set, a pseudo seed set is obtained, the entity pairs in the set are further corrected, and a final pseudo seed set is obtained. According to the method, the coverage rate is increased while the quality of pseudo seeds is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge graph entity alignment, in particular to a multi-modal entity alignment pseudo seed generation method, medium, device and product. BACKGROUND

[0002] In the knowledge graph (KG) alignment task, some "seed alignment" (i.e., entity pairs in two KGs representing the same entity) are usually needed as a supervision signal to train the model. However, obtaining high-quality real seed alignment usually requires expensive manual annotation. Pseudo seed generation is proposed to solve this problem. It aims to find high-confidence, potential entity alignment pairs from two KGs as "pseudo seeds" to replace or supplement real seed alignment, thereby reducing the dependence on manual annotation.

[0003] The process of generating pseudo seeds is usually as follows: first, learn embedding vectors for entities in two KGs, then calculate the cosine or Euclidean distance between entity pairs in a unified vector space, and finally select entity pairs with a distance less than a certain threshold as pseudo seeds.

[0004] Integrating multi-modal data from different sources (such as pictures, text, videos, audio, etc.) can provide strong support for multi-modal search, cross-modal reasoning, and cross-modal retrieval applications, significantly improving information processing and analysis capabilities. Multi-modal entity alignment technology aims to eliminate data silos and achieve comprehensive integration and collaborative use of data by discovering equivalent entities across different data sources and modalities. To overcome the limitation of being difficult to obtain labeled seed pairs, recent research has shifted to an unsupervised paradigm using pseudo alignment seeds. However, there is still a research gap in unsupervised entity alignment in the multi-modal scenario, which mainly stems from: the automatic generation method inevitably produces errors. If the wrong pseudo seeds (noise) are introduced into the training set, the model will be "polluted", leading to a decline in model performance, and the error will be amplified in the iteration process, forming a vicious cycle (error propagation); and multi-modal information integration often leads to the problem of uneven coverage of pseudo seeds in knowledge graphs. SUMMARY

[0005] The purpose of the present application is to solve the problems of introducing noise and uneven coverage of pseudo seeds in knowledge graph pseudo seed generation. A multi-modal entity alignment pseudo seed generation method is proposed, which includes the following steps: S1, obtaining two multi-modal knowledge graphs to be matched and wherein and represent entities, and represent the relationship of the entity, and representing the visual information of the entity, and representing the visual information of the entity; S2, respectively, the multi-modal information of the two multi-modal knowledge graph entities is encoded, and the multi-modal feature vectors of the two multi-modal knowledge graph entities are obtained; S3, respectively, the multi-modal feature vectors of the two multi-modal knowledge graph entities are fused, and the fusion feature vectors of the two multi-modal knowledge graph entities are obtained; S4, based on the fusion feature vectors of the two multi-modal knowledge graph entities, the entities in the two multi-modal knowledge graphs are clustered, and m entity pairs are selected from each cluster to obtain a pseudo seed set ; S5, respectively, the multi-modal feature vectors of the two multi-modal knowledge graph entities are enhanced, and the enhanced multi-modal feature vectors of the two multi-modal knowledge graph entities are obtained; S6, respectively, the enhanced multi-modal feature vectors of the two multi-modal knowledge graph entities are fused, and the enhanced fusion feature vectors of the two multi-modal knowledge graph entities are obtained; S7, based on the enhanced fusion feature vectors of the two multi-modal knowledge graph entities, the similarity between all entity pairs in the two multi-modal knowledge graphs is calculated, and entity pairs not in the pseudo seed set are screened, and the n entity pairs with the highest similarity are selected as a new pseudo seed set ; S8, the similarity matrix of the entities in the pseudo seed set is calculated, and the pseudo seed set is corrected according to the similarity matrix to obtain a corrected pseudo seed set ; S9, the similarity of the neighbor entities of the entity pairs in the pseudo seed set is calculated, and the entity pairs whose neighbor entities have a similarity higher than a set threshold η and are not in the set are screened to obtain a pseudo seed set , and the entity pairs in the pseudo seed set are further corrected to obtain a final pseudo seed set.

[0006] Further, the multi-modal knowledge graph is represented as , wherein E represents an entity, R represents a relationship of the entity, A represents an attribute of the entity, and V represents visual information of the entity; The visual information of the entity is encoded using a ResNet model and represented as:

[0007] wherein, denotes the feature vector of the i-th entity visual information, denotes the ResNet model, denotes the i-th entity visual information; The relationship of the entity and the attribute of the entity are encoded by using the pre-trained BERT, and are denoted as:

[0008] wherein, denotes the feature vector of the i-th entity attribute, denotes the feature vector of the i-th entity relationship, AVE denotes the operation of averaging, and BERT denotes the pre-trained BERT encoding, denotes all attributes of the i-th entity, denotes all relationships of the i-th entity.

[0009] Further, the multi-modal feature vectors are fused by using a splicing manner, and are denoted as follows:

[0010] wherein, denotes the final feature vector of the i-th entity, denotes the feature vector of the i-th entity visual information, denotes the feature vector of the i-th entity attribute, denotes the feature vector of the i-th entity relationship, denotes vector splicing.

[0011] Further, S4 is specifically: The K-means clustering method is used to divide the entities in the two multi-modal knowledge graphs into K clusters, the similarity of the entity pairs in each cluster is calculated, all entity pairs in each cluster are sorted according to the similarity value from high to low, and the entity pairs are selected from each cluster, wherein the number of the entity pairs selected from each cluster satisfies the following formula:

[0012] wherein, denotes the number of entity pairs selected from the j-th cluster, and n denotes the total number of required pseudo seeds, denotes the number of entities in the j-th cluster, and denote the number of entities in and respectively.

[0013] Further, S5 is specifically: The feature vector of the entity visual information is mapped through an independent linear layer, and then neighborhood information is aggregated by using a graph attention network GAT to obtain an enhanced feature vector of the entity visual information. The feature vector of the entity attribute and the feature vector of the entity relationship are respectively converted by using independent linear layers to obtain an enhanced feature vector of the entity attribute and an enhanced feature vector of the entity relationship. The ICL loss is used when training the above feature enhancement process:

[0014] wherein loss represents a total loss, represents an ICL loss, respectively represent enhanced feature vectors of entity visual information in the two knowledge graphs, respectively represent enhanced feature vectors of entity attributes in the two knowledge graphs, respectively represent enhanced feature vectors of entity relationships in the two knowledge graphs.

[0015] Further, S8 is specifically: a pseudo seed set entity pairs in the pseudo seed set and The feature matrix of and is and According to and , the similarity matrix of entities in the pseudo seed set is calculated The diagonal element of the similarity matrix is removed from the pseudo seed set if the value of is not the maximum value. and , wherein is the column number where the kth row similarity maximum value in the similarity matrix M is located.

[0016] Further, the similarity of the neighbor entities of the entity pair is calculated according to the following formula:

[0017] wherein represents the similarity of and , and and respectively represent a neighbor entity of the entity and , and and respectively represent a neighbor entity set of the entity and , and represents​ the i-th entity in the middle, denotes the j-th entity in the middle, denotes the enhanced fusion feature vector of the i-th entity, denotes the enhanced fusion feature vector of the i-th entity, denotes the enhanced fusion feature vector of the i-th entity, denotes the enhanced fusion feature vector of the i-th entity.

[0018] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the multi-modal entity alignment pseudo seed generation method.

[0019] The application further provides an electronic device, which comprises a processor and a memory, wherein the processor and the memory are connected to each other, the memory is used for storing a computer program, the computer program comprises computer readable instructions, and the processor is configured to call the computer readable instructions to execute the multi-modal entity alignment pseudo seed generation method.

[0020] The application further provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the steps of the multi-modal entity alignment pseudo seed generation method.

[0021] The application provides the technical scheme and has the beneficial effects: The multi-modal entity alignment unsupervised pseudo seed generation method provided by the application gradually expands the pseudo seed scale through the three-stage pseudo seed generation and optimization process of multi-modal fusion, clustering sampling, global sampling and verification, and neighborhood expansion and verification, effectively reduces the introduction of error pseudo seeds, and ensures the overall precision of the pseudo seed set. The clustering and global sampling strategy is introduced in the pseudo seed selection, which avoids the concentration of pseudo seeds in a small number of high-frequency entities or local areas, realizes the balanced coverage of pseudo seeds in the knowledge graph, improves the coverage rate while ensuring the quality of pseudo seeds, and realizes the dynamic balance of quantity and precision. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of a multi-modal entity alignment pseudo seed generation method according to an embodiment of the application; Figure 2 is a block diagram of an electronic device according to an example embodiment of the application. DETAILED DESCRIPTION

[0023] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described below with reference to the drawings.

[0024] The flowchart of the multi-modal entity alignment pseudo-seed generation method of the embodiments of the present application is as shown in Figure 1 , and specifically includes the following steps: S1, obtaining two multi-modal knowledge graphs to be matched and , wherein and represent entities, and represent the relationship of the entity, and represent the attribute of the entity, and represent the visual information of the entity.

[0025] S2, respectively encoding the multi-modal information of the entities of the two multi-modal knowledge graphs, and obtaining the multi-modal feature vectors of the entities of the two multi-modal knowledge graphs.

[0026] Specifically, the visual information of the entity is encoded by using a ResNet model, and the image data is processed by the ResNet network to generate a 2048-dimensional feature vector, represented as:

[0027] wherein, represents the feature vector of the visual information of the i-th entity, i represents the ResNet model, represents the i-th entity visual information. i The relationship of the entity and the attribute of the entity are encoded by using a pre-trained BERT to obtain a 768-dimensional feature vector, represented as:

[0028]

[0029] wherein, represents the feature vector of the i-th entity attribute, represents the feature vector of the i-th entity relationship, AVE represents the operation of averaging, and BERT represents the pre-trained BERT encoding, represents all attributes of the i-th entity, i represents all relationships of the i-th entity. S3, respectively fusing the multi-modal feature vectors of the entities of the two multi-modal knowledge graphs, and obtaining the respective fusion feature vectors of the entities of the two multi-modal knowledge graphs.

[0030] ​​​

[0031] The multimodal feature vector is fused in a splicing manner and is represented as follows:

[0032] wherein, represents the fusion feature vector of the i-th entity, represents the feature vector of the visual information of the i-th entity, represents the feature vector of the attribute of the i-th entity, represents the feature vector of the relationship of the i-th entity, represents vector splicing.

[0033] S4, cluster sampling: based on the fusion feature vectors of the entities in the two multimodal knowledge graphs, the entities in the two multimodal knowledge graphs are clustered, m entity pairs are selected from each cluster, and a pseudo seed set in the first stage is obtained .

[0034] The present application uses the K-means clustering method to divide the entities in the two multimodal knowledge graphs into K clusters, denoted as , calculates the similarity of the entity pairs in each cluster, and sorts all the entity pairs in each cluster according to the similarity value from high to low, and selects entity pairs from each cluster, wherein the number of entity pairs selected from each cluster satisfies the following formula:

[0035] wherein, represents the number of entity pairs selected from the j-th cluster, and n represents the total number of required pseudo seeds, represents the number of entities in the j-th cluster, and respectively represent the number of entities in and .

[0036] S5, global sampling: the multimodal feature vectors of the entities of the two multimodal knowledge graphs are enhanced respectively, and the enhanced multimodal feature vectors of the entities of the two multimodal knowledge graphs are obtained. For the visual feature of the entity , first, mapping processing is performed through an independent linear layer, and then neighborhood information is aggregated by using a graph attention network GAT to enhance the expression of the visual feature. For the attribute and relationship features of the entity, independent linear layers are respectively used to convert the attribute features and the relationship features, and finally new entity attribute features and relationship features are obtained, and the specific implementation manner is as follows:

[0037] wherein, , and respectively represent the enhanced features of , and , , and respectively represent the feature vector of the i-th entity visual information, the feature vector of the attribute and the feature vector of the relationship.

[0038] The ICL loss is used when training the above feature enhancement process:

[0039] wherein loss represents the total loss, represents the ICL loss, respectively represent the enhanced feature vectors of the entity visual information in the two knowledge graphs, respectively represent the enhanced feature vectors of the entity attributes in the two knowledge graphs, respectively represent the enhanced feature vectors of the entity relationships in the two knowledge graphs.

[0040] S6, the enhanced multi-modal feature vectors of the two multi-modal knowledge graph entities are fused respectively by means of vector splicing, to obtain the respective enhanced fusion feature vectors of the two multi-modal knowledge graph entities.

[0041] S7, based on the enhanced fusion feature vectors of the two multi-modal knowledge graph entities, the similarity between all entity pairs in the two multi-modal knowledge graphs is calculated, and according to the similarity score, the entity pairs not in the pseudo seed set are screened, and the top n entity pairs with the highest similarity are selected as the new pseudo seed set .

[0042] S8, multi-modal information error correction: the similarity matrix of the entities in the pseudo seed set is calculated, and the pseudo seed set is corrected according to the similarity matrix, to obtain the corrected pseudo seed set .

[0043] Specifically: The feature matrices of the entity pairs and in the pseudo seed set are and , wherein is the entity in , is the entity in . According to and , the similarity matrix of the entities in the pseudo seed set is calculated Diagonal elements in the similarity matrix If the value is not the maximum value, then from the pseudo-seed set Remove fake seeds and ,in, is the column number of the k-th row in the similarity matrix M containing the maximum similarity value.

[0044] The similarity matrix M is as follows:

[0045] in, Representing entities and entity similarity, Representing entities and entity similarity, Representing entities and entity similarity, Representing entities and entity The similarity.

[0046] S9. Neighborhood Expansion: This increases the number of seeds by expanding the pseudo-aligned seed pairs of adjacent entities, while also supplementing the knowledge graph around scattered entities, thereby enhancing the attention paid to the originally scattered entity parts during gradient updates. Calculate the pseudo-seed set. The similarity between neighboring entities of an entity pair is calculated using the following formula:

[0047] in, express and similarity, and Representing entities respectively and A neighboring entity, and Representing entities respectively and The set of neighboring entities, express The i-th entity in the middle, express The j-th entity in the middle, express The fused feature vector, express Enhanced fusion feature vector, express The fused feature vector, express the enhanced fusion feature vector.

[0048] Screening the similarity of neighbor entities is higher than the set threshold η, and the entity pair is not in the set The pseudo seed set is obtained by further correcting the entity pair in the pseudo seed set The pseudo seed set is obtained by further correcting the entity pair in the pseudo seed set The final pseudo seed set is obtained by using the information correction method in S8.

[0049] In order to verify the effectiveness of the method of the application, the method of generating pseudo seeds (PSQE) of the application and the existing different entity alignment methods are combined, and three public data sets are selected: DBP15K (Chinese and English alignment), DBP15K (Japanese and English alignment) and DBP15K (French and English alignment). The existing different methods include: RDGCN (relation-aware double graph convolutional network), RPR-RHGT (from the paper: Entity Alignment with Reliable Path Reasoning and Relation-Aware Heterogeneous Graph Transformer), DESAlign (from the paper: Towards Semantic Consistency: Dirichlet Energy Driven Robust Multi-Modal Entity Alignment), EVA (from the paper: Visual pivoting for (unsupervised) entity alignment), EVA+PSQE, MCLEA (from the paper Multi-modal Contrastive Representation Learning for Entity Alignment), MCLEA+PSQE, MEAformer (from the paper MEAformer: Multi-modal entity alignment transformer for meta modality hybrid), MEAformer+PSQE, PCMEA (from the paper Pseudo-Label Calibration Semi-supervised Multi-Modal Entity Alignment), PCMEA+PSQE. The experimental results of the method of the application combined with different entity alignment methods on three public data sets are compared in reference table 1. The indexes used include MRR (Mean Reciprocal Ranking) and HITS@n.

[0050]

[0051] where S is the set of triples, |S| is the number of triples, denotes the link prediction rank of the ith triple. The larger the MRR indicator is, the better.

[0052]

[0053] where II denotes the indicator function, which is 1 if the condition is true, and 0 otherwise. The larger the HITS@n indicator is, the better.

[0054] Table 1

[0055] The multi-modal entity alignment unsupervised pseudo seed generation method provided by the application can significantly improve the quality and stability of pseudo seed generation through a multi-stage iterative optimization strategy. Relying on the precision control and distribution control mechanism, the application can effectively eliminate false pseudo seeds and avoid the concentration of pseudo seeds in local areas, thereby ensuring the balance of the generated results in accuracy and coverage. At the same time, combined with the multi-modal feature fusion and expression enhancement technology, the discriminability and robustness of entity vector representation are improved, so that good alignment performance can be maintained even in the presence of noise and information imbalance. Overall, the precision and coverage of the pseudo seeds generated by the method under unsupervised conditions are better than those of existing methods, which provides a solid support for cross-modal alignment and fusion of multi-modal knowledge graphs.

[0056] In an exemplary embodiment, a computer readable storage medium is included, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the multi-modal entity alignment pseudo seed generation method described above.

[0057] Please refer to Figure 2 In an exemplary embodiment, an electronic device is also included, which includes at least one processor, at least one memory, and at least one communication bus.

[0058] The memory has a computer program stored thereon, and the computer program includes computer readable instructions. The processor invokes the computer readable instructions stored in the memory through the communication bus to execute the multi-modal entity alignment pseudo seed generation method described above.

[0059] In an exemplary embodiment, a computer program product is provided, which includes a computer program / instruction. The computer program / instruction is executed by a processor to implement the steps of the multi-modal entity alignment pseudo seed generation method described above.

[0060] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-modal entity alignment pseudo seed generation method, characterized in that, The method comprises the following steps: S1, acquiring two multi-modal knowledge graphs to be matched and ; S2, respectively, to and Encode the multi-modal information of the entity, obtain and Multi-modal feature vector of the entity; S3, respectively to and fuse the multi-modal feature vectors of the entities to obtain and the respective fused feature vectors of the entities; S4, based on and the respective fusion feature vectors of the entities, clustering the entities in the entity set, selecting m entity pairs from each cluster, and obtaining a first-stage pseudo seed set and ;​ S5、respectively to and enhancing the multi-modal feature vector of the entity to obtain and the enhanced multi-modal feature vector of the entity; S6、respectively to and fuse the enhanced multi-modal feature vectors of the entities to obtain and the respective enhanced fusion feature vectors of the entities; S7、based on and enhanced fusion feature vector of the entity, calculating and the similarity between all entity pairs in the set , selecting the top n entity pairs with the highest similarity as the new pseudo seed set ; S8、calculating the set a similarity matrix of the entities, and correcting the set according to the similarity matrix to obtain a corrected pseudo-seed set ; S9、computing the set the similarity of the neighbor entities of the entity pair in the set the entity pair in the set , and further refining the entity pair in the set , to obtain the final pseudo seed set.

2. The method of claim 1, wherein, Multi-modal knowledge graph representation is wherein E represents an entity, R represents a relationship of the entity, A represents an attribute of the entity, and V represents visual information of the entity. The visual information of the entity is encoded by using a ResNet model, and is represented as: in, The feature vector representing the visual information of the i-th entity. Represents the ResNet model. Indicates the first i Visual information of each entity; The relationship of the entity and the attribute of the entity are encoded by using a pre-trained BERT, and are represented as: wherein, denotes a feature vector of the i-th entity attribute, denotes a feature vector of the i-th entity relation, AVE denotes an averaging operation, and BERT denotes a pre-trained BERT encoding, denotes all attributes of the i-th entity, denotes all relations of the i-th entity.

3. The method of claim 1, wherein, The multi-modal feature vectors are fused by using a splicing manner, and are represented as: wherein, represents the final feature vector of the i-th entity, represents the feature vector of the i-th entity visual information, represents the feature vector of the i-th entity attribute, represents the feature vector of the i-th entity relationship, represents vector concatenation.

4. The method of claim 1, wherein, S4 is specifically: K-means clustering method is used to divide the entities in the two multi-modal knowledge graphs into K clusters, the similarity of the entity pairs in each cluster is calculated, all entity pairs in each cluster are sorted according to the similarity value from high to low, and entity pairs are selected from each cluster, wherein the number of entity pairs selected from each cluster satisfies the following formula: wherein, represents the number of entity pairs selected from the jth cluster, n represents the total number of required pseudo seeds, represents the number of entities in the jth cluster, and respectively represent the number of entities in and .

5. The method of claim 1, wherein, S5 is specifically: After the feature vector of the entity visual information is mapped through an independent linear layer, the neighborhood information is aggregated by using a graph attention network GAT to obtain an enhanced feature vector of the entity visual information; The feature vector of the entity attribute and the feature vector of the entity relationship are respectively converted by using independent linear layers to obtain an enhanced feature vector of the entity attribute and an enhanced feature vector of the entity relationship; The ICL loss is used when training the above feature enhancement process: wherein loss denotes the total loss, denotes the ICL loss, denote the enhanced feature vectors of the entity visual information in the two knowledge graphs, respectively, denote the enhanced feature vectors of the entity attributes in the two knowledge graphs, respectively, denote the enhanced feature vectors of the entity relationships in the two knowledge graphs, respectively.

6. The method of claim 1, wherein, S8 is specifically: Pseudo-seed set Middle entity pair And The feature matrix of And , according to And Calculate the pseudo-seed set The similarity matrix of the middle entity The diagonal element in the similarity matrix If the value of Remove the pseudo-seed And , wherein The column number where the kth row similarity maximum value in the similarity matrix M is located.

7. The method of claim 1, wherein, The similarity of the neighbor entities of the entity pair is calculated according to the following formula: wherein, denotes and a similarity of and denote a neighbor entity of entities and respectively, and denote a set of neighbor entities of entities and respectively, denotes the i-th entity in denotes the j-th entity in denotes the i-th entity in denotes the j-th entity in denotes a fused feature vector of denotes an enhanced fused feature vector of denotes a fused feature vector of denotes an enhanced fused feature vector of denotes a fused feature vector of denotes an enhanced fused feature vector of denotes a fused feature vector of denotes an enhanced fused feature vector of 8. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to realize the method of any one of claims 1-7.

9. An electronic device, comprising: The computer program is executed by the processor to realize the method of any one of claims 1-7.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps of the method of any one of claims 1-7.

Images