A multi-modal entity alignment method and system based on information bottleneck

Abstract

The application relates to a multi-modal entity alignment method and system based on an information bottleneck. The method acquires data of a multi-modal knowledge graph; a multi-modal variational encoder is used to extract features of the data of the multi-modal knowledge graph to obtain specific modal features; a multi-modal information regularization term is used to perform regularization processing on the specific modal features based on an information bottleneck, minimizes irrelevant information for alignment, maximizes relevant information for alignment, and obtains mixed modal features through a weighted modal fusion strategy; and the processing result of the multi-modal information regularization term is used to complete a multi-modal entity alignment task. The application can maximize the retention of relevant information for alignment and minimize the influence of irrelevant information for alignment, thereby improving the accuracy of multi-modal knowledge graph entity alignment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge graph in natural language processing, and particularly relates to a multi-modal entity alignment method and system based on information bottleneck. BACKGROUND

[0002] Traditional knowledge graph (KG) is constructed based on text or structured data, which contains connections between entities, relationships and attributes, and mainly focuses on semantic association and structured knowledge representation. Multi-modal knowledge graph (MMKG) refers to a knowledge graph that combines multiple data modalities (such as text, image, video, speech, etc.), which can integrate information from different modalities and provide more comprehensive knowledge representation.

[0003] In the field of knowledge graph, entity alignment (EA) aims to find entities in different knowledge graphs that refer to the same real-world object, so as to enrich the current knowledge graph from another complementary knowledge graph, thereby improving the quality and coverage of the knowledge graph and promoting knowledge fusion. Similar to traditional knowledge graphs, multi-modal knowledge graphs also have the problem of incompleteness and limited coverage. Multi-modal entity alignment (MMEA) aims to identify the same entities between two multi-modal knowledge graphs, so as to complement the same entity information of the current multi-modal knowledge graph with the attribute information of another multi-modal knowledge graph.

[0004] Existing multi-modal entity alignment works focus on the integration of multi-modal information in multi-modal knowledge graphs, which can be roughly divided into two categories: the first category is to form a joint embedding representation of entities by fusing multi-modal embedding representations, such as using direct concatenation, manual weight, and dynamic weight to concatenate embedding representations of different modalities. The second category is to transfer multi-modal features by constructing inter-modal constraints, such as using entity triple constraint information to utilize image information.

[0005] Existing research methods first learn multi-modal features at a coarse granularity, and then fuse the multi-modal features, which ignores the correlation between modal features of two multi-modal knowledge graphs and unintentionally introduces irrelevant information to entity representation, which may mask the features related to alignment and mislead the accuracy of multi-modal entity alignment. SUMMARY

[0006] The application aims to provide a new method for multi-modal entity alignment based on information bottleneck, which can maximize the retention of alignment-related information and minimize the influence of alignment-irrelevant information, thereby improving the accuracy of multi-modal knowledge graph entity alignment.

[0007] The technical solutions adopted by the application are as follows:

[0008] A multi-modal entity alignment method based on information bottleneck comprises the following steps:

[0009] Obtain data of a multi-modal knowledge graph;

[0010] Extract features of the data of the multi-modal knowledge graph by using a multi-modal variational encoder to obtain specific modal features;

[0011] Regularize the specific modal features based on information bottleneck by using a multi-modal information regularization term, minimize alignment-irrelevant information, maximize alignment-related information, and obtain hybrid modal features through a weighted modal fusion strategy;

[0012] Complete a multi-modal entity alignment task by using the processing result of the multi-modal information regularization term.

[0013] Further, the multi-modal variational encoder comprises a variational graph structure encoder, a variational visual encoder, a variational attribute encoder, and a variational relationship encoder; and the specific modal features comprise features of four kinds of modalities, i.e., a graph structure modality, a visual modality, an attribute modality, and a relationship modality.

[0014] Further, the variational graph structure encoder uses a graph attention network GAT to calculate graph structure latent representations by calculating mean and variance; the variational visual encoder uses a pre-trained VGG-16 model to extract features for the visual modality, and for entities without images, the average value of image information of neighbors of the entity is taken as the image representation of the current entity; the variational attribute encoder and the variational relationship encoder use a bag-of-words model to calculate features for the attribute modality and the relationship modality.

[0015] Further, the multi-modal information regularization term includes a specific modal information bottleneck regularization term and a mixed modal information comparison regularization term; the specific modal information bottleneck regularization term plays a role of minimizing alignment irrelevant information and maximizing alignment relevant information; the mixed modal information comparison regularization term adopts a weighted modal fusion strategy, the weighted modal fusion strategy calculates a mixed modal feature variable complementary to each modal based on the complementary effect between different modalities, and then uses a contrast learning method to fuse equivalent entity data in the two multi-modal knowledge graphs, and a weighted attention mechanism is used in the fusion process to enable different weights to be represented for the specific modal features of each entity; the multi-modal entity alignment task is completed after the integration of the two regularization terms of the specific modal information bottleneck regularization term and the mixed modal information comparison regularization term.

[0016] Further, the specific modal information bottleneck regularization term performs information bottleneck processing on the feature variables of each modality to obtain an information bottleneck regularization loss function L m :

[0017]

[0018] wherein represents original information from two multi-modal knowledge graphs G1 and G2, Y m represents supervised information of entity alignment, represents information obtained from different modalities of G1 and G2, β m represents the control of the degree of effective information retention in different modalities, and I represents mutual information calculation; for a plurality of modal information, an information bottleneck regularization term is defined

[0019]

[0020] wherein g, v, a, and r represent graph structure modality g, visual modality v, attribute modality a, and relationship modality r, respectively.

[0021] Further, the mixed modal information comparison regularization term adopts a weighted modal fusion strategy, utilizes the complementary effect between different modalities to generate a mixed modal feature variable, and then adopts a contrast learning to maximize the mutual information between equivalent entity pairs of the two multi-modal knowledge graphs:

[0022]

[0023] wherein a m is a weight corresponding to each modality, and Z m is a final information representation of the modality obtained by reparameterization operation on each modality m∈{g,v,a,r}; the mixed modal information comparison regularization term is:

[0024]

[0025] wherein, denotes a mixed modal representation of entities in two multi-modal knowledge graphs.

[0026] A multi-modal entity alignment system based on information bottleneck comprises:

[0027] A data acquisition module is configured to acquire data of multi-modal knowledge graphs.

[0028] A multi-modal variational encoder module is configured to extract features of the data of the multi-modal knowledge graphs to obtain specific modal features.

[0029] A multi-modal information regularization term module is configured to utilize a multi-modal information regularization term to perform information bottleneck-based regularization processing on the specific modal features, minimize alignment irrelevant information, maximize alignment relevant information, and obtain mixed modal features through a weighted modal fusion strategy; and complete a multi-modal entity alignment task by using a processing result of the multi-modal information regularization term.

[0030] The method provided by the present application has the following advantages in processing multi-modal knowledge graph entity alignment:

[0031] 1. The present application is very effective in processing multi-modal knowledge graph entity alignment data sets in benchmark tests, and is superior to previous advanced methods.

[0032] 2. The present application makes full use of the hierarchical structure of the multi-modal knowledge graph itself, and provides four multi-modal variational encoders for different modalities to generate specific modal representations using probability distributions, which helps to maximize the retention of entity alignment related information and minimize entity alignment redundant information in the subsequent multi-modal knowledge graph.

[0033] 3. The present application introduces a novel multi-modal entity alignment framework based on variational information bottleneck for multi-modal entity alignment, and proposes two regularization term methods to improve specific modal representations and mixed modal representations. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a step flowchart of the multi-modal entity alignment method based on information bottleneck of the present application. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below through specific examples and drawings.

[0036] The present application aims to solve one of the technical problems existing in the prior art. To this end, the present application discloses a multi-modal entity alignment method based on information bottleneck. In view of the problem that the existing multi-modal entity alignment method cannot effectively distinguish between alignment-related information and alignment-irrelevant information, the information bottleneck (IB) theory is introduced to compress the representation and forget some information while retaining information conducive to prediction. The information bottleneck theory believes that the network is like squeezing information out of a bottleneck, removing noise input data containing irrelevant details, and only retaining features most relevant to general concepts. For multi-modal entity alignment, the information bottleneck between multi-modal knowledge graphs is adjusted separately for each modality, thereby ensuring that the misleading information irrelevant to alignment in the modality can be correctly compressed, while the information relevant to alignment is strengthened. At this point, the modality representation with stronger prediction ability can be generated by the information bottleneck constraint of a specific modality, thereby facilitating multi-modal information integration.

[0037] The present application provides a multi-modal entity alignment method based on information bottleneck, and the main steps of the method are as shown in Figure 1 The present application provides a multi-modal entity alignment method based on information bottleneck, and the main steps of the method are as shown in

[0038] The multimodal variational encoder mentioned in the method, wherein the variational graph structure encoder is mainly used to fully utilize the hierarchy of the multimodal knowledge graph, calculate the importance of entity neighbors for the entity alignment task, and mainly use a graph attention network (GAT, Graph Attention Networks) to calculate the graph structure latent representation by first calculating the mean and variance. For the visual modality, an independent fully connected layer is designed for hidden representation learning for the attribute modality and the relationship modality, which is used for learning the subtle differences of the modal features. Among them, the pre-trained VGG-16 model is used to extract features for the visual modality, and the average value of the image information of the neighbors of the entity is used as the image representation of the current entity for the entity without an image. The bag-of-words model is used to calculate the features for the attribute modality and the relationship modality. The multilayer perceptron (MLP, Multilayer Perceptron) is used to determine the sampling and reparameterization of the multivariate Gaussian distribution to obtain the representation of a specific modality. The mean and standard deviation are used to sample to obtain the deterministic representation of each modality (i.e. Figure 1 the specific modality feature in the formula), and finally the deterministic representations of different modalities (specific modality features) are obtained through the weighted modality fusion strategy in the multimodal information regularization term to obtain the hybrid modal feature.

[0039] The multimodal information regularization term mentioned in the method is mainly used to enhance the resolution capability of the multimodal feature, and two regularization term methods, i.e., the specific modality information bottleneck regularization term and the hybrid modal information comparison regularization term, are used. The former uses the information bottleneck regularization term defined by the information bottleneck on each information modality to constrain the specific modality feature variables of the equivalent entity pairs from the two multimodal knowledge graphs, respectively, weakens the entity information of the own multimodal through the penalty term, and completes the multimodal entity alignment task through the enhancement of the alignment term. The hybrid modal information comparison regularization term is a weighted modality fusion strategy designed by the application, which is based on the complementary effect between different modalities. First, the hybrid modal feature variables complementary to each other are calculated, and then the equivalent entity data in the two multimodal knowledge graphs are fused by using the contrast learning method. The weighted attention mechanism is used in the fusion process to make the specific modality features of each entity have different weight representations. The multimodal entity alignment task is completed through the integration of the specific modality information bottleneck regularization term and the hybrid modal information comparison regularization term.

[0040] The technical solutions of the application are described as follows:

[0041] 1. Obtain the data of two multimodal knowledge graphs, which is defined as wherein E, R, A and V are the sets of entities, relationships, attributes and visual images, respectively, E, R, A and V are the sets of entities, relationships, attributes and visual images, respectively, where each entity e e e can be associated with attributes and images.

[0042] 2. A multi-modal entity alignment method based on information bottleneck, the main steps of which include: (1) a multi-modal variational encoder, which first extracts features of multi-modal knowledge graphs according to their inherent multi-modal characteristics using different feature extractors to obtain four kinds of modal feature representations (graph structure modal (g), visual modal (v), attribute modal (a) and relationship modal (r)); (2) a multi-modal information regularization term, which performs information bottleneck regularization processing on the four kinds of modal feature representations in (1) to minimize alignment irrelevant information and maximize alignment relevant information.

[0043] 3. The multi-modal variational encoder mentioned in 2, wherein the variational graph structure encoder uses a graph structure latent representation (Z g ) to represent the latent graph structure information in the data, and obtains a graph structure information mean (μ μ ) and a graph structure information variance (σ σ ) using independent graph attention network encoders (GAT g ) and GAT g ) respectively to calculate the latent representation:

[0044] μ g =GAT μ (X g ,A),

[0045] σ g =GAT σ (X g ,A),

[0046]

[0047] where X g is a feature matrix of initial nodes of the graph, A is an adjacency matrix of a specific multi-modal knowledge graph, and GAT transmits graph knowledge structure through an information co-weight parameter; represents a multi-dimensional normal distribution with mean μ g and variance diagonal matrix .

[0048] 4. The multi-modal variational encoder mentioned in 2, wherein for the visual modal (v), the attribute modal (a) and the relationship modal (r), an independent fully connected layer is designed to learn the intermediate hidden representation:

[0049]

[0050] where Φ is a ReLU function, a real matrix, X l represents the input features of the corresponding modal, and FCl denotes the fully connected layer for the corresponding modality. The latent information representation (Z l ) of visual, relation, attribute, etc. modality is obtained using a multi-layer perceptron (MLP).

[0051]

[0052]

[0053]

[0054] where, denotes the mean μ l and the variance is a diagonal matrix .

[0055] Finally, the final information representation Z m of each modality m e {g, v, a, r} is obtained by reparameterization operation.

[0056]

[0057] where ε is a standard Gaussian random variable, and denotes the element-wise product. Thus, the specific representation of each modality entity is obtained, which is used for further operation.

[0058] 5、The regularization term mentioned in 2, two kinds of regularization terms (specific modality information bottleneck regularization term and mixed modality information comparison regularization term) are proposed to constrain the specific modality features and mixed modality features. For the former, information bottleneck processing is performed on the feature variables of each modality, and an information bottleneck regularization loss function L m is designed:

[0059]

[0060] where denotes the original information from graph G1 and graph G2, Y m denotes the supervision information of entity alignment, denotes the information representation of different modalities from graph G1 and graph G2 obtained in 4, β m denotes the control of the degree of effective information preservation in different modalities, and I denotes mutual information calculation. The regularization term can minimize the irrelevant information of alignment and maximize the relevant information of alignment. For multiple modalities, the entity overall information bottleneck regularization term L

[0061]

[0062] Through the information bottleneck regularization term, the model is equivalent to squeeze the information of different modalities out of a bottleneck in general, remove the information containing irrelevant noise content, and only keep the modal specific features most relevant to each modality and entity alignment task.

[0063] 6、In 5, a hybrid modal information contrast regularization term is proposed, for each entity representation, a hybrid modal feature variable is generated by using the complementary effect between different modalities through a weighted modal fusion strategy, and then contrast learning is used to maximize the mutual information between the equivalent entity pairs of the two multi-modal knowledge graphs:

[0064]

[0065] Where a m is the weight corresponding to each modality, and the method for obtaining a m is as follows:

[0066]

[0067]

[0068] Where are all learnable parameters, and the hybrid modal information contrast regularization term designed by the application is

[0069]

[0070] Where, represents the hybrid modal embedding representation of the entity in the graph G1, represents the hybrid modal representation of the entity in the graph G2.

[0071] In this way, the similarity of the entity pair is measured from the two multi-modal knowledge graphs, and the representation is enhanced by using the complementary effect between different modalities.

[0072] 7、Finally, through the overall target loss function

[0073]

[0074] Get the entity embedding representation processed by minimizing the alignment irrelevant information and maximizing the alignment relevant information, to complete the multi-modal entity alignment task.

[0075] Examples of the application:

[0076] Example 1, a multi-modal entity alignment method based on information bottleneck

[0077] This example uses the FB15K-DB15K and FB15K-YAGO15K public datasets, which are considered as the most representative datasets in the multi-modal entity alignment task. The information of the number of entities, the number of relations, and the number of triples in the datasets is shown in Table 1, where Rel. represents the relation, Rel tr. represents the relation triple, Attr. represents the attribute, and Attr tr. represents the attribute triple.

[0078] Table 1: Statistics of two multi-modal knowledge graph datasets

[0079]

[0080] The two datasets are divided into training and test subsets in the ratio of 2:8, 5:5, and 8:2, respectively.

[0081] The evaluation indicators used include Hits@n and Mean Reciprocal Rank (MRR). Hits@n represents the proportion of correct entities ranked in the top n according to similarity calculation. MRR represents the average reciprocal rank. The higher the value of Hits@n and MRR, the stronger the performance of the method.

[0082] The specific results obtained are shown in Tables 2 and 3. The best results are in bold, and the suboptimal results are underlined. * indicates the results copied from the official source code.

[0083] Table 2. Experimental results on the FB15K-DB15K dataset

[0084]

[0085] Table 3. Experimental results on the FB15K-YAGO15K dataset

[0086]

[0087] Table 2 and Table 3 show the overall average results of FB15K-DB15K and FB15K-YAGO15K benchmark datasets respectively, where the best results are in bold and the suboptimal are underlined. From the tables, the following points can be seen: 1) The multi-modal entity alignment method model proposed by the present application is superior to all baselines in three measurement indicators (hits@1, hits@10, MRR) on two multi-modal knowledge graph datasets under three different dataset divisions (20%, 50%, 80%), which proves the robustness and generalization ability of the model. 2) Compared with the multi-modal entity alignment baseline, the model of the present application has a significant improvement in Hits@1, specifically, it achieves an improvement from 7.8% to 15.9% (average 12.2%) and from 6.3% to 12.6% (average 9.13%) on FB15K-DB15K and FB15K-YAGO15K respectively compared with the most advanced baseline. This accurate alignment capability is attributed to the information bottleneck regularizer of the present application, which minimizes irrelevant information for alignment and maximizes relevant information for alignment. 3) The performance of the multi-modal entity alignment baseline model is generally superior to the traditional entity alignment method (EA), highlighting the importance of multi-modal information. It is worth noting that the Hits@1 performance of the model of the present application using 20% seeds exceeds 80% of the traditional entity alignment method baseline model. This shows that the information contrast regularizer of the present application can utilize the complementary information of various modalities through mixed modal representation. 4) It is noted that the entity alignment method model of the present application shows more significant performance improvement at a ratio of 20% compared with a higher ratio, which proves its adaptability in low-resource scenarios. In addition, it is also noted that there is a continuous and significant improvement in Hits@1, which shows that the model of the present application can provide more accurate predictions when facing limited candidate objects. In summary, the above experimental results prove the effectiveness of the multi-modal entity alignment method proposed by the present application.

[0088] The multi-modal entity alignment realized by the present application can identify the same entities between two multi-modal knowledge graphs, so as to complement the same entity information of the current multi-modal knowledge graph with the attribute conditions of another multi-modal knowledge graph.

[0089] For example, in the field of cross-domain information retrieval, users want to query information in knowledge graphs across multiple domains, such as querying entity relationships in medical and chemical fields at the same time, and the present application can help to correspond entities in different fields to provide more comprehensive information retrieval results.

[0090] For example, in the field of intelligent question answering, an intelligent question answering system is constructed based on a knowledge graph, and users can obtain relevant information through natural language questioning, and the present application can ensure that the system can correctly understand the entities involved in the user's questioning and give accurate answers.

[0091] For example, in the field of personalized recommendation, a personalized recommendation system is built based on user's interest and behavior data to recommend relevant entities and information. The application can help the system better understand the user's interest and preference, and provide more accurate and personalized recommendation results.

[0092] For example, in the field of enterprise knowledge management, there is a large amount of knowledge distributed in different systems and departments within an enterprise, which needs to be integrated and managed. The application can help to correspond entities in different systems within an enterprise, build a unified knowledge management system, and improve knowledge utilization efficiency and decision support capability.

[0093] Another embodiment of the application provides a multi-modal entity alignment system based on information bottleneck, which comprises:

[0094] A data acquisition module is configured to acquire data of a multi-modal knowledge graph.

[0095] A multi-modal variational encoder module is configured to extract features of the data of the multi-modal knowledge graph to obtain specific modal features.

[0096] A multi-modal information regularization term module is configured to perform regularization processing on the specific modal features based on the information bottleneck using a multi-modal information regularization term, minimize irrelevant information for alignment, maximize relevant information for alignment, and obtain hybrid modal features through a weighted modal fusion strategy.

[0097] The processing result of the multi-modal information regularization term is used to complete the multi-modal entity alignment task.

[0098] The specific implementation process of each module is described above in the description of the method of the application.

[0099] Another embodiment of the application provides a computer device (computer, server, smart phone, etc.), which comprises a memory and a processor. The memory stores a computer program configured to be executed by the processor. The computer program comprises instructions for executing each step of the method of the application.

[0100] Another embodiment of the application provides a computer readable storage medium (such as ROM / RAM, magnetic disk, optical disk), which stores a computer program. When the computer program is executed by a computer, each step of the method of the application is implemented.

[0101] While the foregoing detailed description has set forth various specific embodiments of the application, it is to be understood that the disclosure is not to be limited to the details of construction or the arrangement of parts as set forth in the foregoing description. The foregoing detailed description is to be considered in all respects only as illustrative and not as restrictive.

Claims

1. A method for multi-modal entity alignment based on information bottleneck, characterized in that, The method comprises the following steps: obtaining data of a multi-modal knowledge graph; extracting features of the data of the multi-modal knowledge graph by using a multi-modal variational encoder to obtain specific modal features; regularizing the specific modal features based on an information bottleneck by using a multi-modal information regularization term, and obtaining mixed modal features by using a weighted modal fusion strategy; completing a multi-modal entity alignment task by using a processing result of the multi-modal information regularization term; the multi-modal information regularization term comprises a specific modal information bottleneck regularization term and a mixed modal information comparison regularization term; The specific modal information bottleneck regularization term performs information bottleneck processing on the feature variables of each modality to obtain an information bottleneck regularization loss function L m : wherein represents the original information from two multi-modal knowledge graphs G1 and G2, Y m represents the supervision information of entity alignment, represents the obtained information representation from different modalities of G1 and G2, β m represents the control of the degree of effective information retention in different modalities, I represents mutual information calculation; Defining information bottleneck regularization terms for multiple modalities of information wherein g, v, a, and r represent a graph structure modality g, a visual modality v, an attribute modality a, and a relationship modality r; the mixed modal information comparison regularization term uses a weighted modal fusion strategy to generate mixed modal feature variables by using a complementary effect between different modalities, and then uses contrastive learning to maximize mutual information between equivalent entity pairs of two multi-modal knowledge graphs: where a m is the weight corresponding to each modality, Z m is the final information representation of the modality obtained by performing a reparameterization operation on each modality m∈{g,v,a,r}; the mixed modality information comparison regularization term is: wherein, denotes a mixed modality representation of an entity in the two multi-modal knowledge graphs.

2. The method of claim 1, wherein, the multi-modal variational encoder comprises a variational graph structure encoder, a variational visual encoder, a variational attribute encoder, and a variational relationship encoder; and the specific modal features comprise features of four modalities, namely, a graph structure modality, a visual modality, an attribute modality, and a relationship modality.

3. The method of claim 2, wherein, the variational graph structure encoder uses a graph attention network GAT to calculate a graph structure latent representation by calculating a mean value and a variance; the variational visual encoder uses a pre-trained VGG-16 model to extract features for the visual modality, and averages image information of neighbors of an entity to obtain an image representation of the entity for an entity without an image; and the variational attribute encoder and the variational relationship encoder use a bag-of-words model to calculate features for the attribute modality and the relationship modality.

4. The method of claim 1, wherein, obtaining a weight a corresponding to each modality m The method is as follows: where q f , W m , b m are learnable parameters.

5. An information bottleneck based multi-modal entity alignment system employing the method of any one of claims 1-4, wherein, The method comprises: a data acquisition module configured to acquire data of a multi-modal knowledge graph; a multi-modal variational encoder module configured to extract features of the data of the multi-modal knowledge graph to obtain specific modal features; a multi-modal information regularization term module configured to regularize the specific modal features based on an information bottleneck by using a multi-modal information regularization term, minimize alignment irrelevant information, maximize alignment relevant information, and obtain mixed modal features by using a weighted modal fusion strategy; and complete a multi-modal entity alignment task by using a processing result of the multi-modal information regularization term.

6. A computer device, comprising: The computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a computer to implement the method in any one of claims 1-4.

Images