Entity alignment method and device based on multi-modal information and medium

Abstract

The invention discloses an entity alignment method and device based on multi-modal information and a medium, and relates to the field of entity alignment, and the method comprises the steps: obtaining the multi-modal information of each entity; according to the multi-modal information of each entity, a cross-modal consistency integration network is adopted to obtain an embedded representation of the entity; the cross-modal consistency integration network comprises a multi-head attention mechanism, two knowledge maps, a standardization and softmax function and an RMSprop optimizer; respectively determining a similarity matrix of each piece of modal information according to the multi-modal information; combining the similarity matrix between the entities calculated by the embedded representation of the entities with the similarity matrix of each piece of modal information to obtain a comprehensive similarity matrix; and performing entity alignment evaluation according to the comprehensive similarity matrix. According to the method, the complexity in the multi-modal data can be effectively processed, and the specific information of each modal is reserved, so that the precision and robustness of multi-modal entity alignment are improved.

Description

Technical Field

[0001] The present application relates to the field of entity alignment, and in particular to an entity alignment method, device and medium based on multimodal information. Background Art

[0002] Knowledge Graph (KG) is a structured semantic knowledge base that usually uses a resource description framework or an attribute graph model to organize data. It aims to describe entities and their relationships for better information retrieval and knowledge reasoning. With the rapid development of social media and the Internet, the amount of image and text data has increased significantly. More multimodal information such as visual and text has been integrated into the knowledge graph, forming a multimodal knowledge graph (MMKG). This cross-modal data fusion is widely used in fields such as semantic search, recommendation systems, and artificial intelligence, greatly expanding the scope of application of knowledge graphs.

[0003] Although multimodal knowledge graphs have significant advantages, they still face many challenges in practical applications. Existing MMKGs often have problems such as incomplete data and isolated information, which limits their knowledge coverage and accuracy. In addition, multiple MMKGs may contain different descriptions of the same entity, resulting in information redundancy and conflict. These problems seriously affect the practicality and accuracy of knowledge graphs.

[0004] In order to improve the knowledge coverage and accuracy of MMKG, a multi-modal entity alignment method (MMEA) was proposed. MMEA integrates rich information by identifying and matching the same or related entities in different MMKGs, eliminating redundancy and conflict, and building a more comprehensive and accurate knowledge system.

[0005] When performing MMEA tasks, although MMKG contains rich visual and attribute knowledge, how to effectively use this information to enhance the alignment effect is a major challenge. Traditional methods usually rely on shared features in multimodality to learn entity representations, which includes two stages: acquiring the embedding of each modality and fusing modal information. However, this approach often ignores the unique characteristics of the modality, resulting in partial loss of data. Other methods effectively retain modality-specific features by independently learning attribute, visual, and structural information in MMKG, but this strategy does not fully utilize cross-modal consistency features for MMEA.

[0006] Existing studies either only use the consistency information between modalities or only focus on the specific information within the modality to construct entity representation. None of them simultaneously obtains the consistency information and specific information between multiple modalities for MMEA, which affects the final effect.

[0007] Multimodal data has the characteristics of complementarity between modalities and specificity of each modality. Complementarity means that data of different modalities can provide complementary information and enhance each other, making the combined information richer than any single modal information; modal specificity refers to the unique characteristics and information of each modal data. Therefore, there are the following difficulties in obtaining useful information using the characteristics of multimodal data:

[0008] Existing methods represent entity embedding by fusing all information. However, multimodal data usually contains a lot of noise, and the complexity of processing this data is high. These noises and complexities will affect the effect of data fusion, resulting in a decrease in the accuracy of entity representation. Each modality, such as attributes and visual information, interprets entities from different perspectives and there are differences between them. The fused entity embedding mainly focuses on the commonality of information, resulting in the loss of some distinctive information.

[0009] Therefore, how to effectively process multimodal information, remove noise and optimize data fusion to obtain accurate and reliable entity representation and how to retain and utilize discriminative information are still urgent issues to be solved. Summary of the invention

[0010] The purpose of this application is to provide an entity alignment method, device and medium based on multimodal information, which can effectively handle the complexity of multimodal data, retain the specific information of each modality, and thus improve the accuracy and robustness of multimodal entity alignment.

[0011] To achieve the above objectives, this application provides the following solutions:

[0012] In a first aspect, the present application provides an entity alignment method based on multimodal information, and the entity alignment method based on multimodal information includes:

[0013] Acquire multimodal information of each entity; the multimodal information includes: visual embedding of the image, attribute embedding of the text, initial entity embedding and initial relationship embedding; attribute embedding includes: attribute name embedding and attribute value embedding;

[0014] According to the multimodal information of each entity, a cross-modal consistency integration network is used to obtain an embedded representation of the entity; the cross-modal consistency integration network includes: a multi-head attention mechanism, two knowledge graphs, a normalization and softmax function, and an RMSprop optimizer;

[0015] Determine the similarity matrix of each modal information respectively according to the multimodal information;

[0016] The similarity matrix between entities calculated by the embedding representation of the entity is combined with the similarity matrix of each modal information to obtain a comprehensive similarity matrix;

[0017] Entity alignment evaluation is performed based on the comprehensive similarity matrix.

[0018] Optionally, multimodal information of each entity is obtained, including:

[0019] The attribute name set and attribute value set of the text are encoded into attribute name embedding and attribute value embedding respectively using the sentence embedding technology of Siamese network;

[0020] Obtain visual embedding of images using a hierarchical visual transformer with shifted windows;

[0021] Randomly initialize through the embedding layer to obtain the initial entity embedding and the initial relationship embedding.

[0022] Optionally, the step of obtaining the multimodal information of each entity further includes:

[0023] When there is embedded information in the entity, get all the IDs of the entity, traverse the entities corresponding to each ID, and perform weighted summation on different embedded information respectively; the embedded information includes: visual embedding of images and attribute embedding of texts;

[0024] When there is no embedding information in the entity, the modal embedding corresponding to the current entity is obtained by aggregating the embedding information of the first-order neighbor entities.

[0025] Optionally, the training process of the cross-modal consistency integration network specifically includes:

[0026] Use multi-head attention mechanism to update visual embedding, attribute embedding and entity embedding;

[0027] Compute the loss based on the entity embeddings of the two knowledge graphs;

[0028] Use normalization and softmax function to handle loss;

[0029] According to the gradient obtained from the processed loss, the RMSprop optimizer is used to obtain the updated weights.

[0030] Optionally, determining the similarity matrix of each modal information separately according to the multimodal information specifically includes:

[0031] According to the attribute name embedding matrix, the corresponding attribute name-based similarity matrix is ​​obtained;

[0032] The correlation between attribute values ​​is obtained according to the inverse of the difference between attribute values ​​in attribute value embedding;

[0033] The attribute similarity matrix is ​​obtained according to the correlation between the attribute name similarity matrix and the attribute value;

[0034] The visual similarity matrix is ​​obtained according to the visual embedding matrix.

[0035] Optionally, performing entity alignment evaluation according to the comprehensive similarity matrix specifically includes:

[0036] Transform and Sinkhorn normalize the comprehensive similarity matrix;

[0037] Calculate the number of the first K hits, denoted by Hit@K;

[0038] Calculate the average ranking and the average reciprocal ranking;

[0039] Entity alignment evaluation is achieved based on Hit@K, average rank, and average reciprocal rank.

[0040] In a second aspect, the present application provides an entity alignment device based on multimodal information, and the entity alignment device based on multimodal information includes:

[0041] An embedding acquisition module, used to acquire multimodal information of each entity; the multimodal information includes: visual embedding of the image, attribute embedding of the text, initial entity embedding and initial relationship embedding; attribute embedding includes: attribute name embedding and attribute value embedding;

[0042] An embedding processing and iterative update module, which is used to obtain an embedding representation of the entity based on the multimodal information of each entity using a cross-modal consistency integration network; the cross-modal consistency integration network includes: a multi-head attention mechanism, two knowledge graphs, a normalization and softmax function, and an RMSprop optimizer;

[0043] A modality similarity matrix generation module is used to determine the similarity matrix of each modality information according to the multi-modal information;

[0044] An information integration module is used to combine the similarity matrix between entities calculated by the embedding representation of the entity with the similarity matrix of each modal information to obtain a comprehensive similarity matrix;

[0045] Evaluation module, used to evaluate entity alignment based on the comprehensive similarity matrix.

[0046] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the entity alignment method based on multimodal information.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the entity alignment method based on multimodal information.

[0048] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0049] The present application provides an entity alignment method, device and medium based on multimodal information, which is an efficient Multi-modal Consistency and Specificity Fusion Framework (MCSFF). In MCSFF, the noise in the information is removed by the Cross-Modal Consistency Integration Network (CMCINet), and the trained embedded information is fused to obtain a more robust entity representation; for the specificity of modal information, the similarity matrix of each modal information is determined, and the corresponding similarity matrix is ​​obtained for different information to retain more useful information. The present application can effectively handle the complexity in multimodal data, retain the specific information of each modality, and ultimately improve the accuracy and robustness of multimodal entity alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of an entity alignment method based on multimodal information provided in one embodiment of the present application;

[0052] Figure 2 This is the specific structure diagram of MCSFF;

[0053] Figure 3 It is the overall framework flow chart;

[0054] Figure 4 It is the CMCINet flow chart;

[0055] Figure 5 Align flow charts for entities. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0058] In an exemplary embodiment, Figure 1-Figure 5 As shown, a method for entity alignment based on multimodal information is provided, which is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method includes the following S101 to S105. Among them:

[0059] S101, obtaining multimodal information of each entity; the multimodal information includes: visual embedding of the image, attribute embedding of the text, initial entity embedding and initial relationship embedding; attribute embedding includes: attribute name embedding and attribute value embedding;

[0060] S101 obtains vectorized representations of vision and attributes through encoders, which can convert complex, multi-dimensional data (such as images and attribute information) into fixed-length numerical vectors. Vectorized representations facilitate processing and analysis in machine learning and deep learning models, helping to improve model performance and accuracy.

[0061] Different encoders are used for text and images in entities for vector representation;

[0062] For text, the sentence embedding technology of Siamese network (Sentence Embeddings using Siamese BERT-Networks, Sentence Transformer) is used to obtain high-quality, context-aware semantic representation; first, the attribute triples are read and divided into attribute name sets Attribute value set Then we get the embedding by calculating

[0063] in, Represents the attribute name and attribute value of an entity. represents the corresponding embedding representation obtained by encoding the encoder, and n represents the number of attribute triplets.

[0064] For images, a Hierarchical Vision Transformer using Shifted Windows (Swin Transformer) is used to obtain visual embedding. Swin Transformer extracts image features at different scales through a hierarchical architecture and sliding window mechanism, and finally integrates these multi-scale features into a global embedding vector to capture the comprehensive information of the image. It is encoded by the Swin Transformer network as

[0065]

[0066] in, A picture representing an entity, represents the corresponding embedded representation obtained by encoding the encoder, and m represents the number of pictures.

[0067] Randomly initialize the embedding layer to obtain temporary entity embedding and relation embedding

[0068]

[0069] Among them, rel_size represents the total number of relations, rel_hidden represents the dimension of relation embedding, node_size represents the total number of nodes, and node_hidden represents the dimension of node embedding.

[0070] Considering that an entity corresponds to 0 or more attributes and visual embeddings, two cases are processed: one is an entity with embedded information, and the other is an entity without embedded information. Therefore, S101 also includes:

[0071] S1, when there is embedded information in the entity, obtain all the IDs of the entity, traverse the entities corresponding to each ID, and perform weighted summation on different embedded information respectively; the embedded information includes: visual embedding of images and attribute embedding of texts;

[0072] First get all the ids of the entity:

[0073] S(id)={ID1,ID2,ID3,...,ID z};

[0074] Among them, z is the total number of entities, and S(id) is the collection of entity ids.

[0075] Next, traverse the entities corresponding to each id, process different embeddings respectively, obtain the embedding corresponding to the entity, and calculate its weighted sum representation:

[0076]

[0077] Among them, X ij represents the jth information embedding of entity i, which can be the visual embedding I ij Or attribute information is embedded in K ij or V ij .w ij is the weight of the jth information. i is the number of information items owned by entity i. It is a comprehensive embedding of entity i, which can represent both visual information and attribute information.

[0078] S2: When there is no embedding information in the entity, the modal embedding corresponding to the current entity is obtained by aggregating the embedding information of the first-order neighbor entities.

[0079] First, get the neighbor entity id, then get the corresponding embedding of the neighbor entity, and then represent its own embedding by weighted summing up the embeddings of all neighbors of the entity

[0080]

[0081] in, is the embedding representation of entity i that is adjacent to entity j at layer 0. N(j) is the set of entities adjacent to entity j. ij is a normalizing constant. (0) is the weight matrix of layer 0. σ is the nonlinear activation function.

[0082] S102, according to the multimodal information of each entity, a cross-modal consistency integration network is used to obtain an embedded representation of the entity; the cross-modal consistency integration network includes: a multi-head attention mechanism, two knowledge graphs, a normalization and softmax function, and an RMSprop optimizer;

[0083] The training process of the cross-modal consistency integration network includes:

[0084] S201, using multi-head attention mechanism to update visual embedding, attribute embedding and entity embedding;

[0085] For each layer l and each head h, calculate the attention coefficient

[0086]

[0087] Among them, a(h) is the parameter of the attention mechanism, W (h) and is the linear transformation matrix. is the i-th entity embedding (or visual embedding, attribute embedding) in the l-th layer, r ij is the embedding of the relationship between entities i and j, and ‖ represents the vector concatenation operation.

[0088] For each node i, the attention coefficient of the neighbor node j is normalized to get

[0089]

[0090] Among them, N(i) is the set of neighbor nodes of node i.

[0091] Use the obtained attention coefficients to weighted sum to get the new embedding for each node

[0092]

[0093] Where H is the number of attention heads and σ is the activation function.

[0094] Then we get the updated visual embedding Attribute embedding (including attribute name embedding: Attribute value embedding: Relation Embedding Entity Embedding Where x represents the number of entities. The updated embeddings are concatenated to obtain the entity embedding E:

[0095] E=I||K||V||R||E;

[0096] S202, calculating the loss based on the entity embedding of the two knowledge graphs;

[0097] The entity embeddings of the entity pairs in the two knowledge graphs are respectively expressed by L emb 、Re mb表 As shown, all entity embeddings are represented by emb, and the positive sample distance pos_dis is calculated through it:

[0098] pos_dis=||L emb -R emb || 2 ;

[0099] Then calculate the distances l_neg_dis and r_neg_dis between negative samples:

[0100] l_neg_dis=squared_dist(L emb,emb);

[0101] r_neg_dis=ssquared_dist(R emb ,emb);

[0102] The loss l_loss and r_loss are calculated by the distance between negative samples:

[0103] l_loss=pos_dis-l_neg_dis+γ;

[0104] r_loss=pos_dis-r_neg_dis+γ;

[0105] Here, γ is the margin loss.

[0106] S203, processing loss using normalization and softmax function;

[0107]

[0108] l_loss=log∑exp(λ·l_loss+τ);

[0109] r_loss=logΣexp(λ·r_loss+τ);

[0110] Then normalize the loss to get the final loss final_loss:

[0111]

[0112] Calculate the gradient g through the loss t :

[0113]

[0114] S204, according to the gradient obtained from the processed loss, an RMSprop optimizer is used to obtain updated weights.

[0115]

[0116]

[0117] Among them, v t is the exponentially weighted moving average of the squared gradient, β is the momentum parameter, and g t is the current gradient, η is the learning rate, and ∈ is a small constant to prevent division by zero.

[0118] With the goal of reducing loss, CMCINet updates the embedding through forward propagation and back propagation, so that the distance between positive samples gradually decreases and the distance between negative samples gradually increases. Thus, the integration of cross-modal consistency is achieved.

[0119] S103, determining a similarity matrix of each modal information according to the multimodal information;

[0120] In order to retain the unique content of each modal information, the similarity matrix of each modal information is calculated. The following describes the similarity matrix determination process of each modality.

[0121] (1) Attribute similarity matrix calculation

[0122] The similarity matrix calculation of attributes is divided into the similarity matrix of attribute name and attribute value.

[0123] For text information, after obtaining the embedding matrix, the corresponding real name similarity matrix is ​​obtained by matrix multiplication and other methods. For the attribute values, process them in a data manner, determine the inverse of the difference between the attribute values, and obtain the correlation between the attribute values Merge attribute information by adding X s Indicates the number of attributes in the source dataset, X t represents the number of attributes of the target data set. The merged matrix is ​​processed as follows to obtain the matrix S representing the similarity of the two knowledge graph entities: A :

[0124]

[0125] in, is an aggregate function, through which

[0126] (2) Visual similarity matrix calculation

[0127] The obtained visual embedding matrix is ​​dotted to obtain the correlation between multiple visual embeddings

[0128] On this basis, the data is processed and the correlation calculated from the two visual embeddings with the largest correlation is used as the correlation degree of the two entities in the visual modality. Depend on The matrix composed is the visual similarity matrix S I .

[0129] S104, combining the similarity matrix between entities calculated by the embedding representation of the entity with the similarity matrix of each modal information to obtain a comprehensive similarity matrix;

[0130] The similarity score is calculated based on the entity embedding set (Lvec, Rvec) in the validation set entity pair, and a similarity matrix S based on entity embedding is constructed. E :

[0131] S E =Lvec·Rvec T ;

[0132] Add the similarity matrices of vision, attributes, and entities to obtain the final comprehensive similarity matrix for detection:

[0133] S=S E +S I +S V ;

[0134] S105, performing entity alignment evaluation based on the comprehensive similarity matrix.

[0135] The comprehensive similarity matrix is ​​transformed and Sinkhorn normalized to obtain S sinkhorn :

[0136] S′=1-S;

[0137] S sinkhorn = Sinkhorn(S′ i );

[0138] Calculate the number of the first K hits, expressed as Hit@K (mainly verified by Hit@1, Hit@5, and Hit@10):

[0139]

[0140] Among them, I represents the matching index, L represents the link matrix of entity pairs, and 1() is an indicator function, which is 1 when the condition is met and 0 otherwise.

[0141] Calculate the average ranking (MeanRank, MR):

[0142]

[0143] Among them, rank(L[i], S[i,:]) represents the ranking of entity L[i] in the similarity matrix S[i,:].

[0144] Calculate the Mean Reciprocal Rank (MRR):

[0145]

[0146] By calculating the above evaluation indicators, we can observe the effect of the entity alignment method: the lower the Hit@K and MR, the better the method effect, and the higher the MRR, the better the method effect.

[0147] Compared with the prior art, this application has the following beneficial technical effects:

[0148] (1) In terms of information processing, by processing various embedded information, noise reduction is achieved and semantically accurate entity embedding is obtained. This embedding can better represent entities and outperforms most existing technologies in entity alignment tasks.

[0149] (2) Based on the unique features of each modal information, the unimodal similarity matrix is ​​obtained by using the corresponding method. These similarity matrices provide complementary information, so after adding the unimodal similarity matrix, the entity alignment effect is significantly improved.

[0150] (3) The technology solves the problem of under-utilization of modal information. By learning the consistency and specificity information of entities, the modal information is effectively utilized, and the experimental results are better than the baseline model.

[0151] In order to solve the problem of insufficient information utilization in the multimodal entity alignment task, the MCSFF overall network framework and the CMCINet consistency information processing network are constructed. Figure 4 As shown in the figure. By utilizing the complementarity and specificity of multimodal information, the consistency information between modalities and the specific information within the modalities are learned, thus achieving effective utilization of modal information. Compared with existing popular methods, the method performs better in alignment effect. Specifically, the multimodal knowledge graph MMKB is taken as an example for further explanation.

[0152] The selected MMKB dataset is a multimodal knowledge graph that integrates information from multiple modalities, including text, images, and structured data, to support multimodal machine learning tasks. DB15K, FB15K, and YAGO15K in MMKB are three typical multimodal knowledge graphs, of which DB15K and YAGO15K are source datasets, and FB15K is the target dataset. Image data was downloaded according to the url value corresponding to the entity name. 66989, 88402, and 59144 images were downloaded from the DB15K, FB15K, and YAGO15K datasets, respectively. Figure 3 As shown, in the specific implementation process of this application, performing multimodal entity alignment includes the following steps:

[0153] First, get the embedded representation of the information:

[0154] All image data are vectorized using SwinTransformer to obtain image embedding representations corresponding to the three knowledge graphs, with each vector dimension being 1536. All attribute information is vectorized using Sentence Transformer to obtain attribute embedding representations with a dimension of 1024, including attribute name embedding representations and attribute value embedding representations.

[0155] Traverse each entity according to the entity name number. If the entity has an embedding representation of the corresponding modality, perform a weighted sum of multiple embedding information. For entities with modal embedding information, obtain its modal embedding by aggregating the modal embedding representations of its neighbors. Repeat the above operations so that each entity has a unique corresponding visual embedding, attribute name embedding, and attribute value embedding.

[0156] In this example, some hyperparameters in the CMCINet model are assigned. This application uses the RMSprop optimizer in training, updates the network parameters for 300 epochs, and sets the initial learning rate to 5e-3. All experiments are performed on an NVIDIA GeForce GTX 3090Ti GPU.

[0157] The number of seeds in the DB15K-FB15K and YAGO15K-FB15K datasets are 12846 and 11199 pairs respectively. The training pairs and test pairs are divided into training pairs and testing pairs in a ratio of 2:8, and the parameters are trained by reducing the loss.

[0158] After obtaining the embedding vector, the corresponding similarity matrix is ​​generated by combining the characteristics of each modality. In visual embedding, the image score between the source dataset and the target dataset is calculated by maximizing the score, and the maximum score is used as the specific score between the two entities. The score between the DB15K-FB15K and YAGO15K-FB15K datasets is calculated by the above method, and the final score is saved in the form of an array. The final evaluation scores of the two similarity matrices are 16.73% and 17.01%. In attribute embedding, attribute data are all numerical data. In order to effectively use the attribute name information and attribute value information, the attribute name and attribute value are processed separately, and the score between the attribute embeddings is calculated based on the embedding representation of the attribute name. For attribute values, the score between the attribute values ​​is represented by the inverse of the difference between the calculated values. The two score matrices are combined by addition, and the final score is saved in the form of an array. The final evaluation scores of the two similarity matrices are 32.98% and 34.45%.

[0159] The trained CMCINet is used to obtain the embedded representation of the entity, which integrates visual, attribute and relationship information. Then the embedded representation of the entity is used to calculate the similarity score between entities and form a similarity matrix. The effect of the network on the two data sets obtained through the score matrix is: 67.25% and 68.22%.

[0160] The similarity matrix obtained based on entity embedding is added to the vision-based and attribute-based similarity matrices to achieve the fusion of consistency information and specificity information.

[0161] The effect of entity alignment calculated based on the final score matrix is ​​80.27% and 82.72% on the two datasets DB15K-FB15K and YAGO15K-FB15K respectively.

[0162] Based on the same inventive concept, the embodiment of the present application also provides a multimodal information-based entity alignment device for implementing the multimodal information-based entity alignment method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multimodal information-based entity alignment device embodiments provided below can refer to the limitations of the multimodal information-based entity alignment method above, and will not be repeated here.

[0163] In an exemplary embodiment, a multimodal information-based entity alignment device is provided, comprising:

[0164] An embedding acquisition module, used to acquire multimodal information of each entity; the multimodal information includes: visual embedding of the image, attribute embedding of the text, initial entity embedding and initial relationship embedding; attribute embedding includes: attribute name embedding and attribute value embedding;

[0165] An embedding processing and iterative update module, which is used to obtain an embedding representation of the entity based on the multimodal information of each entity using a cross-modal consistency integration network; the cross-modal consistency integration network includes: a multi-head attention mechanism, two knowledge graphs, a normalization and softmax function, and an RMSprop optimizer;

[0166] A modality similarity matrix generation module is used to determine the similarity matrix of each modality information according to the multi-modal information;

[0167] An information integration module is used to combine the similarity matrix between entities calculated by the embedding representation of the entity with the similarity matrix of each modal information to obtain a comprehensive similarity matrix;

[0168] Evaluation module, used to evaluate entity alignment based on the comprehensive similarity matrix.

[0169] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an entity alignment method based on multimodal information is implemented.

[0170] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0171] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0172] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0173] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0174] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0175] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An entity alignment method based on multimodal information, characterized in that: The entity alignment method based on multimodal information includes: Acquire multimodal information of each entity; the multimodal information includes: visual embedding of the image, attribute embedding of the text, initial entity embedding and initial relationship embedding; attribute embedding includes: attribute name embedding and attribute value embedding; According to the multimodal information of each entity, a cross-modal consistency integration network is used to obtain an embedded representation of the entity; the cross-modal consistency integration network includes: a multi-head attention mechanism, two knowledge graphs, normalization and softmax functions, and an RMSprop optimizer; Determine the similarity matrix of each modal information respectively according to the multimodal information; The similarity matrix between entities calculated by the embedding representation of the entity is combined with the similarity matrix of each modal information to obtain a comprehensive similarity matrix; Entity alignment evaluation is performed based on the comprehensive similarity matrix.

2. The entity alignment method based on multimodal information according to claim 1, characterized in that: Get multimodal information for each entity, including: The attribute name set and attribute value set of the text are encoded into attribute name embedding and attribute value embedding respectively using the sentence embedding technology of Siamese network; Obtain visual embedding of images using a hierarchical visual transformer with shifted windows; Randomly initialize through the embedding layer to obtain the initial entity embedding and the initial relationship embedding.

3. The entity alignment method based on multimodal information according to claim 1, characterized in that: The step of obtaining multimodal information of each entity further includes: When there is embedded information in the entity, get all the ids of the entity, traverse the entities corresponding to each id, and perform weighted summation on different embedded information respectively; the embedded information includes: visual embedding of images and attribute embedding of texts; When there is no embedding information in the entity, the modal embedding corresponding to the current entity is obtained by aggregating the embedding information of the first-order neighbor entities.

4. The entity alignment method based on multimodal information according to claim 1, characterized in that: The training process of the cross-modal consistency integration network includes: Use multi-head attention mechanism to update visual embedding, attribute embedding and entity embedding; Compute the loss based on the entity embeddings of the two knowledge graphs; Use normalization and softmax function to handle loss; According to the gradient obtained from the processed loss, the RMSprop optimizer is used to obtain the updated weights.

5. The entity alignment method based on multimodal information according to claim 1, characterized in that: The similarity matrix of each modal information is determined according to the multimodal information, specifically including: According to the attribute name embedding matrix, the corresponding attribute name-based similarity matrix is ​​obtained; The correlation between attribute values ​​is obtained according to the inverse of the difference between attribute values ​​in attribute value embedding; The attribute similarity matrix is ​​obtained according to the correlation between the attribute name similarity matrix and the attribute value; The visual similarity matrix is ​​obtained according to the visual embedding matrix.

6. The entity alignment method based on multimodal information according to claim 1, characterized in that: The entity alignment evaluation based on the comprehensive similarity matrix specifically includes: Transform and Sinkhorn normalize the comprehensive similarity matrix; Calculate the number of the first K hits, denoted by Hit@K; Calculate the average ranking and the average reciprocal ranking; Entity alignment evaluation is achieved based on Hit@K, average rank, and average reciprocal rank.

7. An entity alignment device based on multimodal information, characterized in that: The entity alignment device based on multimodal information includes: An embedding acquisition module, used to acquire multimodal information of each entity; the multimodal information includes: visual embedding of the image, attribute embedding of the text, initial entity embedding and initial relationship embedding; attribute embedding includes: attribute name embedding and attribute value embedding; An embedding processing and iterative update module, which is used to obtain an embedding representation of the entity based on the multimodal information of each entity using a cross-modal consistency integration network; the cross-modal consistency integration network includes: a multi-head attention mechanism, two knowledge graphs, a normalization and softmax function, and an RMSprop optimizer; A modality similarity matrix generation module is used to determine the similarity matrix of each modality information according to the multi-modal information; An information integration module is used to combine the similarity matrix between entities calculated by the embedding representation of the entity with the similarity matrix of each modal information to obtain a comprehensive similarity matrix; Evaluation module, used to evaluate entity alignment based on the comprehensive similarity matrix.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the entity alignment method based on multimodal information as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the entity alignment method based on multimodal information described in any one of claims 1 to 6 is implemented.

Images