Knowledge graph alignment method and device, equipment and storage medium

Abstract

The application discloses a knowledge graph alignment method and device, equipment and a storage medium, and belongs to the technical field of artificial intelligence. Through the technical scheme provided in the application, after the image embedding of the multiple images of the entity in the knowledge graph is determined, the multiple images of the entity are filtered based on the image embedding of the images, and the images with a similarity meeting a condition are retained, so that the purpose of removing images that are not beneficial to knowledge graph alignment is achieved. Therefore, when a knowledge graph alignment model is trained based on the filtered images and a labeled alignment result, the knowledge graph alignment model can learn more concise images, and the performance of the knowledge graph alignment model is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a knowledge graph alignment method and device, equipment and a storage medium. BACKGROUND

[0002] A knowledge graph is composed of entities and relationships between the entities, and a knowledge graph alignment technique is of great significance for constructing a large-scale high-quality knowledge graph. Generally, based on a trained knowledge graph alignment model, entities in two knowledge graphs are aligned to introduce the content of other knowledge graphs into one knowledge graph, thereby integrating knowledge graphs of different granularities and different vertical domains.

[0003] In related technologies, an image corresponding to an entity can be used as a reference information for entity alignment. However, a knowledge graph often contains a large number of images, and the image corresponding to the entity can contain noise images, that is, images that are useless for knowledge graph alignment. Therefore, how to screen the images in the knowledge graph to train the knowledge graph alignment model based on the screened images to enhance the performance of the model has become a problem to be solved. SUMMARY

[0004] The embodiments of the present application provide a knowledge graph alignment method, device, equipment and storage medium, which can enhance the performance of the knowledge graph alignment model. The technical solution is as follows:

[0005] On the one hand, a knowledge graph alignment method is provided, which includes:

[0006] Determine, by a knowledge graph alignment model, image embeddings of a plurality of first images of a first entity in a first knowledge graph, and determine image embeddings of a plurality of second images of a second entity in a second knowledge graph;

[0007] Screen the plurality of first images based on the image embeddings of the plurality of first images and the image embeddings of the plurality of second images to obtain a plurality of first target images, and screen the plurality of second images to obtain a plurality of second target images, wherein the similarity of the plurality of first target images meets a similarity condition, and the similarity of the plurality of second target images meets the similarity condition;

[0008] Train the knowledge graph alignment model based on the plurality of first target images of the first entity, the plurality of second target images of the second entity and a labeled alignment result, wherein the labeled alignment result is used to represent whether the first entity and the second entity are aligned;

[0009] The knowledge graph alignment model is used to output an alignment result of entities in two input knowledge graphs based on images of the entities in the two knowledge graphs.

[0010] In an aspect, a knowledge graph alignment method is provided, the method comprising:

[0011] inputting entity information of a third entity in a first knowledge graph and entity information of a fourth entity in a second knowledge graph into a knowledge graph alignment model, the entity information comprising an entity name, a relationship, an attribute, and a plurality of images;

[0012] determining, by the knowledge graph alignment model, a second entity similarity of each entity pair based on entity embeddings of the third entity and the fourth entity included in the entity pair;

[0013] outputting, based on the second entity similarity, an alignment result of the first knowledge graph and the second knowledge graph, the alignment result being used to represent whether the third entity and the fourth entity are aligned;

[0014] The knowledge graph alignment model is trained by the knowledge graph alignment method of the above aspect.

[0015] In an aspect, a knowledge graph alignment device is provided, the device comprising:

[0016] a first determining module configured to determine, by a knowledge graph alignment model, image embeddings of a plurality of first images of a first entity in a first knowledge graph and image embeddings of a plurality of second images of a second entity in a second knowledge graph;

[0017] a screening module configured to screen the plurality of first images based on the image embeddings of the plurality of first images and the image embeddings of the plurality of second images to obtain a plurality of first target images, and screen the plurality of second images to obtain a plurality of second target images, a similarity of the plurality of first target images meeting a similarity condition, and a similarity of the plurality of second target images meeting the similarity condition;

[0018] a training module configured to train the knowledge graph alignment model based on the plurality of first target images of the first entity, the plurality of second target images of the second entity, and a labeled alignment result, the labeled alignment result being used to represent whether the first entity and the second entity are aligned;

[0019] The knowledge graph alignment model is used to output an alignment result of entities in two input knowledge graphs based on images of the entities in the two knowledge graphs.

[0020] In some embodiments, the screening module is configured to, for each of the first images, determine a first image similarity between an image embedding of the first image and an image embedding of each of the second images; and prune, from the plurality of first images, a first image with a first pruning score higher than a pruning threshold, to obtain the plurality of first target images, the first pruning score being a difference between 1 and a mean of the plurality of first image similarities.

[0021] In some embodiments, the training module comprises:

[0022] a determining unit configured to determine a mean image embedding of the plurality of first target images as a first image embedding of the first entity, and determine a mean image embedding of the at least one second target image as a second image embedding of the second entity;

[0023] a training unit configured to train the knowledge graph alignment model based on the first image embedding of the first entity, the second image embedding of the second entity, and the annotation alignment result.

[0024] In some embodiments, the training unit is configured to determine an i-th image loss value based on the first image embedding of the first entity, the second image embedding of the second entity, and the annotation alignment result, i being a positive integer not less than 1.

[0025] In a case where the image loss value meets a loss value condition, stop training the knowledge graph alignment model;

[0026] In a case where the image loss value does not meet the loss value condition, screen a plurality of first target images of the first entity based on a first entity embedding of the first entity and a second entity embedding of the second entity to obtain at least one third target image, and screen a plurality of second target images of the second entity to obtain at least one fourth target image, wherein a similarity between the third target image and the fourth target image meets the similarity condition, the first entity embedding is determined based on the first image embedding, and the second entity embedding is determined based on the second image embedding;

[0027] based on the image loss value, the at least one third target image of the first entity, and the at least one fourth target image of the second entity, perform an i+1-th training on the knowledge graph alignment model, until the image loss value meets the loss value condition.

[0028] In some embodiments, the training unit is configured to determine the ith image loss value based on a target entity pair and a corresponding loss value, wherein the target entity pair is an entity pair with a labeled alignment result, and the target entity pair includes the first entity and the second entity, and the loss value is determined based on a first image embedding of the first entity and a second image embedding of the second entity.

[0029] In some embodiments, the training unit is configured to, for each first target image, determine a second image similarity between an image embedding of the first target image and image embeddings of each second target image, and determine a second pruning score of the first target image as a difference between 1 and a mean value of the second image similarities.

[0030] For each first target image, determine a first entity similarity between the first entity embedding and a second entity embedding of a second entity to which the corresponding second target image belongs.

[0031] From the plurality of first target images, prune the first target images with a third pruning score higher than a pruning threshold to obtain the at least one third target image, wherein the third pruning score is a product of the first entity similarity and the second pruning score.

[0032] In some embodiments, the apparatus further includes:

[0033] The third determining module is configured to determine, by the knowledge graph alignment model, a first name embedding, a first relation embedding, and a first attribute embedding of the first entity, and determine a second name embedding, a second relation embedding, and a second attribute embedding of the second entity; determine a first entity embedding of the first entity based on the first name embedding, the first relation embedding, the first attribute embedding, and the first image embedding; and determine a second entity embedding of the second entity based on the second name embedding, the second relation embedding, the second attribute embedding, and the second image embedding.

[0034] In some embodiments, the third determining module is configured to perform weighted average on the first name embedding, the first relation embedding, the first attribute embedding, and the first image embedding to obtain the first entity embedding of the first entity.

[0035] In some embodiments, the knowledge graph alignment model includes a name embedding determination model, a relation embedding determination model, and an attribute embedding determination model.

[0036] The third determining module is configured to determine, by the name embedding determination model, a first name embedding of the first entity; determine, by the relationship embedding determination model, a first relationship embedding of the first entity; and determine, by the attribute embedding determination model, a first attribute embedding of the first entity.

[0037] In an aspect, a knowledge graph alignment apparatus is provided, and the apparatus comprises:

[0038] The input module is configured to input entity information of a third entity in a first knowledge graph and entity information of a fourth entity in a second knowledge graph into a knowledge graph alignment model, the entity information comprising an entity name, a relationship, an attribute, and a plurality of images.

[0039] The second determining module is configured to determine, by the knowledge graph alignment model, a second entity similarity of each entity pair, the second entity similarity being determined based on entity embeddings of the third entity and the fourth entity included in the entity pair.

[0040] The output module is configured to output an alignment result of the first knowledge graph and the second knowledge graph based on the second entity similarity, the alignment result being used to represent whether the third entity and the fourth entity are aligned.

[0041] The knowledge graph alignment model is trained by the knowledge graph alignment method of the above aspect.

[0042] In an aspect, a computer device is provided, and the computer device comprises one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the knowledge graph alignment method of the above aspect.

[0043] In an aspect, a computer readable storage medium is provided, and the computer readable storage medium stores at least one computer program, the computer program being loaded and executed by a processor to implement the knowledge graph alignment method of the above aspect.

[0044] In an aspect, a computer program product is provided, and the computer program product comprises a computer program stored in a computer readable storage medium, a processor of a computer device reading the computer program from the computer readable storage medium, and the processor executing the computer program to enable the computer device to implement the knowledge graph alignment method of the above aspect.

[0045] Through the technical solutions provided in the embodiments of the present application, after the image embedding of the plurality of images of the entity in the knowledge graph is determined, the plurality of images of the entity are filtered based on the image embedding of the images, and the images with a similarity meeting a condition are retained, so that the purpose of removing images that are not beneficial to knowledge graph alignment is achieved, thereby when the knowledge graph alignment model is trained based on the filtered images and the labeled alignment result, the knowledge graph alignment model can learn more concise images, and thus the performance of the knowledge graph alignment model is enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0046] To make the technical solutions in the embodiments of the present application clearer, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0047] Figure 1 is a schematic diagram of an implementation environment of a knowledge graph alignment method provided by an embodiment of the present application;

[0048] Figure 2 is a flowchart of a knowledge graph alignment method provided by an embodiment of the present application;

[0049] Figure 3 is a flowchart of a knowledge graph alignment method provided by an embodiment of the present application;

[0050] Figure 4 is a schematic diagram of a knowledge graph before and after pruning provided by an embodiment of the present application;

[0051] Figure 5 is a flowchart of a knowledge graph alignment method provided by an embodiment of the present application;

[0052] Figure 6 is a flowchart of a knowledge graph alignment method provided by an embodiment of the present application;

[0053] Figure 7 is a flowchart of a knowledge graph alignment method provided by an embodiment of the present application;

[0054] Figure 8 is a structural schematic diagram of a knowledge graph alignment device provided by an embodiment of the present application;

[0055] Figure 9 is a structural schematic diagram of a knowledge graph alignment device provided by an embodiment of the present application;

[0056] Figure 10 is a structural schematic diagram of a terminal provided by an embodiment of the present application;

[0057] Figure 11 is a structural schematic diagram of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] For the purpose, technical solutions and advantages of the present application to be clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0059] In the present application, the terms "first", "second" and the like are used to distinguish the same or similar items with substantially the same function and action, and it should be understood that there is no logical or time sequence dependency between "first", "second", "nth", and the number and execution order are not limited.

[0060] In the present application, the term "at least one" means one or more, and the meaning of "multiple" is two or more, for example, multiple images means two or more images.

[0061] Artificial intelligence (AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] Computer vision technology (CV) is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process images so that the computer processing is more suitable for human eye observation or image transmission to instrument detection.

[0063] The technical solutions provided by the embodiments of the present application can also be combined with cloud technology, for example, deploying the trained knowledge graph alignment model on a cloud server. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software and network in a wide area network or local area network to realize data calculation, storage, processing and sharing.

[0064] Among them, the medical cloud in cloud technology refers to using "cloud computing" to create a medical and health service cloud platform based on cloud computing, mobile technology, multimedia, 4G communication, big data, and other new technologies, combined with medical technology, to realize the sharing of medical resources and the expansion of the scope of medical treatment. Because of the combination of cloud computing technology, the medical cloud improves the efficiency of medical institutions and facilitates residents to seek medical treatment. Like the hospital's appointment registration, electronic medical records, medical insurance, etc. are the products of the combination of cloud computing and the medical field. The medical cloud also has the advantages of data security, information sharing, dynamic expansion, and global layout. Illustratively, the knowledge graph alignment model provided in the embodiments of the present application is deployed on a medical and health service cloud platform.

[0065] Modality: each source or form of information can be called a modality. For example, the media of information are voice, pictures, or text, etc.; for another example, various sensors such as radar sensors, infrared sensors, or accelerometers, etc. Each of the above can be called a modality.

[0066] Multimodal machine learning aims to realize the ability to process and understand multimodal information through the method of machine learning. The current popular research direction is multimodal learning between images, videos, audio, or semantics.

[0067] Knowledge graph is a semantic network that reveals the relationship between entities, and is also a graph-based data structure.

[0068] Knowledge graph alignment, also known as entity alignment, aims to determine whether two or more different knowledge graphs point to the same object in the real world. If multiple entities represent the same object, an alignment relationship is established between these entities.

[0069] Entity is an objective existence and can be distinguished from each other, such as "dog" or "cat", etc.; at the same time, entity is also the basic unit of knowledge graph, and is an important language unit that carries information in text.

[0070] Optionally, the computer device provided in the embodiments of the present application provides a terminal or a server, and the implementation environment composed of the terminal and the server is introduced below.

[0071] Figure 1 is a schematic diagram of an implementation environment of a knowledge graph alignment method provided by the embodiments of the present application, referring to Figure 1 The implementation environment includes a terminal 110 and a server 120.

[0072] The terminal 110 is connected to the server 120 through a wireless network or a wired network. Optionally, the terminal 110 is a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart watch, and the like, but is not limited thereto. Optionally, the terminal 110 is installed and runs an application program supporting knowledge graph alignment, which is provided with a background service by the server 120.

[0073] Optionally, the server 120 is a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (CDN), and big data and artificial intelligence platform.

[0074] Optionally, the terminal 110 generally refers to one of multiple terminals, and the embodiments of the present application take the terminal 110 as an example. Those skilled in the art know that the number of the terminal 110 can be more or less. For example, the terminal 110 is one, or the terminal 110 is dozens or hundreds, or more, and at this time, the above-mentioned implementation environment also includes other terminals. The number and type of the terminal are not limited in the embodiments of the present application.

[0075] In some embodiments, the knowledge graph contains a large amount of information, so the knowledge graph can be stored in a database based on a table structure, or stored in a database based on a graph structure. The sample knowledge graph in the embodiments of the present application is obtained from the database by the server.

[0076] After introducing the implementation environment of the knowledge graph alignment method provided by the embodiments of the present application, the application scenarios of the knowledge graph alignment method provided by the embodiments of the present application are described below. It should be noted that the terminal in the following description process is the terminal 110 in the above-mentioned implementation environment, and the server is the server 120 in the above-mentioned implementation environment. The knowledge graph alignment method provided by the embodiments of the present application can be applied in various knowledge graph alignment scenarios, such as medical knowledge graph alignment scenarios, and the embodiments of the present application do not limit this.

[0077] In some embodiments, taking the alignment of two medical knowledge graphs as an example, the terminal obtains the identifier of the medical knowledge graph to be aligned, sends the identifier to the server, obtains the medical knowledge graph corresponding to the identifier by the server, aligns the two medical knowledge graphs by the knowledge graph alignment method provided by the embodiments of the present application, obtains the alignment result, and sends the alignment result to the terminal by the server. The terminal receives the alignment result, so that the user can view the alignment result of the knowledge graph through the terminal.

[0078] It can be seen that the knowledge graph alignment model trained by the knowledge graph alignment method provided in the embodiments of the present application can help to quickly align two or even multiple large-scale heterogeneous knowledge graphs (entities in the knowledge graph include images or do not include images), especially the alignment of knowledge graphs with different granularities (for example, aligning a general knowledge graph in the medical field and a cancer vertical knowledge graph). Moreover, the obtained alignment result can effectively help the smooth progress of subsequent tasks, such as knowledge graph representation learning or knowledge graph fusion. In addition, the knowledge graph alignment method provided in the embodiments of the present application can also be applied in the construction and updating iteration process of the knowledge graph, thereby enriching the knowledge coverage and knowledge accuracy of the knowledge graph.

[0079] After introducing the implementation environment, application scenario and composition of the knowledge graph alignment model of the embodiments of the present application, the knowledge graph alignment method provided in the embodiments of the present application will be described below. In the embodiments of the present application, the knowledge graph alignment method provided in the embodiments of the present application can be implemented by a server or a terminal as an execution subject, or the knowledge graph alignment method provided in the present application can be implemented through interaction between the terminal and the server, wherein the terminal is the terminal 110 in the above implementation environment, and the server is the server 120 in the above implementation environment. The embodiments of the present application do not limit the execution subject.

[0080] In some embodiments, the server trains the knowledge graph alignment model, stores the trained knowledge graph alignment model, and when it is necessary to align the knowledge graphs, acquires the knowledge graphs corresponding to the identifiers sent by the terminal, aligns the knowledge graphs through the trained knowledge graph alignment model, and returns the alignment result to the terminal, thereby saving the workload of the terminal. In other embodiments, the server can periodically align two or more knowledge graphs through the trained knowledge graph alignment model.

[0081] Figure 2 is a flowchart of a knowledge graph alignment method provided in the embodiments of the present application, referring to Figure 2 The embodiments of the present application are executed by the server, and the method comprises:

[0082] 201. The server determines the image embedding of the multiple first images of the first entity in the first knowledge graph through the knowledge graph alignment model, and determines the image embedding of the multiple second images of the second entity in the second knowledge graph.

[0083] The first knowledge graph and the second knowledge graph are both knowledge graphs stored in a database. Taking the first knowledge graph as an example, the first knowledge graph includes a plurality of first entities, and the first knowledge graph is used to represent the relationship between the plurality of first entities. For each first entity, the first knowledge graph further includes a plurality of first images corresponding to the first entity. For example, the first entity is “apple”, and the first image of the first entity includes an image of an apple.

[0084] The image embedding is an embedding vector capable of representing image features. In the embodiments of the present application, the server converts (reduces dimension) the image into a fixed-size image embedding, so as to facilitate subsequent processing and calculation.

[0085] It should be noted that the image embedding of the plurality of first images of the first entity includes the image embedding of each first image, that is, the plurality of image embeddings of the first entity, and each image embedding corresponds to a first image.

[0086] In the embodiments of the present application, the sample entity used to train the knowledge graph alignment model is part of the entities in the first knowledge graph and the second knowledge graph. The trained knowledge graph alignment model is used to align the entities in the first knowledge graph and the second knowledge graph.

[0087] 202. The server filters the plurality of first images based on the image embedding of the plurality of first images and the image embedding of the plurality of second images to obtain a plurality of first target images, and filters the plurality of second images to obtain a plurality of second target images, the similarity of the plurality of first target images meets the similarity condition, and the similarity of the plurality of second target images meets the similarity condition.

[0088] The similarity of the first target image is the average of the similarity of the first target image and each second target image in the second knowledge graph. The similarity of the second target image is the average of the similarity of the second target image and each first target image in the first knowledge graph.

[0089] For example, the first entity A in the first knowledge graph corresponds to 3 first images, the plurality of first target images obtained by filtering are image A and image B, the second knowledge graph includes 3 second entities, each second entity corresponds to 2 second images, and the sum of the number of second target images obtained by filtering the 3 second entities is 4. The similarity of image A is the average of the similarity of image A and the 4 second target images, and the similarity of image B is the average of the similarity of image A and the 4 second target images.

[0090] 203. The server trains a knowledge graph alignment model based on the plurality of first target images of the first entity, the plurality of second target images of the second entity, and the labeled alignment result, and the labeled alignment result is used to represent whether the first entity and the second entity are aligned.

[0091] The knowledge graph alignment model is configured to output an alignment result of entities in the two knowledge graphs based on images of the entities in the input two knowledge graphs. In some embodiments, sample entities in the first knowledge graph and the second knowledge graph are labeled in advance in a manual labeling manner to obtain a labeled alignment result.

[0092] By the technical solutions provided in the embodiments of the present application, after the image embeddings of the multiple images of the entities in the knowledge graph are determined, the multiple images of the entities are filtered based on the image embeddings of the images, and the images with a similarity meeting a condition are retained, so that the purpose of removing images that are not beneficial to knowledge graph alignment is achieved. Therefore, when the knowledge graph alignment model is trained based on the filtered images and the labeled alignment result, the knowledge graph alignment model can learn more concise images, and thus the performance of the knowledge graph alignment model is enhanced.

[0093] In the embodiments of the present application, the training process of the knowledge graph alignment model includes multiple iteration processes. In the first iteration process, the server inputs the entity information of the sample knowledge graph into the knowledge graph alignment model to obtain a prediction result of the first iteration process. Based on the prediction result of the first iteration process, a loss value is determined, and the model parameters of the model are adjusted based on the loss value. The model parameters adjusted in the first iteration are taken as the model parameters of the second iteration, and the second iteration process is performed again. The above iteration process is repeated multiple times until the training meets a target condition, and a trained knowledge graph alignment model is obtained.

[0094] In some embodiments, the target condition met by the training is that the number of iterations of the model training reaches a target number, and the target number is a pre-set training iteration number, such as 1000 times. Alternatively, the target condition met by the training is that the loss value meets a loss value condition, such as the loss value being less than 0.0001. The setting of the target condition is not limited in the embodiments of the present application. In the embodiments of the present application, the case where the target condition is that the loss value meets the loss value condition is taken as an example for illustration.

[0095] Figure 3 is a flowchart of a knowledge graph alignment method provided in the embodiments of the present application, referring to Figure 3 The embodiments of the present application take the i-th iteration process of the knowledge graph alignment model as an example for illustration, i is a positive integer not less than 1, and the method includes the following steps.

[0096] 301、The server determines, through the knowledge graph alignment model, image embeddings of multiple first images of a first entity in a first knowledge graph, and determines image embeddings of multiple second images of a second entity in a second knowledge graph.

[0097] The knowledge graph alignment model comprises an image embedding determination model, which is configured to determine an image embedding of an input image. In some embodiments, the implementation of step 301 comprises: inputting, by the server, a plurality of first images of the first entity in the first knowledge graph and a plurality of second images of the second entity in the second knowledge graph into the image embedding determination model, and outputting, by the image embedding determination model, an image embedding of each first image and an image embedding of each second image.

[0098] The image embedding determination model is a model pre-trained by the server. In the embodiments of the present application, for images, since image information and text information are different, the text information can be directly given a reasonable embedding by using a word vector or a sentence vector, for example, the attributes of an entity can be in the form of a word or a sentence; however, the image does not have the concept of a word vector or a sentence vector, and the image embedding determination model can be used to extract the image embedding, i.e., the feature vector of the image.

[0099] In some embodiments, the image embedding determination model is a convolutional neural network model, for example, a ResNet (Residual Network) model. The image embedding determination model comprises a feature extraction layer and a fully connected layer. The feature extraction layer comprises a convolutional layer and a pooling layer, and the number of the convolutional layer and the pooling layer is not limited in the present application. The number of nodes of the fully connected layer is the same as the length of the output image embedding.

[0100] 302. The server determines, for each first image, a first image similarity between the image embedding of the first image and the image embedding of each second image.

[0101] The first image similarity is used to represent the similarity between the first image and the second image. In some embodiments, the server measures the similarity between two images by the cosine similarity between the image embeddings of the two images, and the implementation of the server determining the first image similarity between the image embedding of the first image and the image embedding of each second image can be achieved by the following formula one:

[0102] Formula one:

[0103] wherein a i is the i-th image in the first knowledge graph, i is the serial number of the image, i = 1, 2, 3, …, m, m is the number of images in the first knowledge graph; b j is the image in the second knowledge graph, j is the serial number of the image, j = 1, 2, 3, …, n, n is the number of images in the second knowledge graph; sim(a i ,b j ) is the cosine similarity between ai and b j The value range of the cosine similarity sim(a i ,b j ) is (0, 1); Enc() is image embedding, and ||Enc(a i )|| is the vector length of the image embedding of a i In step 402, a i may be the i th first image, m may be the number of first images, b j may be the j th second image, n may be the number of second images, and sim(a i ,b j ) may be the first image similarity between a i and b j .

[0104] In the embodiments of the present application, since the image embedding of each image can represent the semantic features of the image, and the cosine similarity between two image embeddings can represent the closeness of the two image embeddings, and the cosine similarity is positively correlated with the closeness, the greater the cosine similarity between two image embeddings, the closer the two image embeddings, and the cosine similarity between two image embeddings can also represent the similarity between two images, so the accuracy of the determined image similarity is high.

[0105] 303. The server prunes, from the plurality of first images, a first image whose first pruning score is higher than a pruning threshold to obtain a plurality of first target images, the first pruning score being a difference between 1 and a mean value of the plurality of first image similarities, and the similarity of the first target image meeting a similarity condition.

[0106] For each first image, the server determines, through step 302, a plurality of first image similarities, the number of the plurality of first image similarities being the same as the number of second images included in the second knowledge graph. The pruning threshold can be set as needed, and the embodiments of the present application do not limit this. In some embodiments, the determination process of the first pruning score can be implemented through the following Formula Two:

[0107] Formula Two:

[0108] Wherein, prune is the pruning score; a i is the i th image, i is the serial number of the image, i = 1, 2, 3, …, m, m is the number of images in the first knowledge graph; B is an image set of a plurality of images included in the second knowledge graph, b j is the j th image in B, j is the serial number of the image, j = 1, 2, 3, …, n, n is the number of images in the second knowledge graph; sim(ai j i j i j i j i j i j

[0109] In the embodiments of the present application, the first image similarity of two images is low, indicating that the two images are not similar. In the case that all second images b in the image set B are not similar to the first image a i i i i i i

[0110] In some embodiments, steps 302-303 are an implementation in which the server screens the plurality of first images based on the image embeddings of the plurality of first images and the image embeddings of the plurality of second images to obtain the plurality of first target images.

[0111] In the embodiments of the present application, by pruning the first image with a high first pruning score from the plurality of first images, that is, removing the corresponding relationship between the first entity and the first image, the plurality of first target images corresponding to the first entity are obtained. Since the first pruning score is determined based on the first image similarity of the first image, the high or low of the first pruning score can indicate whether there is a similar second image of the first image in another knowledge graph, so that the plurality of first target images determined according to the first pruning score have high accuracy.

[0112] 304、The server screens the plurality of second images based on the image embeddings of the plurality of first images and the image embeddings of the plurality of second images to obtain a plurality of second target images, and the similarity of the second target images meets the similarity condition.

[0113] ​​​​​​​​​​​​​​​​​​In some embodiments, the implementation of step 304 is the same as the implementation of the server filtering the plurality of first images based on the image embeddings of the plurality of first images and the image embeddings of the plurality of second images to obtain the plurality of first target images, that is, referring to steps 302-303, which will not be repeated here.

[0114] In the embodiments of the present application, the knowledge graph alignment model in step 301 is trained based on the first knowledge graph and the second knowledge graph without pruning. After pruning the first knowledge graph and the second knowledge graph through steps 301-304, the server can further train the trained knowledge graph alignment model based on the plurality of first target images and the plurality of second target images, so as to obtain a knowledge graph alignment model with higher accuracy. Optionally, after filtering the plurality of first target images and the plurality of second target images, the knowledge graph alignment method provided by the embodiments of the present application further comprises the following steps: the server inputs the plurality of first target images and the plurality of second target images into the knowledge graph alignment model, and outputs the image embedding of each first target image and the image embedding of each second target image through the knowledge graph alignment model.

[0115] In the embodiments of the present application, the i-th iteration process of the knowledge graph alignment model refers to the iteration process of training the knowledge graph alignment model based on the plurality of first target images and the plurality of second target images. After obtaining the image embedding of the first target image and the image embedding of the second target image, the server performs the operation of step 305.

[0116] 305、The server takes the average image embedding of the plurality of first target images as the first image embedding of the first entity, and takes the average image embedding of the plurality of second target images as the second image embedding of the second entity.

[0117] Since each first entity corresponds to a plurality of first target images, the first image embedding of the first entity can be represented by the average image embedding of the plurality of first target images, and similarly, the second image embedding of the second entity can be represented by the average image embedding of the plurality of second target images.

[0118] In the embodiments of the present application, considering that there can be multiple images for each entity, the average image embedding of the plurality of first target images is taken as the first image embedding of the first entity. Since the first image embedding refers to the image embedding of each first target image, the first image embedding can intuitively reflect the image features of the first entity as a whole.

[0119] It should be noted that the number of first target images filtered can be 1, and in this case, the server directly takes the image embedding of the first target image as the first image embedding of the first entity, and at this time, the server does not need to calculate the average value.

[0120] It should be noted that the entity in the knowledge graph may not have a corresponding image, and in this case, the server sets the image embedding of the entity corresponding to the image to a 0 vector, which is equal in length to the image embedding of other entities.

[0121] For example, referring to Figure 4 The above figure is the alignment effect of the first knowledge graph KG1 and the second knowledge graph KG2. The left side is the entity node in KG1, and the right side is the entity node in KG2. Entity A2 in KG1 corresponds to 4 images, and entity A3 corresponds to 6 images. When entity A3 in KG1 and entity B4 in KG2 are aligned (i.e., indicating the same object in the real world), the corresponding image set can also find the corresponding "image to image" alignment mapping, and the image node a is aligned to an image node b, that is, the "alignment" (YES) label, and is not aligned to other image nodes, that is, the "non-alignment" (NO) label.

[0122] The purpose of pruning is to reduce the images in the image set of the entity that are useless for the alignment task. When a certain image s in the image set S1 of any entity in KG1 cannot find an aligned image in the image set S2 of multiple entities in KG2 (e.g., the pruning score is high), the image s is pruned, that is, the connection between the image s and the entity in the knowledge graph is removed. Referring to Figure 4 The following figure is the first knowledge graph and the second knowledge graph after pruning, wherein entity A1 in KG1 is pruned by 2 images, and entity B1 in KG2 is pruned by 4 images.

[0123] In some embodiments, after the server performs step 301, it continues to perform steps 302-304 to prune the first knowledge graph and the second knowledge graph, and then performs the operation of step 305. The pruning operation in this process can be regarded as a pre-pruning operation on the first knowledge graph and the second knowledge graph. The pre-pruning operation is to prune the two knowledge graphs before the knowledge graph alignment model is trained, and the pre-pruning operation utilizes the visual semantic information of the images of the entities. Therefore, when performing pre-pruning, the server does not consider other modal information in the knowledge graph, such as name information and relationship information, and therefore the pre-pruning operation may prune images that are beneficial to the alignment of the knowledge graph.

[0124] In some embodiments, after the server performs step 301, the server directly performs the operation of step 305, that is, the server does not perform the pre-pruning on the first knowledge graph and the second knowledge graph to ensure the integrity of the images of the entities in the knowledge graph. Then, step 305 can be replaced by: the server takes the average image embedding of the plurality of first images as the first image embedding of the first entity, and takes the average image embedding of the plurality of second images as the second image embedding of the second entity.

[0125] In the embodiments of the present application, the server continues to perform steps 302-304 after performing step 301 is taken as an example for illustration.

[0126] 306. The server determines an i-th image loss value based on the first image embedding of each first entity, the second image embedding of each second entity, and the annotation alignment result, where i is a positive integer not less than 1.

[0127] In some embodiments, the training target function of the image embedding determination model can be represented by the following formula three:

[0128] Formula three:

[0129] where f img is the training target function of the image embedding determination model, h (4) is the embedding representation of the entity in the image view, p is the image corresponding to the entity, c is the number of images corresponding to the entity, k is the serial number of the image, k = 1, 2, 3, …, c, FF represents the full connection layer in the image embedding determination model, and ResNet represents the feature extraction layer in the image embedding determination model. The training target of the model is to make f img tend to 0.

[0130] In some embodiments, the image loss value can be determined based on the training target function shown in formula three. The implementation of step 306 includes: the server determines an i-th image loss value based on the loss value of a target entity pair, where the target entity pair is an entity pair with an annotation alignment result, the target entity pair includes a first entity and a second entity, and the loss value of the target entity pair is determined based on the first image embedding of the first entity and the second image embedding of the second entity.

[0131] The loss value of each entity can be calculated by Formula Three. In some embodiments, the server determines the implementation of the ith image loss value based on the corresponding loss value of the target entity, including: the server processes the corresponding loss value of the target entity based on a target loss function to obtain the ith image loss value. The target loss function can be a logistic loss function or other loss function, which is not limited in the present application.

[0132] In the embodiments of the present application, the image loss value in the model training process is determined based on the loss value of the known target entity pair in the annotation alignment result, so that a more accurate image loss value can be obtained, and the accuracy of model training is improved.

[0133] 307. The server stops training the knowledge graph alignment model when the image loss value meets the loss value condition.

[0134] The image loss value meets the loss value condition, indicating that the knowledge graph alignment model has reached the target condition, that is, the model has been trained.

[0135] 308. The server filters a plurality of first target images of the first entity based on the first entity embedding of the first entity and the second entity embedding of the second entity to obtain at least one third target image when the image loss value does not meet the loss value condition.

[0136] The first entity embedding is determined based on the first image embedding, and the similarity of the third target image meets the similarity condition. The determination process of the first entity embedding and the second entity embedding is described in the embodiment of FIG. 3. Figure 5 In the embodiments of the present application, the image loss value does not meet the loss value condition, indicating that the knowledge graph alignment model has not reached the target condition, and the model still needs to be trained.

[0137] In some embodiments, the server filters the images of the first entity again, that is, filters the plurality of first target images, and performs an i+1th iteration process on the knowledge graph alignment model based on the filtered images. Alternatively, the server filters a plurality of first target images of the first entity based on the first entity embedding of the first entity and the second entity embedding of the second entity to obtain at least one third target image, including the following steps (1)-(3):

[0138] (1) For each first target image, the server determines the second image similarity between the image embedding of the first target image and the image embedding of each second target image, and takes the difference between 1 and the average of the plurality of second image similarities as the second pruning score of the first target image.

[0139] The second image similarity is used to represent the similarity between the first target image and the second target image. In some embodiments, the server determines the second image similarity between the image embedding of the first target image and the image embedding of each second target image. The implementation manner of step (1) is the same as that of step 302, which will not be repeated here.

[0140] (2) For each first target image, the server determines a first entity similarity between the first entity embedding and the second entity embedding of the second entity to which the corresponding second target image belongs.

[0141] The first entity similarity can be represented by the cosine similarity between the first entity embedding and the second entity embedding. Alternatively, the implementation manner of step (2) is the same as that of step 302, which will not be repeated here.

[0142] (3) The server prunes the first target image with a third pruning score higher than a pruning threshold from the plurality of first target images to obtain at least one third target image, the third pruning score being the product of the first entity similarity and a second pruning score.

[0143] The pruning threshold can be the same as or different from the pruning threshold in step 303, which is not limited in the present application. In some embodiments, steps (1)-(3) can be implemented by the following formula four:

[0144] Formula four:

[0145] wherein e a is the entity embedding of the entity corresponding to a i , e b is the entity embedding of the entity corresponding to b j , and sim(e a , e b ) is the cosine similarity between e a and e b . In step (3), prune can be the third pruning score; a i may be the i-th first target image, i is the serial number of the first target image, i = 1, 2, 3, …, m, and m is the number of first target images; B can be an image set of a plurality of second target images included in the second knowledge graph, b j may be a second target image in B, j is the serial number of the second target image, j = 1, 2, 3, …, n, and n is the number of second target images; sim(a i , b j ) can be the second image similarity of a i and b j , sim(a ib j ) is in the value interval (0, 1); e a may be the first entity embedding, b may be the second entity embedding, sim(e a , e b ) may be the first entity similarity between the first entity embedding and the second entity embedding.

[0146] In the embodiments of the present application, by pruning the image with a higher third pruning score in the plurality of first target images, that is, removing the correspondence between the first entity and the image, a plurality of third target images corresponding to the first entity are obtained. Since the third pruning score is determined based on the second image similarity of the first target image, the high and low of the third pruning score can represent whether the first target image has a similar second target image in another knowledge graph, so that the plurality of third target images determined according to the third pruning score have high accuracy.

[0147] 309. The server filters the plurality of second target images of the second entity based on the first entity embedding of each first entity and the second entity embedding of each second entity, and obtains at least one fourth target image.

[0148] The similarity of the fourth target image meets the similarity condition, and the second entity embedding is determined based on the second image embedding. In some embodiments, the implementation manner of step 309 is the same as that of step 308, and details are not repeated here. The pruning operations in steps 308 and 309 can be regarded as post-pruning of the first knowledge graph and the second knowledge graph. In the post-pruning operation, the server not only considers that the pruned image is not similar to all images in another knowledge graph in the image modality, but also refers to more other modality information in the knowledge graph, and the other modality information is introduced in the form of entity embedding.

[0149] 310. The server performs an i+1 iteration process on the knowledge graph alignment model based on the image loss value, the at least one third target image of the first entity, and the at least one fourth target image of the second entity, until the image loss value meets the loss value condition.

[0150] In some embodiments, the server adjusts the model parameters of the knowledge graph alignment model based on the image loss value to obtain adjusted model parameters; inputs the at least one third target image of the first entity and the at least one fourth target image of the second entity into the knowledge graph alignment model, and the knowledge graph alignment model outputs the image embedding of each third target image and the image embedding of each fourth target image based on the adjusted model parameters, and then performs the operations of steps 305-310.

[0151] In some embodiments, the server does not screen the plurality of first target images, that is, does not perform post-pruning on the first knowledge graph and the second knowledge graph, and steps 308-310 can be replaced by: in a case where the image loss value does not meet the loss value condition, the server performs an (i+1)th iteration process on the knowledge graph alignment model based on the image loss value, the plurality of first target images of the first entity, and the plurality of second target images of the second entity, until the image loss value meets the loss value condition.

[0152] In the embodiments of the present application, the server can only screen the plurality of first target images of the first entity and the plurality of second target images of the second entity in the first iteration process of the knowledge graph alignment model, and no longer screen in subsequent iteration processes; or the server can also screen the plurality of first target images of the first entity and the plurality of second target images of the second entity in each iteration process of the knowledge graph alignment model, which is not limited in the embodiments of the present application.

[0153] In some embodiments, steps 305-310 are an implementation of the server training the knowledge graph alignment model based on the first image embedding of each first entity, the second image embedding of each second entity, and the annotation alignment result.

[0154] In the embodiments of the present application, by screening the images corresponding to the first entity and the second entity again in the training process of the knowledge graph alignment model, the model can learn more concise data in the iteration process, reducing the redundancy of the training data, thereby improving the efficiency of model training.

[0155] In the embodiments of the present application, the performance of the knowledge graph alignment model trained by the technical personnel is compared with the performance of the knowledge graph alignment model trained by other methods in related technologies. For example, the baseline model for comparison is the MultiKE model (a multi-modal knowledge graph alignment model). Taking testing on five data sets respectively as an example, in the testing process, the knowledge graph alignment model provided in the embodiments of the present application is divided into three types: the knowledge graph alignment model 1 using pre-pruning, the knowledge graph alignment model 2 using post-pruning, and the knowledge graph alignment model 3 using pre-pruning and post-pruning. The evaluation index uses the accuracy of the model. The test results are shown in Table 1.

[0156] Table 1

[0157] Model Dataset 1 Dataset 2 Dataset 3 Dataset 4 Dataset 5 Baseline Model 0.793 0.762 0.761 0.985 0.994 Model 1 0.857 0.738 0.730 0.980 0.990 Model 2 0.895 0.742 0.730 0.986 0.996 Model 3 0.895 0.741 0.727 0.987 0.995

[0158] In the proposed knowledge graph alignment model provided by the embodiments of the present application, under the pre-pruning or post-pruning strategy, the alignment performance of the model is not inferior to or even better than that of the baseline model (such as dataset 1). When the pre-pruning and post-pruning strategies are used at the same time, the overall performance of the model has been improved to a certain extent, but the improvement compared with the post-pruning strategy is not obvious. However, the knowledge graph alignment model using the pre-pruning and post-pruning strategies has sufficient leading performance in the alignment task of the image set-oriented knowledge graph.

[0159] Through the technical solutions provided by the embodiments of the present application, after determining the image embeddings of the multiple images of the entity in the knowledge graph, the multiple images of the entity are filtered based on the image embeddings of the images, and the images with a similarity meeting the condition are retained, thereby achieving the purpose of removing images that are not beneficial to knowledge graph alignment. Therefore, when training the knowledge graph alignment model based on the filtered images and the labeled alignment results, the knowledge graph alignment model can learn more concise images, thereby enhancing the performance of the knowledge graph alignment model.

[0160] In the embodiments of the present application, the knowledge graph alignment model is a multi-modal knowledge graph alignment model, that is, the knowledge graph alignment model can determine the embedding representation of an entity from multiple dimensions such as the name, relationship, attribute and image of the entity, so as to determine a more accurate entity embedding according to the entity embedding representations of the multiple dimensions, thereby improving the accuracy of knowledge graph alignment. In some embodiments, in addition to the image embedding determination model, the knowledge graph alignment model also includes a name embedding determination model, a relationship embedding determination model and an attribute embedding determination model.

[0161] Figure 5 is a flowchart of a knowledge graph alignment method provided by an embodiment of the present application, referring to Figure 5 The embodiments of the present application are executed by a server, and the method comprises the following steps.

[0162] 501. The server determines, through the knowledge graph alignment model, the first name embedding, the first relationship embedding and the first attribute embedding of each first entity, and determines the second name embedding, the second relationship embedding and the second attribute embedding of each second entity.

[0163] In some embodiments, the implementation manner in which the server determines, through the knowledge graph alignment model, the first name embedding, the first relationship embedding and the first attribute embedding of each first entity comprises the following steps (1)-(3):

[0164] (1) The server determines, through the name embedding determination model, the first name embedding of each first entity.

[0165] The name embedding determination model is used to determine the name embedding of the input entity name. In this embodiment, the server pre-trains the name embedding determination model before training other models in the knowledge graph alignment model. In some embodiments, the server assigns a corresponding name embedding determination model to the entity name based on the text type of the entity name. The text type includes words or characters. Characters are in Chinese form; words are in a language other than Chinese. Accordingly, the name embedding determination model includes a word embedding model and a character embedding model. The implementation of step (1) includes: when the text type of the entity name is a word, the server determines the name embedding of the entity name through the word embedding model; when the text type of the entity name is a character, the server determines the name embedding of the entity name through the character embedding model, which is the following formula five:

[0166] Formula 5:

[0167] Where, l = (o1, o2, ..., o d ), where l is the input text sequence, o i Let i be the characters in text l, i = 1, 2, 3, ..., d, where d is the number of characters, LP is the name embedding, word_embed is the word embedding model, and char_embed is the character embedding model.

[0168] In some embodiments, the process by which the server determines the name embedding of an entity using a name embedding model is as shown in Formula Six below:

[0169] Formula Six:

[0170] Among them, h (1) This is an embedded representation of an entity in the name view, where h is the entity and name(h) is the entity's name. This is for name embedding. The name embedding is determined by Formula 5 above.

[0171] (2) The server determines the model through relation embedding and determines the first relation embedding of each first entity.

[0172] The knowledge graph includes the relational structure between entities. Relationships in the knowledge graph are represented as triples (h, r, t), where h is the head entity, t is the tail entity, and r is the relation between the head and tail entities. The input data for the relation embedding determination model is the triples in the knowledge graph. This application does not limit the type of relation embedding determination model; for example, the relation embedding determination model may be a TransE model.

[0173] In some embodiments, the training objective function of the relation embedding determination model can be expressed by the following formula:

[0174] Formula Seven: f rel (h (2) , r, t (2) ) = -||h (2) + r - t (2) ||.

[0175] where ||·|| is L1 or L2 norm, f rel is a training objective function of the relation embedding determination model, h (2) is an embedding representation of the head entity h in the relation view, and t (2) is an embedding representation of the tail entity t in the relation view. The training objective of the relation embedding determination model is to make f rel tend to 0 through the entity embedding and the relation embedding output by the model.

[0176] In some embodiments, the loss function of the relation embedding determination model can be constructed based on Formula Seven; for example, the server uses a sigmoid function (a kind of activation function) and a Logistic loss function to construct. In the embodiments of the present application, for a triple (h, r, t) in the knowledge graph G, if it is known that t is aligned with t' in the knowledge graph G', then (h, r, t') also has a great probability of appearing in the knowledge graph G'. Correspondingly, if it is known that h is aligned with h' in the knowledge graph G', then (h', r, t) also has a great probability of appearing in the knowledge graph G', then a loss function as shown in Formula Eight can be constructed in the relation view:

[0177] Formula Eight: L = ∑ (h，r，t)∈G，h≡h′ log(1 + exp(-f rel (h (2)′ , r, t (2) )) ) + ∑ (h，r，t)∈G，t≡t′ log(1 + exp(-f rel (h (2) , r, t (2)′ )))

[0178] where L is the loss function, exp is the exponential function with the natural constant e as the base, h (2)′ is an embedding representation of the head entity h' in the relation view, and t (2)′ is an embedding representation of the tail entity t' in the relation view. In the embodiments of the present application, the training objective of the relation embedding determination model can also be to maximize the probability that the triple (h, r, t') is true on the basis that f rel tends to 0.

[0179] (3) The server determines the first attribute embedding of each first entity through the attribute embedding determination model.

[0180] In some embodiments, the type of the attribute embedding determination model is not limited, for example, the attribute embedding determination model is a convolutional neural network model. The attribute of the entity includes an attribute item and an attribute value, for example, the attribute item is height, and the attribute value is 168 centimeters. In some embodiments, taking the attribute embedding determination model as a convolutional neural network model as an example, the implementation manner of step (3) can be implemented through the following formula nine:

[0181] Formula nine: CNN(a;v) = σ(vec(σ(a;v * Ω))W)

[0182] Wherein, a, v represent an attribute item-attribute value pair of an entity, W is a transformation matrix, Ω is a convolution kernel, and σ is an activation function. CNN is an attribute embedding determination model.

[0183] In some embodiments, the training objective function of the attribute embedding determination model can be represented by the following formula ten:

[0184] Formula ten: f attr (h (3) , a, v) = -||h (3) -CNN(a;v) ||

[0185] Wherein, f attr is the training objective function of the attribute embedding determination model, h (3) is the embedding representation of the entity in the attribute view. The training objective of the attribute embedding determination model is to make the output attribute embedding as close as possible to the embedding representation of the entity, that is, to make the value of f attr (h (3) , a, v) tend to 0.

[0186] 502. The server determines a first entity embedding of the first entity based on the first name embedding, the first relation embedding, the first attribute embedding, and the first image embedding.

[0187] Wherein, in the process of one iteration training of the knowledge graph alignment model, the server can obtain the embedding representation of an entity in multiple dimensions (h (1) , h (2) , h (3) , h (4) ). In some embodiments, the implementation manner of step 502 includes that the server performs weighted average on the first name embedding, the first relation embedding, the first attribute embedding, and the first image embedding to obtain the first entity embedding of the first entity.

[0188] The weights of the embedding representations in each dimension can be set as needed, and this application embodiment does not limit this. The first name embedding, the first relation embedding, the first attribute embedding, and the first image embedding are the embedding representations of the entity in the four dimensions, respectively. By weighting and averaging the embedding representations in the four dimensions, the first entity embedding can combine the embedding representations in each dimension, thereby approximating the true embedding representation of the entity as a whole, and thus improving the accuracy of entity embedding.

[0189] 503. The server determines the second entity embedding of the second entity based on the second name embedding, the second relation embedding, the second attribute embedding, and the second image embedding.

[0190] In some embodiments, step 503 is implemented in the same way as step 502, and will not be described again here.

[0191] In some embodiments, since the knowledge graph alignment model includes models corresponding to multiple dimensions, where the name embedding determination model is pre-trained, the relation embedding determination model, attribute embedding determination model, and image embedding determination model can be jointly trained during the training of the knowledge graph alignment model. For example, in one iteration, the relation embedding determination model is trained first, and after completing one iteration, the attribute embedding determination model begins training, and after completing one iteration, the image embedding determination model begins training, until the image embedding determination model completes the iteration. Each dimension's model updates its model parameters based on its respective loss value, and the next iteration is performed based on the updated model parameters.

[0192] In this embodiment, since the knowledge graph alignment model can determine the embedding representation of an entity from multiple dimensions such as entity name, relationship, attribute and image, a more accurate entity embedding can be determined based on the entity embedding representation from multiple dimensions, thereby improving the accuracy of knowledge graph alignment.

[0193] In this embodiment of the application, a knowledge graph alignment model trained by the first knowledge graph and the second knowledge graph is used to align entities in the first knowledge graph and the second knowledge graph. Figure 6 This is a flowchart of a knowledge graph alignment method provided in an embodiment of this application. See also... Figure 6 This application uses a server as the execution subject for illustration, and the method includes:

[0194] 601. The server inputs the entity information of the third entity in the first knowledge graph and the entity information of the fourth entity in the second knowledge graph into the knowledge graph alignment model. The entity information includes the entity name, relationship, attributes and multiple images.

[0195] The knowledge graph alignment model is trained by the knowledge graph alignment method provided in the embodiment shown in Figure 3 The knowledge graph alignment method provided in the embodiment shown in the figure is trained, and details are not repeated here. The third entity can be any entity in the first knowledge graph, and the number of the third entity is not limited by the present application; for example, the third entity is the first entity or other entity other than the first entity. Similarly, the fourth entity can be any entity in the second knowledge graph, and the number of the fourth entity is not limited by the present application.

[0196] 602、The server determines the second entity similarity of each entity pair through the knowledge graph alignment model, and the second entity similarity is determined based on the entity embedding of the third entity and the entity embedding of the fourth entity included in the entity pair.

[0197] Each entity pair includes a third entity and a fourth entity. The second entity similarity is used to represent the similarity of the third entity and the fourth entity in the entity pair.

[0198] 603、The server outputs the alignment result of the first knowledge graph and the second knowledge graph based on the second entity similarity, and the alignment result is used to represent whether the third entity and the fourth entity are aligned.

[0199] In the embodiment of the present application, since the knowledge graph alignment model is trained based on the first knowledge graph and the second knowledge graph, the alignment of the entities in the first knowledge graph and the second knowledge graph is performed by means of the knowledge graph alignment model, so that the alignment result is more accurate, thereby improving the accuracy of the knowledge graph alignment.

[0200] In the embodiment of the present application, the knowledge graph alignment model is trained offline on the server side, and after the training is completed, the knowledge graph alignment model is deployed on the server side. Therefore, the user sends the identifiers of the two knowledge graphs to be aligned through the terminal, and the server calls the knowledge graph alignment model to align the two knowledge graphs. Alternatively, the server regularly aligns two or more knowledge graphs. Figure 7 is a flowchart of a knowledge graph alignment method provided in an embodiment of the present application, see Figure 7 The embodiment of the present application takes the data interaction between the terminal and the server as an example for illustration, and the method comprises the following steps:

[0201] 701、The terminal sends a graph alignment request to the server, and the graph alignment request carries the identifier of the first knowledge graph and the identifier of the second knowledge graph.

[0202] When the user wants to align the first knowledge graph and the second knowledge graph, the user obtains the identifiers of the first knowledge graph and the second knowledge graph through the terminal, and triggers the terminal to send the graph alignment request to the server.

[0203] 702、The server receives the atlas alignment request, obtains entity information of a third entity in the first knowledge graph based on the identifier of the first knowledge graph, and obtains entity information of a fourth entity in the second knowledge graph based on the identifier of the second knowledge graph. The entity information includes entity name, relationship, attribute, and multiple images.

[0204] In some embodiments, the knowledge graphs are stored in a database. Accordingly, after receiving the atlas alignment request, the server obtains the knowledge graph corresponding to each identifier from the database based on the multiple identifiers carried by the atlas alignment request.

[0205] 703、The server inputs the entity information of the third entity and the entity information of the fourth entity into a knowledge graph alignment model.

[0206] The knowledge graph alignment model is trained by the knowledge graph alignment method provided by the embodiment shown in the figure, and will not be described here. Figure 3

[0207] 704、The server determines the second entity similarity of each entity pair through the knowledge graph alignment model. The second entity similarity is determined based on the entity embedding of the third entity and the entity embedding of the fourth entity included in the entity pair.

[0208] The server determines the entity embedding of the third entity and the entity embedding of the fourth entity through the knowledge graph alignment model, and determines the second entity similarity based on the entity embedding of the third entity and the entity embedding of the fourth entity.

[0209] 705、The server outputs the alignment result of the first knowledge graph and the second knowledge graph based on the second entity similarity. The alignment result is used to indicate whether the third entity and the fourth entity are aligned.

[0210] The second entity similarity of the entity pair is higher, indicating that the third entity and the fourth entity in the entity pair are more similar. Therefore, it is indicated that the third entity and the fourth entity have a greater probability of being aligned, that is, indicating the same object in the real world. In some embodiments, the implementation manner of the server for outputting the alignment result of the entity pair based on the second entity similarity includes: the server outputs the alignment result in the case that the second entity similarity is higher than a similarity threshold. The alignment result is used to indicate that the third entity and the fourth entity are aligned. In the case that the second entity similarity is not higher than the similarity threshold, the server outputs the alignment result. The alignment result is used to indicate that the third entity and the fourth entity are not aligned.

[0211] 706、The server sends the alignment result of the first knowledge graph and the second knowledge graph to the terminal.

[0212] ​In some embodiments, the alignment result of the first knowledge graph and the second knowledge graph includes an entity pair in which the third entity is aligned with the fourth entity. In this way, the server only needs to send the alignable entity pair to the terminal.

[0213] 707、The terminal receives the alignment result and displays the alignment result.

[0214] After the terminal receives the alignment result, the alignment result is displayed in the display interface, so that the user can view the alignment result through the terminal.

[0215] In the embodiments of the present application, since the knowledge graph alignment model is trained based on the first knowledge graph and the second knowledge graph, the alignment of the entities in the first knowledge graph and the second knowledge graph is performed by means of the knowledge graph alignment model, so that the alignment result is more accurate, thereby improving the accuracy of knowledge graph alignment.

[0216] Figure 8 is a structural schematic diagram of a knowledge graph alignment device provided by the embodiments of the present application, referring to Figure 8 , the device comprises:

[0217] The first determination module 801 is configured to determine image embeddings of a plurality of first images of a first entity in a first knowledge graph and determine image embeddings of a plurality of second images of a second entity in a second knowledge graph by using a knowledge graph alignment model.

[0218] The screening module 802 is configured to screen the plurality of first images based on the image embeddings of the plurality of first images and the image embeddings of the plurality of second images, to obtain a plurality of first target images, and screen the plurality of second images to obtain a plurality of second target images, the similarity of the plurality of first target images meeting a similarity condition, and the similarity of the plurality of second target images meeting the similarity condition.

[0219] The training module 803 is configured to train the knowledge graph alignment model based on the plurality of first target images of the first entity, the plurality of second target images of the second entity, and a labeled alignment result, the labeled alignment result being used to indicate whether the first entity and the second entity are aligned.

[0220] The knowledge graph alignment model is configured to output an alignment result of entities in two input knowledge graphs based on images of the entities in the two knowledge graphs.

[0221] In some embodiments, the screening module 802 is configured to, for each first image, determine a first image similarity between the image embedding of the first image and the image embedding of each second image; and from the plurality of first images, prune a first image whose first pruning score is higher than a pruning threshold to obtain the plurality of first target images, the first pruning score being a difference between 1 and a mean value of the plurality of first image similarities.

[0222] In some embodiments, the training module 803 comprises:

[0223] a determining unit configured to embed an average image of the plurality of first target images as a first image embedding of the first entity, and embed an average image of the at least one second target image as a second image embedding of the second entity;

[0224] a training unit configured to train the knowledge graph alignment model based on the first image embedding of the first entity, the second image embedding of the second entity, and the annotation alignment result.

[0225] In some embodiments, the training unit is configured to determine an i-th image loss value based on the first image embedding of the first entity, the second image embedding of the second entity, and the annotation alignment result, where i is a positive integer not less than 1.

[0226] In a case where the image loss value meets a loss value condition, stop training the knowledge graph alignment model.

[0227] In a case where the image loss value does not meet the loss value condition, perform screening on the plurality of first target images of the first entity based on a first entity embedding of the first entity and a second entity embedding of the second entity to obtain at least one third target image, and perform screening on the plurality of second target images of the second entity to obtain at least one fourth target image, where the similarity of the third target image and the fourth target image both meet a similarity condition, the first entity embedding is determined based on the first image embedding, and the second entity embedding is determined based on the second image embedding.

[0228] perform i+1-th training on the knowledge graph alignment model based on the image loss value, the at least one third target image of the first entity, and the at least one fourth target image of the second entity, until the image loss value meets the loss value condition.

[0229] In some embodiments, the training unit is configured to determine the i-th image loss value based on a loss value of a target entity pair, where the target entity pair is an entity pair with an annotation alignment result, the target entity pair includes the first entity and the second entity, and the loss value is determined based on the first image embedding of the first entity and the second image embedding of the second entity.

[0230] In some embodiments, the training unit is configured to, for each first target image, determine a second image similarity between the image embedding of the first target image and the image embedding of each second target image, and determine a second pruning score of the first target image as a difference between 1 and a mean value of a plurality of second image similarities.

[0231] For each first target image, determine a first entity similarity between the first entity embedding and a second entity embedding of a second entity to which the corresponding second target image belongs;

[0232] From the plurality of first target images, prune the first target image whose third pruning score is higher than a pruning threshold to obtain at least one third target image, the third pruning score being a product of the first entity similarity and a second pruning score.

[0233] In some embodiments, the apparatus further comprises:

[0234] The third determination module is configured to determine, by the knowledge graph alignment model, the first name embedding, the first relationship embedding and the first attribute embedding of the first entity, and determine the second name embedding, the second relationship embedding and the second attribute embedding of the second entity; determine the first entity embedding of the first entity based on the first name embedding, the first relationship embedding, the first attribute embedding and the first image embedding; and determine the second entity embedding of the second entity based on the second name embedding, the second relationship embedding, the second attribute embedding and the second image embedding.

[0235] In some embodiments, the third determination module is configured to perform weighted average on the first name embedding, the first relationship embedding, the first attribute embedding and the first image embedding to obtain the first entity embedding of the first entity.

[0236] In some embodiments, the knowledge graph alignment model comprises a name embedding determination model, a relationship embedding determination model and an attribute embedding determination model.

[0237] The third determination module is configured to determine, by the name embedding determination model, the first name embedding of the first entity; determine, by the relationship embedding determination model, the first relationship embedding of the first entity; and determine, by the attribute embedding determination model, the first attribute embedding of the first entity.

[0238] Through the technical scheme provided by the embodiments of the present application, after the image embedding of the plurality of images of the entity in the knowledge graph is determined, the plurality of images of the entity are screened based on the image embedding of the image, and the images with similar degrees meeting the conditions are retained, thereby achieving the purpose of removing the images that are not beneficial to the knowledge graph alignment. Therefore, when the knowledge graph alignment model is trained based on the screened images and the labeled alignment results, the knowledge graph alignment model can learn more concise images, thereby enhancing the performance of the knowledge graph alignment model.

[0239] Figure 9 is a structural schematic diagram of a knowledge graph alignment apparatus provided by an embodiment of the present application, referring to Figure 9 The apparatus comprises:

[0240] The input module 901 is configured to input entity information of the third entity in the first knowledge graph and entity information of the fourth entity in the second knowledge graph into the knowledge graph alignment model, where the entity information includes an entity name, a relationship, an attribute, and a plurality of images.

[0241] The second determination module 902 is configured to determine, by the knowledge graph alignment model, a second entity similarity of each entity pair, where the second entity similarity is determined based on entity embeddings of the third entity and the fourth entity included in the entity pair.

[0242] The output module 903 is configured to output an alignment result of the first knowledge graph and the second knowledge graph based on the second entity similarity, where the alignment result is used to indicate whether the third entity and the fourth entity are aligned.

[0243] The knowledge graph alignment model is obtained by training the knowledge graph alignment method.

[0244] In the embodiments of the present application, since the knowledge graph alignment model is trained based on the first knowledge graph and the second knowledge graph, the alignment of the entities in the first knowledge graph and the second knowledge graph by means of the knowledge graph alignment model makes the alignment result more accurate, thereby improving the accuracy of knowledge graph alignment.

[0245] It should be noted that the knowledge graph alignment apparatus provided in the above embodiments is only used as an example for the division of the above functional modules in the knowledge graph alignment, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the above-described functions. In addition, the apparatus and method for aligning the knowledge graph provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be repeated here.

[0246] The embodiments of the present application provide a computer device, which includes one or more processors and one or more memories, and the one or more memories store at least one computer program, which is loaded and executed by the one or more processors to implement the knowledge graph alignment method. In some embodiments, the computer device is provided as a terminal or a server, and the structure of the terminal will be introduced first as follows:

[0247] Figure 10 is a structural schematic diagram of a terminal provided in the embodiments of the present application. The terminal 1000 can be: a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart watch, etc., but is not limited thereto.

[0248] Generally, the terminal 1000 includes one or more processors 1011 and one or more memories 1002.

[0249] The processor 1011 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1011 can be implemented in the form of at least one of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1011 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1011 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content to be displayed by the display screen. In some embodiments, the processor 1011 can further include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0250] The memory 1002 can include one or more computer-readable storage media that can be non-transitory. The memory 1002 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one computer program for being executed by the processor 1011 to implement the knowledge graph alignment method provided by the method embodiments in the present application.

[0251] In some embodiments, the terminal 1000 can also optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1011, the memory 1002, and the peripheral device interface 1003 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1003 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, a positioning assembly 1008, and a power supply 1009.

[0252] The peripheral interface 1003 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 1011 and the memory 1002. In some embodiments, the processor 1011, the memory 1002 and the peripheral interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1011, the memory 1002 and the peripheral interface 1003 can be implemented on a separate chip or circuit board, and the present embodiments are not limited to this.

[0253] The radio frequency circuit 1004 is used to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1004 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like.

[0254] The display screen 1005 is used to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on or above the surface of the display screen 1005. The touch signals can be input as control signals to the processor 1011 for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards.

[0255] The camera assembly 1006 is used to collect images or videos. Optionally, the camera assembly 1006 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is disposed on the front panel of the terminal, and the rear-facing camera is disposed on the back of the terminal.

[0256] The audio circuit 1007 can include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals input to the processor 1011 for processing, or input to the radio frequency circuit 1004 to realize voice communication.

[0257] The positioning assembly 1008 is used to locate the current geographical position of the terminal 1000 to realize navigation or LBS (Location Based Service).

[0258] The power supply 1009 is configured to supply power to each component of the terminal 1000. The power supply 1009 can be AC, DC, a disposable battery, or a rechargeable battery.

[0259] In some embodiments, the terminal 1000 further includes one or more sensors 1100. The one or more sensors 1100 include, but are not limited to, an acceleration sensor 1011, a gyroscope sensor 1012, a pressure sensor 1013, a fingerprint sensor 1014, an optical sensor 1015, and a proximity sensor 1016.

[0260] The acceleration sensor 1011 can detect the acceleration magnitude in three coordinate axes of a coordinate system established by the terminal 1000.

[0261] The gyroscope sensor 1012 can detect the body orientation and rotation angle of the terminal 1000. The gyroscope sensor 1012 can cooperate with the acceleration sensor 1011 to collect the 3D motion of the user with respect to the terminal 1000.

[0262] The pressure sensor 1013 can be disposed on the side frame of the terminal 1000 and / or the lower layer of the display screen 1005. When the pressure sensor 1013 is disposed on the side frame of the terminal 1000, the pressure sensor 1013 can detect the holding signal of the user with respect to the terminal 1000. The processor 1011 can perform left-hand or right-hand recognition or a shortcut operation based on the holding signal collected by the pressure sensor 1013. When the pressure sensor 1013 is disposed on the lower layer of the display screen 1005, the processor 1011 can control the operable control on the UI interface based on the pressure operation of the user with respect to the display screen 1005.

[0263] The fingerprint sensor 1014 is configured to collect the fingerprint of the user. The processor 1011 can identify the identity of the user based on the fingerprint collected by the fingerprint sensor 1014, or the fingerprint sensor 1014 can identify the identity of the user based on the collected fingerprint.

[0264] The optical sensor 1015 is configured to collect the ambient light intensity. In an embodiment, the processor 1011 can control the display brightness of the display screen 1005 based on the ambient light intensity collected by the optical sensor 1015.

[0265] The proximity sensor 1016 is configured to collect the distance between the user and the front of the terminal 1000.

[0266] Those skilled in the art can understand that Figure 10 The structure shown in the above description does not constitute a limitation on the terminal 1000, and can include more or fewer components than those shown in the figure, or combine certain components, or use different component arrangements.

[0267] The computer device can also be provided as a server, and the structure of the server is introduced as follows:

[0268] Figure 11 FIG. 11 is a structural diagram of a server provided by an embodiment of the present application. The server 1100 can have great differences due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 1101 and one or more memories 1102. The one or more memories 1102 store at least one computer program, which is loaded and executed by the one or more processors 1101 to implement the method for aligning knowledge graphs provided by each of the above methods. Of course, the server 1100 can also have a wired or wireless network interface, a keyboard, an input and output interface, and other components for realizing the functions of the device, and thus input and output can be performed. The server 1100 can also include other components for realizing the functions of the device, which are not described herein.

[0269] In an example embodiment, a computer readable storage medium, for example, a memory including a computer program, is also provided. The computer program can be executed by a processor to complete the method for aligning knowledge graphs in the above embodiments. For example, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0270] In an example embodiment, a computer program product is also provided. The computer program product includes a computer program stored in a computer readable storage medium. A processor of a computer device reads the computer program from the computer readable storage medium, and the processor executes the computer program to enable the computer device to perform the above method for aligning knowledge graphs.

[0271] In some embodiments, the computer program related to the embodiments of the present application can be deployed to execute on one computer device, or on multiple computer devices located in one place, or on multiple computer devices distributed in multiple places and interconnected through a communication network. The multiple computer devices distributed in multiple places and interconnected through a communication network can constitute a blockchain system.

[0272] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by a program to complete the related hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0273] The above is only an optional embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A knowledge graph alignment method, characterized in that, The method includes: Using a knowledge graph alignment model, the image embeddings of multiple first images of a first entity in a first knowledge graph are determined, and the image embeddings of multiple second images of a second entity in a second knowledge graph are determined. For each first image, a first image similarity is determined between the image embedding of the first image and the image embedding of each second image; from the plurality of first images, the first images with a first pruning score higher than the pruning threshold are pruned to obtain a plurality of first target images; if it is determined based on the plurality of first image similarities that there is an image similar to the first image among the plurality of second images, then the first pruning score is lower than the pruning threshold. For each second image, a second image similarity is determined between the image embedding of the second image and the image embedding of each first image; from the plurality of second images, the second images with a second pruning score higher than the pruning threshold are pruned to obtain a plurality of second target images; if it is determined based on the plurality of second image similarities that there is an image similar to the second image among the plurality of first images, then the second pruning score is lower than the pruning threshold. Based on the plurality of first target images of the first entity, the plurality of second target images of the second entity, and the annotation alignment results, the knowledge graph alignment model is trained, wherein the annotation alignment results are used to indicate whether the first entity and the second entity are aligned; The knowledge graph alignment model is used to output the alignment result of entities in two knowledge graphs based on the images of entities in the two input knowledge graphs.

2. The method according to claim 1, characterized in that, The first pruning score is the difference between 1 and the mean of the similarity scores of multiple first images, and the second pruning score is the difference between 1 and the mean of the similarity scores of multiple second images.

3. The method according to claim 1, characterized in that, The step of training the knowledge graph alignment model based on the plurality of first target images of the first entity, the plurality of second target images of the second entity, and the annotation alignment results includes: The average image embedding of the plurality of first target images is used as the first image embedding of the first entity, and the average image embedding of at least one second target image is used as the second image embedding of the second entity; The knowledge graph alignment model is trained based on the first image embedding of the first entity, the second image embedding of the second entity, and the annotation alignment result.

4. The method according to claim 3, characterized in that, The step of training the knowledge graph alignment model based on the first image embedding of the first entity, the second image embedding of the second entity, and the annotation alignment result includes: Based on the first image embedding of the first entity, the second image embedding of the second entity, and the annotation alignment result, the i-th image loss value is determined, where i is a positive integer not less than 1; If the image loss value meets the loss value condition, stop training the knowledge graph alignment model; If the image loss value does not meet the loss value condition, based on the first entity embedding of the first entity and the second entity embedding of the second entity, multiple first target images of the first entity are filtered to obtain at least one third target image, and multiple second target images of the second entity are filtered to obtain at least one fourth target image, wherein the similarity of the third target image and the fourth target image both meet the similarity condition, the first entity embedding is determined based on the first image embedding, and the second entity embedding is determined based on the second image embedding; Based on the image loss value, the at least one third target image of the first entity, and the at least one fourth target image of the second entity, the knowledge graph alignment model is trained for the (i+1)th time until the image loss value meets the loss value condition.

5. The method according to claim 4, characterized in that, The determination of the i-th image loss value based on the first image embedding of the first entity, the second image embedding of the second entity, and the annotation alignment result includes: The i-th image loss value is determined based on the loss value corresponding to the target entity pair; The target entity pair is an entity pair with an annotation alignment result, and the target entity pair includes the first entity and the second entity. The loss value is determined based on the first image embedding of the first entity and the second image embedding of the second entity.

6. The method according to claim 4, characterized in that, The step of filtering multiple first target images of the first entity based on the first entity embedding and the second entity embedding of the second entity to obtain at least one third target image includes: For each first target image, a second image similarity is determined between the image embedding of the first target image and the image embedding of each second target image, and the difference between 1 and the mean of multiple second image similarities is used as the second pruning score of the first target image; For each first target image, determine the first entity similarity between the first entity embedding and the second entity embedding of the second entity to which the corresponding second target image belongs; From the plurality of first target images, the first target images with a third pruning score higher than the pruning threshold are pruned to obtain at least one third target image, wherein the third pruning score is the product of the first entity similarity and the second pruning score.

7. The method according to claim 4, characterized in that, The method further includes: Using the knowledge graph alignment model, the first name embedding, the first relation embedding, and the first attribute embedding of the first entity are determined, and the second name embedding, the second relation embedding, and the second attribute embedding of the second entity are determined. Based on the first name embedding, the first relation embedding, the first attribute embedding, and the first image embedding, the first entity embedding of the first entity is determined; Based on the second name embedding, the second relation embedding, the second attribute embedding, and the second image embedding, the second entity embedding of the second entity is determined.

8. The method according to claim 7, characterized in that, The step of determining the first entity embedding of the first entity based on the first name embedding, the first relation embedding, the first attribute embedding, and the first image embedding includes: The first entity embedding of the first entity is obtained by weighting the first name embedding, the first relationship embedding, the first attribute embedding, and the first image embedding.

9. The method according to claim 7, characterized in that, The knowledge graph alignment model includes a name embedding determination model, a relation embedding determination model, and an attribute embedding determination model; The step of determining the first name embedding, first relation embedding, and first attribute embedding of the first entity through the knowledge graph alignment model includes: The name embedding model is determined by the name embedding to determine the first name embedding of the first entity; The first relation embedding of the first entity is determined by the relation embedding model. The model is determined by the attribute embedding, and the first attribute embedding of the first entity is determined.

10. A knowledge graph alignment method, characterized in that, The method includes: The entity information of the third entity in the first knowledge graph and the entity information of the fourth entity in the second knowledge graph are input into the knowledge graph alignment model. The entity information includes entity name, relationship, attribute and multiple images. The knowledge graph alignment model is used to determine the second entity similarity of each entity pair. The second entity similarity is determined based on the entity embedding of the third entity included in the entity pair and the entity embedding of the fourth entity. Based on the second entity similarity, the alignment result of the first knowledge graph and the second knowledge graph is output, and the alignment result is used to indicate whether the third entity and the fourth entity are aligned; The knowledge graph alignment model is obtained by training the knowledge graph alignment method described in claims 1-9.

11. A knowledge graph alignment device, characterized in that, The device includes: The first determining module is used to determine the image embeddings of multiple first images of a first entity in a first knowledge graph through a knowledge graph alignment model, and to determine the image embeddings of multiple second images of a second entity in a second knowledge graph. The filtering module is used to determine, for each first image, a first image similarity between the image embedding of the first image and the image embedding of each second image; and to prune the first images with a first pruning score higher than a pruning threshold from the plurality of first images to obtain a plurality of first target images. If, based on the similarity of the plurality of first images, it is determined that there is an image similar to the first image among the plurality of second images, then the first pruning score is lower than the pruning threshold. The filtering module is further configured to, for each second image, determine a second image similarity between the image embedding of the second image and the image embedding of each first image; prune the second images from the plurality of second images whose second pruning scores are higher than the pruning threshold to obtain a plurality of second target images; if, based on the similarity of the plurality of second images, it is determined that there is an image similar to the second image among the plurality of first images, then the second pruning score is lower than the pruning threshold. The training module is used to train the knowledge graph alignment model based on the plurality of first target images of the first entity, the plurality of second target images of the second entity, and the annotation alignment results, wherein the annotation alignment results are used to indicate whether the first entity and the second entity are aligned. The knowledge graph alignment model is used to output the alignment result of entities in two knowledge graphs based on the images of entities in the two input knowledge graphs.

12. A knowledge graph alignment device, characterized in that, The device includes: The input module is used to input the entity information of the third entity in the first knowledge graph and the entity information of the fourth entity in the second knowledge graph into the knowledge graph alignment model. The entity information includes entity name, relationship, attribute and multiple images. The second determining module is used to determine the second entity similarity of each entity pair through the knowledge graph alignment model. The second entity similarity is determined based on the entity embedding of the third entity included in the entity pair and the entity embedding of the fourth entity. The output module is used to output the alignment result between the first knowledge graph and the second knowledge graph based on the second entity similarity, wherein the alignment result is used to indicate whether the third entity and the fourth entity are aligned; The knowledge graph alignment model is obtained by training the knowledge graph alignment method described in claims 1-9.

13. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the knowledge graph alignment method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the knowledge graph alignment method as described in any one of claims 1 to 10.

15. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, a processor of a computer device reading the computer program from the computer-readable storage medium, and the processor executing the computer program to cause the computer device to perform the knowledge graph alignment method as described in any one of claims 1 to 10.

Images