Entity alignment method, device and equipment for knowledge graph

Abstract

The application discloses a kind of entity alignment method, device and equipment of knowledge graph, it is related to knowledge graph technical field, comprising: if benchmark entity does not have benchmark image in target knowledge graph, benchmark mixed feature is generated according to benchmark semantic feature and benchmark topological structure feature;The benchmark mixed feature is processed by variational autoencoder, and the probability distribution parameter of benchmark entity in each latent feature dimension is obtained;Probability distribution parameter is input to image feature prediction model and is reparameterized, and the predicted image feature corresponding to benchmark entity is obtained as benchmark image feature;According to benchmark image feature and the image feature to be aligned of entity to be aligned, entity alignment is carried out to benchmark entity and entity to be aligned.The application realizes when knowledge graph is missing entity image, the effect of predicting entity image feature using entity semantic feature and topological structure feature, improves the accuracy of entity feature identification, further improves entity alignment precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge graph, and particularly relates to a knowledge graph entity alignment method, device and equipment. BACKGROUND

[0002] As a core task of knowledge graph fusion, entity alignment aims to identify entities in heterogeneous knowledge graphs that point to the same real world, and plays a key role in improving knowledge integrity and sharing. With the rapid growth of multi-modal data, more rich clues are provided for entity alignment.

[0003] However, the existing multi-modal knowledge graph generally has the problem of missing entity images, which undoubtedly leads to inaccurate entity feature identification and directly affects the accuracy of entity alignment. SUMMARY

[0004] The present application provides a knowledge graph entity alignment method, device and equipment to solve the problem of missing entity images in knowledge graphs affecting entity alignment accuracy.

[0005] According to an aspect of the present application, a knowledge graph entity alignment method is provided, which comprises:

[0006] Obtaining a target knowledge graph and other knowledge graphs except the target knowledge graph, and determining a reference entity from the target knowledge graph, and determining an entity to be aligned from the other knowledge graphs;

[0007] If the reference entity does not have a reference image in the target knowledge graph, obtaining the reference semantic feature and the reference topological structure feature corresponding to the reference entity in the target knowledge graph, and performing feature splicing according to the reference semantic feature and the reference topological structure feature to generate a reference mixed feature;

[0008] Processing the reference mixed feature by a variational autoencoder to obtain probability distribution parameters of the reference entity in each latent feature dimension; wherein the probability distribution parameters include mean information and variance information;

[0009] Inputting the probability distribution parameters into an image feature prediction model, reparameterizing the probability distribution parameters using model parameters of the image feature prediction model to obtain predicted image features corresponding to the reference entity as reference image features corresponding to the reference entity in the target knowledge graph;

[0010] According to the reference image features and the image features corresponding to the entity to be aligned in the other knowledge graphs, performing entity alignment on the reference entity and the entity to be aligned.

[0011] According to another aspect of the present application, there is provided an entity alignment apparatus of a knowledge graph, the apparatus comprising:

[0012] an entity determination module configured to obtain a target knowledge graph and other knowledge graphs other than the target knowledge graph, determine a reference entity from the target knowledge graph, and determine an entity to be aligned from the other knowledge graphs;

[0013] a hybrid feature generation module configured to, if the reference entity does not have a reference image in the target knowledge graph, obtain a reference semantic feature and a reference topological structure feature corresponding to the reference entity in the target knowledge graph, and perform feature splicing according to the reference semantic feature and the reference topological structure feature to generate a reference hybrid feature;

[0014] a feature encoding module configured to process the reference hybrid feature by a variational autoencoder to obtain a probability distribution parameter of the reference entity in each latent feature dimension; wherein the probability distribution parameter comprises mean information and variance information;

[0015] an image feature prediction module configured to input the probability distribution parameter into an image feature prediction model, reparameterize the probability distribution parameter by using model parameters of the image feature prediction model, and obtain a predicted image feature corresponding to the reference entity as a reference image feature corresponding to the reference entity in the target knowledge graph;

[0016] an entity alignment module configured to perform entity alignment of the reference entity and the entity to be aligned according to the reference image feature and an image feature corresponding to the entity to be aligned in the other knowledge graphs.

[0017] According to another aspect of the present application, there is provided an electronic device, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory connected in communication with the at least one processor; wherein

[0020] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the entity alignment method of the knowledge graph according to any one of the present application.

[0021] The application has beneficial effects in that when the knowledge graph lacks the image of an entity, the semantic features and the topological structure features of the entity are used to predict the image features of the entity, which can improve the accuracy of entity feature identification and further improve the precision of entity alignment.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0024] Figure 1 A flowchart of an entity alignment method of a knowledge graph provided for the first embodiment of the application;

[0025] Figure 2 A flowchart of an entity alignment method of a knowledge graph provided for the second embodiment of the application;

[0026] Figure 3 A flowchart of an entity alignment method of a knowledge graph provided for the third embodiment of the application;

[0027] Figure 4 A structural schematic diagram of an entity alignment device of a knowledge graph provided for the fourth embodiment of the application;

[0028] Figure 5 A structural schematic diagram of an electronic device implementing the entity alignment method of a knowledge graph of the application. DETAILED DESCRIPTION

[0029] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application, so that those skilled in the art can better understand the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "target", "other", "reference", "to be aligned", "first", "second", "third", "fourth" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment one

[0032] Figure 1 A flowchart of an entity alignment method of a knowledge graph provided by the first embodiment of the present application, the present embodiment can be applicable to the case where the image of the missing entity of the knowledge graph is predicted to obtain the image feature of the entity by using the semantic feature and the topological structure feature of the entity, for entity alignment. The method can be executed by a knowledge graph entity alignment device, which can be realized in the form of hardware and / or software, such as a computer, etc. As shown in the figure, the method comprises: Figure 1

[0033] S101, obtaining a target knowledge graph and other knowledge graphs except the target knowledge graph, determining a reference entity from the target knowledge graph, and determining an entity to be aligned from the other knowledge graphs.

[0034] ​The target knowledge graph refers to a reference benchmark in the entity alignment task and contains a core knowledge system that needs to be enhanced or verified. The other knowledge graphs serve as auxiliary data sources and need to be aligned with the target graph to expand the semantic network. It can be understood that the target knowledge graph and the other knowledge graphs are not specific to one or several knowledge graphs, but refer to the knowledge graphs relied on when performing the entity alignment task. For example, if the entity alignment is performed on "knowledge graph 1" and "knowledge graph 2", then "knowledge graph 1" can be the target knowledge graph and "knowledge graph 2" can be the other knowledge graph; or "knowledge graph 2" can be the target knowledge graph and "knowledge graph 1" can be the other knowledge graph.

[0035] The knowledge graph is a graph structure-based data model used to describe entities, concepts and their mutual relationships in the real world. In this embodiment, the target knowledge graph and the other knowledge graphs are all multi-modal knowledge graphs, which are knowledge representation systems that integrate multiple modal data including text, images and the like, and achieve comprehensive cognition of entities and relationships by associating different modal information.

[0036] The benchmark entity is a selected core entity in the target knowledge graph, serving as a reference benchmark for entity alignment. It can be understood that when the entity alignment is performed on one or several entities in the target knowledge graph, the benchmark entity can be one or several entities in the target knowledge graph; when the entity alignment is performed on all entities in the target knowledge graph, the benchmark entity refers to the set of all entities in the target knowledge graph.

[0037] The entity to be aligned refers to a candidate entity in the other knowledge graph that needs to be matched with the benchmark entity in the target knowledge graph. It is an entity in the other knowledge graph that is in an incomplete matching state and needs to be verified for equivalence with the benchmark entity through entity alignment technology. It can be understood that the entity to be aligned can be one or several entities in the other knowledge graph, or it can refer to all entities in the other knowledge graph.

[0038] In one embodiment, a centralized knowledge graph registration library is maintained, a unique identifier of the target knowledge graph is directly specified, the target graph is retrieved and loaded in the knowledge graph registration library according to the identifier, and the other knowledge graphs are automatically obtained by screening all entries in the knowledge graph registration library except the target identifier.

[0039] Further, in one embodiment, all entities in the target knowledge graph are traversed, and all entities in the target knowledge graph are taken as benchmark entities.

[0040] In another implementation, by defining the core attributes of the entities in the target knowledge graph, entities that meet specific business or technical standards are screened out as reference entities.

[0041] Further, in an implementation, all entities in the other knowledge graph are traversed, and all entities in the other knowledge graph are taken as the entities to be aligned.

[0042] Further, in another implementation, by defining the core attributes of the entities in the other knowledge graph, entities that meet specific business or technical standards are screened out as the entities to be aligned.

[0043] S102, if the reference entity does not have a reference image in the target knowledge graph, the reference semantic features and the reference topological structure features corresponding to the reference entity in the target knowledge graph are obtained, and feature splicing is performed according to the reference semantic features and the reference topological structure features to generate reference mixed features.

[0044] The reference image refers to a standardized visual representation of the reference entity in the target knowledge graph, that is, a representative image that is explicitly bound to the reference entity in the target knowledge graph, for example, the reference image of the "XX tower" entity in the target knowledge graph is a landmark panoramic photo.

[0045] The reference semantic features refer to the abstract semantic information contained in the reference entity in the target knowledge graph. These features capture the core attributes, category attribution, and semantic association with other entities of the reference entity through vectorization.

[0046] The reference topological structure features refer to the structured relationship features of the position of the reference entity in the target knowledge graph network, which are used to quantify the connection mode, hierarchical relationship, and network influence between the reference entity and other entities.

[0047] The reference mixed features refer to a composite feature representation generated by splicing the reference semantic features and the reference topological structure features when the reference entity lacks a reference image. Feature splicing refers to the operation of combining the reference semantic features and the reference topological structure features into a single mixed feature vector through dimension connection.

[0048] In an implementation, the attribute field set of the reference entity in the target knowledge graph is traversed. If there is a predefined image association attribute and the field value is not empty, it is determined that the reference entity has a reference image. Otherwise, if the field is missing or the value is empty, it is determined that the reference entity does not have a reference image.

[0049] In another implementation, if the target knowledge graph does not explicitly label image attributes, then the text description data associated with the baseline entity is extracted, and the text description data is checked for the presence of image-related terms. If such terms appear frequently in the text description data, then a baseline image is determined to exist; if the text description data does not contain related descriptions, then a baseline image is determined to be absent.

[0050] Furthermore, if the benchmark entity does not have a benchmark image in the target knowledge graph, then the entity vocabulary of the benchmark entity in the target knowledge graph is obtained, and the benchmark semantic features corresponding to the benchmark entity in the target knowledge graph are generated based on the entity vocabulary. Also, at least one neighboring entity corresponding to the benchmark entity in the target knowledge graph is determined, and the benchmark topological features corresponding to the benchmark entity in the target knowledge graph are generated based on the topological features of each neighboring entity.

[0051] Furthermore, the baseline semantic features and baseline topological features are concatenated, and the resulting feature vector is used as the baseline hybrid feature. For example, the baseline hybrid feature is obtained as follows: H hyb =[Ht⊕Hs]; where H hyb Ht represents the baseline hybrid feature, Hs represents the baseline semantic feature, and Hs represents the baseline topological feature.

[0052] S103. The baseline hybrid features are processed by a variational autoencoder to obtain the probability distribution parameters of the baseline entity in each potential feature dimension.

[0053] Variational autoencoders are generative neural networks that combine deep learning and probabilistic graphical models. Their core objective is to generate new samples or reconstruct input data by learning the latent distribution of the data.

[0054] Each latent feature dimension refers to an abstract feature representation axis automatically decoupled by the variational autoencoder through unsupervised learning. These dimensions collectively constitute a low-dimensional continuous space used to capture the essential structure of the baseline entity. Taking image generation as an example, latent feature dimensions include, but are not limited to, facial contour sharpness, skin saturation, and expression intensity, etc.

[0055] The probability distribution parameter refers to the key mathematical descriptor of the probability distribution of the reference entity across each latent feature dimension defined by the variational autoencoder. The probability distribution parameter includes mean information and variance information. The mean information represents the expected value of the reference entity across each latent feature dimension, used to capture the stable features of the reference entity; the variance information represents the uncertainty of the reference entity across each latent feature dimension, used to reflect the variable range of the features.

[0056] For example, if the reference entity is a vase, the probability distribution parameters of the reference entity in each potential feature dimension include, but are not limited to, the curvature of the body (μ = 0.8, σ² = 0.1), the saturation of the glaze (μ = -1.5, σ² = 0.4), the complexity of the decoration (μ = 0.3, σ² = 0.05), and the like.

[0057] In an embodiment, the reference mixed features are input into a variational autoencoder, and the reference mixed features are processed by the variational autoencoder to output probability distribution parameters of the reference entity in each latent feature dimension, including mean information and variance information.

[0058] In S104, the probability distribution parameters are input into an image feature prediction model, and the model parameters of the image feature prediction model are used to reparameterize the probability distribution parameters to obtain predicted image features corresponding to the reference entity as the reference image features of the reference entity in the target knowledge graph.

[0059] The image feature prediction model is a converter that decodes the probability distribution parameters in the latent space into structured image feature representations. The model parameters of the image feature prediction model refer to the learnable weights inside the image feature prediction model, which are obtained by optimizing the training data and are used to map the probability distribution parameters to the predicted image features.

[0060] Optionally, the network structure of the image feature prediction model includes an input interface layer, a convolutional conversion layer, and a reparameterization output layer. The input interface layer is composed of a fully connected layer, which is used to receive the probability distribution parameters and perform dimension conversion to adapt to subsequent convolution operations. The convolutional conversion layer performs convolution operations on the probability distribution parameters to obtain multi-scale feature maps. The reparameterization output layer is used to map the multi-scale feature maps to the predicted image features that meet the requirements of the target knowledge graph based on the trained model parameters.

[0061] The predicted image features refer to high-dimensional vectors that represent the visual attributes of the reference entity reconstructed by the image feature prediction model. The reference image features refer to standardized visual representation vectors that uniquely identify and associate the reference entity in the target knowledge graph.

[0062] In an embodiment, the probability distribution parameters are input into the image feature prediction model, and the trained model parameters in the image feature prediction model are used to reparameterize the probability distribution parameters to obtain predicted image features corresponding to the reference entity.

[0063] For example, if the model parameters are z, the mean information in the probability distribution parameters is μ, and the variance information is σ², then the predicted image features Hv' = z·σ²+ μ.

[0064] Further, the predicted image feature output by the image feature prediction model is acquired as the reference image feature corresponding to the reference entity in the target knowledge graph.

[0065] In S105, the reference entity and the entity to be aligned are aligned according to the reference image feature and the image feature corresponding to the entity to be aligned in the other knowledge graph.

[0066] The image feature to be aligned refers to the visual representation vector of the entity to be aligned in the other knowledge graph, which is used for similarity calculation with the reference image feature of the reference entity, and is the core data for realizing cross-modal entity alignment. Entity alignment refers to a technical process of judging whether the entities described in different knowledge graphs refer to the same object in the real world.

[0067] In an embodiment, it is determined whether the entity to be aligned has an image feature to be aligned in the other knowledge graph. If it has, the image feature to be aligned is directly acquired. If it does not have the image feature to be aligned, the semantic feature to be aligned and the topological structure feature to be aligned corresponding to the entity to be aligned in the other knowledge graph are acquired, the mixed feature to be aligned is generated by feature splicing according to the semantic feature to be aligned and the topological structure feature to be aligned, the probability distribution parameter of the entity to be aligned in each latent feature dimension is obtained by processing the mixed feature to be aligned through the variational autoencoder, and the predicted image feature corresponding to the entity to be aligned is obtained by inputting the probability distribution parameter into the image feature prediction model and reparameterizing the probability distribution parameter using the model parameter of the image feature prediction model, as the image feature to be aligned corresponding to the entity to be aligned in the other knowledge graph. It can be understood that if the other knowledge graph does not have the image feature to be aligned of the entity to be aligned, the specific implementation of how to obtain the image feature to be aligned can refer to the specific implementation of how to obtain the reference image feature in this embodiment, which will not be repeated here.

[0068] Further, after acquiring the image feature to be aligned, in an embodiment, similarity calculations are respectively performed according to the reference image feature and the image feature to be aligned, the reference semantic feature and the semantic feature to be aligned, and the reference topological structure feature and the topological structure feature to be aligned, and the reference entity and each entity to be aligned are aligned according to the similarity calculation results between the reference entity and each entity to be aligned.

[0069] The embodiment of the application generates a reference mixed feature by splicing the reference semantic feature and the reference topological structure feature when the target knowledge graph does not have a reference image, processes the reference mixed feature through a variational autoencoder to obtain probability distribution parameters of the reference entity on each latent feature dimension, and then inputs the probability distribution parameters into an image feature prediction model, reparameterizes the probability distribution parameters using model parameters of the image feature prediction model to obtain predicted image features corresponding to the reference entity as reference image features. The beneficial effect is that when the knowledge graph lacks the image of an entity, the semantic feature and the topological structure feature of the entity are used to predict the image feature of the entity, which can improve the accuracy of entity feature identification and further improve the precision of entity alignment.

[0070] Optionally, the model parameters are obtained in the following manner:

[0071] A. Process the sample mixed feature through the variational autoencoder to obtain sample probability distribution parameters of the sample entity on each latent feature dimension.

[0072] The sample mixed feature is a kind of composite feature representation generated by splicing the sample semantic feature and the sample topological structure feature, and is a kind of sample data for model training. The sample semantic feature refers to the abstract semantic information contained in the sample entity in the target knowledge graph, and the sample topological structure feature refers to the structural relationship feature of the position of the sample entity in the target knowledge graph network, which is used to quantify the connection mode, hierarchical relationship and network influence between the sample entity and other entities. The sample entity refers to a specific entity selected from the target knowledge graph for participating in the training of the model parameters of the image feature prediction model. It can be a predefined entity in the target knowledge graph, such as a disease entity in the medical field, a security entity in the financial field, etc., and can also be an entity selected from the target knowledge graph.

[0073] The sample probability distribution parameter refers to a key mathematical description quantity of the probability distribution defined by the variational autoencoder for the sample entity on each latent feature dimension.

[0074] B. Input the sample probability distribution parameters into the to-be-trained model, reparameterize the sample probability distribution parameters using the to-be-trained model parameters of the to-be-trained model to obtain sample predicted image features corresponding to the sample entity.

[0075] The sample predicted image feature refers to a high-dimensional vector representing the visual attributes of the sample entity reconstructed by the to-be-trained model using the to-be-trained model parameters.

[0076] C. Obtain a target loss value, and train the to-be-trained model according to the target loss value to update the to-be-trained model parameters to obtain the model parameters.

[0077] The target loss value is the core index for guiding the optimization of the model, and its essence is to quantify the difference between the model prediction result and the real data through a mathematical function, and to adjust the model parameters in the opposite direction according to the difference.

[0078] The target loss value includes at least one of the following:

[0079] 1) A first loss value calculated according to the real image features corresponding to the sample entity and the sample predicted image features.

[0080] The real image features refer to reference data that can accurately represent the essential attributes of the sample entity image and are obtained by a reliable method, and are usually used as a supervision signal to guide model training.

[0081] In an embodiment, the image feature difference is calculated according to the real image features and the sample predicted image features, and the first loss value is determined according to the image feature difference.

[0082] 2) A second loss value calculated according to the reconstructed mixed features and the sample mixed features.

[0083] The reconstructed mixed features are obtained by decoding the sample predicted image features using the decoder of the variational autoencoder.

[0084] In an embodiment, the reconstructed mixed features are obtained by decoding the sample predicted image features using the decoder of the variational autoencoder, the mixed feature difference is calculated according to the reconstructed mixed features and the sample mixed features, and the second loss value is determined according to the mixed feature difference.

[0085] 3) A third loss value obtained by calculating the divergence loss according to the sample predicted image features.

[0086] The divergence loss calculation specifically refers to using divergence as a measurement standard to evaluate the difference between two probability distributions.

[0087] In an embodiment, the KL divergence loss function is used to calculate the divergence loss according to the sample predicted image features, and the third loss value is obtained.

[0088] 4) A fourth loss value obtained by calculating the similarity constraint loss according to the sample predicted image features.

[0089] The similarity constraint loss calculation refers to using mathematical methods to force the model to learn the similarity relationship between specific features, and quantifying this relationship as a loss value.

[0090] In an embodiment, the similarity constraint loss function is used to calculate the divergence loss according to the sample predicted image features, and the fourth loss value is obtained.

[0091] In an implementation, the first loss value, the second loss value, the third loss value and the fourth loss value are used as constraint conditions for training, and the to-be-trained model is trained to minimize the first loss value, the second loss value, the third loss value and the fourth loss value, so as to update the parameters of the to-be-trained model and obtain the model parameters.

[0092] By setting the target loss value to include at least one of: a first loss value calculated according to the real image features corresponding to the sample entity and the sample predicted image features; a second loss value calculated according to the reconstructed mixed features and the sample mixed features; wherein the reconstructed mixed features are obtained by decoding the sample predicted image features by the decoder of the variational autoencoder; a third loss value obtained by performing divergence loss calculation on the sample predicted image features; and a fourth loss value obtained by performing similarity constraint loss calculation on the sample predicted image features, the beneficial effects are:

[0093] The first loss value directly constrains the model output to approximate the real distribution, ensuring high-precision reconstruction of low-level visual information; the second loss value generates reconstructed features through the decoder, forcing the model to learn decoupled semantic representation, avoiding feature redundancy and improving feature interpretability; the third loss value uses divergence to constrain the matching between the predicted feature distribution and the prior distribution, preventing overfitting and enhancing feature space specification; and the fourth loss value uses contrastive learning to force features of the same type of samples to be clustered and features of different types of samples to be separated, significantly improving the robustness of downstream tasks.

[0094] Embodiment two

[0095] Figure 2 A flowchart of an entity alignment method of a knowledge graph according to Embodiment Two of the present application is provided, which further optimizes and extends the above-mentioned embodiments and can be combined with the above-mentioned optional embodiments. As shown in Figure 2 the method comprises:

[0096] S201, obtaining a target knowledge graph and other knowledge graphs except the target knowledge graph, determining a reference entity from the target knowledge graph, and determining a to-be-aligned entity from the other knowledge graphs.

[0097] Optionally, after determining the reference entity from the target knowledge graph, the method further comprises:

[0098] S2011, if the reference entity has a reference image in the target knowledge graph, pre-processing the reference image to obtain at least one expanded image.

[0099] Pre-processing includes at least one of rotation processing, horizontal flip processing and mirror transformation processing.

[0100] In an implementation, it is determined whether the reference entity has a reference image in the target knowledge graph, and if it is determined that the reference entity has a reference image in the target knowledge graph, the reference image is respectively subjected to rotation processing, horizontal flip processing and mirror transformation processing to obtain at least one expanded image, wherein the rotation angle of the rotation processing includes but is not limited to 90°, 180° and 270°.

[0101] In S2012, feature extraction is performed on the reference image to obtain a first image feature vector, and feature extraction is performed on the expanded image to obtain a second image feature vector.

[0102] In an implementation, the ResNet152 model is used to perform feature extraction on the reference image to obtain the first image feature vector, and the ResNet152 model is used to perform feature extraction on each expanded image to obtain the second image feature vector.

[0103] In S2013, the reference image feature corresponding to the reference entity in the target knowledge graph is determined according to the feature vector mean of the first image feature vector and the second image feature vector.

[0104] In an implementation, the feature vector mean is calculated according to the first image feature vector and each second image feature vector, the image feature vector mean is determined as the reference image feature corresponding to the reference entity in the target knowledge graph.

[0105] In S2014, entity alignment is performed on the reference entity and the entity to be aligned according to the reference image feature and the image feature corresponding to the entity to be aligned in other knowledge graphs.

[0106] By determining whether the reference entity has a reference image in the target knowledge graph, if the reference entity has a reference image in the target knowledge graph, the reference image is preprocessed to obtain at least one expanded image, feature extraction is performed on the reference image to obtain a first image feature vector, and feature extraction is performed on the expanded image to obtain a second image feature vector, the reference image feature corresponding to the reference entity in the target knowledge graph is determined according to the feature vector mean of the first image feature vector and the second image feature vector, and entity alignment is performed on the reference entity and the entity to be aligned according to the reference image feature and the image feature corresponding to the entity to be aligned in other knowledge graphs, and the beneficial effects are as follows:

[0107] Firstly, the reference image is preprocessed to generate diversified expanded images, simulate the perspective changes in real scenes, and enhance the feature robustness.

[0108] Secondly, the feature vector mean calculation can effectively resist noise interference.

[0109] S202, if the reference entity does not have a reference image in the target knowledge graph, obtaining an entity vocabulary corresponding to the reference entity in the target knowledge graph; encoding the entity vocabulary, and determining the reference semantic feature corresponding to the reference entity in the target knowledge graph according to the feature vector obtained by encoding.

[0110] Among them, the entity vocabulary refers to the structured text attribute set describing the reference entity in the target knowledge graph.

[0111] In an embodiment, if the reference entity does not have a reference image in the target knowledge graph, an entity vocabulary corresponding to the reference entity in the target knowledge graph is obtained, the entity vocabulary is encoded using a fastText model, and a reference semantic feature corresponding to the reference entity in the target knowledge graph is determined according to the feature vector obtained by encoding.

[0112] By obtaining the entity vocabulary corresponding to the reference entity in the target knowledge graph; encoding the entity vocabulary, and determining the reference semantic feature corresponding to the reference entity in the target knowledge graph according to the feature vector obtained by encoding, the beneficial effects are:

[0113] First, when the entity has no reference image, the semantic feature is extracted through the entity vocabulary, avoiding feature discontinuity caused by missing images.

[0114] Second, the entity vocabulary encoding fuses multi-source text information, and the generated semantic feature is more comprehensive, reducing the noise bias that a single image may introduce.

[0115] S203, if the reference entity does not have a reference image in the target knowledge graph, determining at least one neighborhood entity corresponding to the reference entity in the target knowledge graph, and obtaining a topological structure feature corresponding to each neighborhood entity in the target knowledge graph as a neighborhood topological structure feature; determining the reference topological structure feature corresponding to the reference entity in the target knowledge graph according to each neighborhood topological structure feature.

[0116] Among them, the neighborhood entity refers to other entities directly associated with the reference entity in the target knowledge graph, and these associations form a local network structure through the edges in the target knowledge graph. Neighborhood entities are key elements for understanding the context environment and relationship network of the reference entity. The neighborhood topological structure feature refers to a mathematical or vectorized representation describing the connection relationship and position attribute of the neighborhood entity in the target knowledge graph structure.

[0117] In an implementation, if the reference entity does not have a reference image in the target knowledge graph, other entities having an edge relationship with the reference entity are determined as neighborhood entities according to the graph structure of the target knowledge graph. Further, the respective neighborhood topological structure features of each neighborhood entity in the target knowledge graph are obtained, and the neighborhood topological structure features are used to extract the topological structure feature of the reference entity in the target knowledge graph as the reference topological structure feature by using a graph encoder. Optionally, the graph encoder includes but is not limited to a Dual-AMN (Dual Attention Matching Network), which greatly reduces the computational complexity while capturing intra-graph and cross-graph relationship information by simplifying the relationship attention layer and the proxy matching attention layer.

[0118] By determining at least one neighborhood entity corresponding to the reference entity in the target knowledge graph, and obtaining the respective topological structure features of each neighborhood entity in the target knowledge graph as neighborhood topological structure features, and determining the reference topological structure feature of the reference entity in the target knowledge graph according to the neighborhood topological structure features, the beneficial effects are as follows:

[0119] In the first aspect, when the reference entity lacks a reference image, the neighborhood topological structure features of the neighborhood entities are used to reconstruct the features of the reference entity, avoiding the failure of entity alignment caused by the lack of single-point data.

[0120] In the second aspect, the link features between the neighborhood entities can encode the implicit logical rules between the entities, providing a representation capability beyond the image vision, and further improving the accuracy of entity alignment.

[0121] S204, the reference semantic features and the reference topological structure features are spliced to generate reference mixed features, and the reference mixed features are processed by a variational autoencoder to obtain probability distribution parameters of the reference entity in each latent feature dimension.

[0122] S205, the probability distribution parameters are input into an image feature prediction model, and the probability distribution parameters are reparameterized by using the model parameters of the image feature prediction model to obtain predicted image features corresponding to the reference entity as the reference image features corresponding to the reference entity in the target knowledge graph.

[0123] S206, the reference image features and the to-be-aligned image features corresponding to the to-be-aligned entity in other knowledge graphs are used to perform entity alignment on the reference entity and the to-be-aligned entity.

[0124] Embodiment Three

[0125] Figure 3A flowchart of an entity alignment method of a knowledge graph provided for the third embodiment of the present application, the present embodiment further optimizes and extends the step "aligning the reference entity and the entity to be aligned according to the reference image features and the corresponding image features of the entity to be aligned in other knowledge graphs" in the above-mentioned embodiments, and can be combined with the above-mentioned various optional embodiments. As shown in FIG. 3, the method comprises the following steps. Figure 3

[0126] S301, acquiring a reference entity name corresponding to the reference entity in the target knowledge graph, and acquiring a to-be-aligned semantic feature, a to-be-aligned topological structure feature, and a to-be-aligned entity name corresponding to each entity to be aligned in other knowledge graphs.

[0127] The reference entity name refers to the standardized identifier of the reference entity to be aligned in the target knowledge graph, usually adopting a unique ID or a standard name. The to-be-aligned entity name refers to the standardized identifier of the to-be-aligned entity in other knowledge graphs that may point to the same real object as the reference entity, usually adopting a unique ID or a standard name.

[0128] S302, determining the image feature similarity of the reference entity and each to-be-aligned entity according to the reference image features and the to-be-aligned image features.

[0129] The image feature similarity refers to the semantic closeness of two entity corresponding images in a high-dimensional feature space quantified by a mathematical method.

[0130] In an embodiment, an image feature similarity calculation algorithm is adopted to calculate the similarity of the reference image features and the to-be-aligned image features, and the image feature similarity of the reference entity and each to-be-aligned entity is determined according to the similarity calculation result.

[0131] S303, determining the semantic feature similarity of the reference entity and each to-be-aligned entity according to the reference semantic features and the to-be-aligned semantic features.

[0132] The semantic feature similarity refers to the correlation strength of two entities at the semantic level quantified by a mathematical model, which essentially converts human understandable semantic information into machine computable numerical indicators.

[0133] In an embodiment, a semantic feature similarity calculation algorithm is adopted to calculate the similarity of the reference semantic features and the to-be-aligned semantic features, and the semantic feature similarity of the reference entity and each to-be-aligned entity is determined according to the similarity calculation result.

[0134] S304, determining the topological structure feature similarity of the reference entity and each to-be-aligned entity according to the reference topological structure features and the to-be-aligned topological structure features.​

[0135] The topological structure feature similarity refers to quantifying the connection mode similarity of two entities in a graph structure by a mathematical method, and the essence is to analyze the structural role equivalence of the entities in respective knowledge graph networks.

[0136] In an embodiment, a topological structure feature similarity calculation algorithm such as a cosine distance algorithm is used to calculate the similarity of the reference topological structure feature and each topological structure feature to be aligned respectively, and the topological structure feature similarity corresponding to the reference entity and each entity to be aligned respectively is determined according to the similarity calculation result.

[0137] S305, determining the entity name character similarity corresponding to the reference entity and each entity to be aligned respectively according to the reference entity name and the entity name of each entity to be aligned.

[0138] The entity name character similarity refers to quantifying the similarity of two entity name strings at the character level by an algorithm, and the essence is to measure the surface form correlation of the name text rather than the deep semantic correlation.

[0139] In an embodiment, a character similarity calculation algorithm such as a Levenshtein algorithm is used to calculate the similarity of the reference entity name and each entity name to be aligned respectively, and the entity name character similarity corresponding to the reference entity and each entity to be aligned respectively is determined according to the similarity calculation result.

[0140] S306, performing entity alignment on the reference entity and each entity to be aligned according to the image feature similarity, the semantic feature similarity, the topological structure feature similarity, and the entity name character similarity.

[0141] In an embodiment, each entity to be aligned is screened according to the image feature similarity, the semantic feature similarity, the topological structure feature similarity, and the entity name character similarity, and the entity aligned with the reference entity is determined from each entity to be aligned.

[0142] The reference entity name corresponding to the reference entity in the target knowledge graph is obtained, and the to-be-aligned semantic features, to-be-aligned topological structure features, and to-be-aligned entity names corresponding to each to-be-aligned entity in other knowledge graphs are obtained; image feature similarities corresponding to the reference entity and each to-be-aligned entity are determined according to the reference image features and each to-be-aligned image feature; semantic feature similarities corresponding to the reference entity and each to-be-aligned entity are determined according to the reference semantic features and each to-be-aligned semantic feature; topological structure feature similarities corresponding to the reference entity and each to-be-aligned entity are determined according to the reference topological structure features and each to-be-aligned topological structure feature; entity name character similarities corresponding to the reference entity and each to-be-aligned entity are determined according to the reference entity name and each to-be-aligned entity name; and entity alignment is performed on the reference entity and each to-be-aligned entity according to the image feature similarities, the semantic feature similarities, the topological structure feature similarities, and the entity name character similarities, so that:

[0143] The image feature similarities can solve visual entity ambiguity; the semantic feature similarities can solve cross-language and term variation; the topological structure feature similarities can capture structural consistency; and the character similarities can quickly screen high-confidence candidates, so that the four features are used for entity alignment, multi-modal evidence complementation is achieved, dependence on a single feature is reduced, the precision of entity alignment is effectively improved, and the robustness of entity alignment is enhanced.

[0144] Optionally, the entity alignment on the reference entity and each to-be-aligned entity according to the image feature similarities, the semantic feature similarities, the topological structure feature similarities, and the entity name character similarities includes:

[0145] The image feature similarities, the semantic feature similarities, the topological structure feature similarities, and the entity name character similarities are weighted and summed to determine fusion similarities corresponding to the reference entity and each to-be-aligned entity; and entity alignment is performed on the reference entity and each to-be-aligned entity according to the fusion similarities.

[0146] In an implementation manner, the fusion similarity is determined in the following manner:

[0147] M = α·Ms + β·Mt + γ·Mv + θ·ML; wherein, Ms represents the topological structure feature similarity, Mt represents the semantic feature similarity, Mv represents the image feature similarity, ML represents the entity name character similarity, α + β + γ + θ = 1, α represents a weight corresponding to the topological structure feature similarity, β represents a weight corresponding to the semantic feature similarity, γ represents a weight corresponding to the image feature similarity, θ represents a weight corresponding to the entity name character similarity, and M represents a fusion similarity between the reference entity and any to-be-aligned entity.

[0148] Optionally, the determination manner of the weight alpha, the weight beta, the weight gamma and the weight theta comprises but is not limited to utilizing a grid search method to determine, that is, finding the most suitable parameter combination in the candidate parameters to be respectively the weight alpha, the weight beta, the weight gamma and the weight theta.

[0149] According to the fusion similarity, each entity to be aligned is screened, and an entity to be aligned corresponding to the reference entity is determined.

[0150] By weighting and summing according to the image feature similarity, the semantic feature similarity, the topological structure feature similarity and the entity name character similarity, the fusion similarity corresponding to the reference entity and each entity to be aligned is determined; and according to the fusion similarity, the entity alignment of the reference entity and each entity to be aligned is performed, and the beneficial effects are as follows:

[0151] In the first aspect, the dynamic weight distribution mechanism is used to realize the optimal combination of multi-modal features, solve the single-feature blind area, and improve the accuracy of entity alignment.

[0152] In the second aspect, by fusing the image feature similarity, the semantic feature similarity, the topological structure feature similarity and the entity name character similarity, most scenes can be covered, and the credibility of entity alignment can be improved.

[0153] Optionally, according to the fusion similarity, the entity alignment of the reference entity and each entity to be aligned is performed, which comprises:

[0154] The fusion similarities are sorted, and the highest fusion similarity is determined according to the sorting result; the matching entity is determined from the entity to be aligned according to the highest fusion similarity, and the entity alignment pair between the matching entity and the reference entity is constructed.

[0155] In an embodiment, the fusion similarities are sorted from high to low, the highest fusion similarity is selected as the highest fusion similarity according to the sorting result, that is, the fusion similarity with the highest value, and the entity to be aligned corresponding to the highest fusion similarity is determined as the matching entity corresponding to the reference entity, and then the entity alignment pair between the matching entity and the reference entity is constructed.

[0156] By sorting the fusion similarities, the highest fusion similarity is determined according to the sorting result; the matching entity is determined from the entity to be aligned according to the highest fusion similarity, and the entity alignment pair between the matching entity and the reference entity is constructed, and the beneficial effects are as follows:

[0157] In the first aspect, the semantic feature similarity, the topological structure feature similarity and the image feature similarity are fused to avoid the limitation of single feature and realize the effect of improving the accuracy of entity alignment by multi-features.

[0158] In the second aspect, only the entity pairs with the fusion similarity reaching the peak are accepted, the interference of suboptimal solutions is avoided, and the reliability of entity alignment is ensured.

[0159] Embodiment Four

[0160] Figure 4 A structural schematic diagram of an entity alignment device for a knowledge graph provided by Embodiment Four of the present application can be applicable to the case where the image of an entity is missing in a knowledge graph, and the image feature of the entity is predicted by using the semantic feature and the topological structure feature of the entity, for entity alignment, as shown in the figure, the device comprises: Figure 4

[0161] An entity determination module 41 is configured to acquire a target knowledge graph and other knowledge graphs except the target knowledge graph, determine a reference entity from the target knowledge graph, and determine an entity to be aligned from the other knowledge graphs;

[0162] A mixed feature generation module 42 is configured to, if the reference entity does not have a reference image in the target knowledge graph, acquire reference semantic features and reference topological structure features corresponding to the reference entity in the target knowledge graph, and perform feature splicing according to the reference semantic features and the reference topological structure features to generate reference mixed features;

[0163] A feature encoding module 43 is configured to process the reference mixed features by using a variational autoencoder to obtain probability distribution parameters of the reference entity in each latent feature dimension; wherein the probability distribution parameters comprise mean information and variance information;

[0164] An image feature prediction module 44 is configured to input the probability distribution parameters into an image feature prediction model, reparameterize the probability distribution parameters by using model parameters of the image feature prediction model, and obtain predicted image features corresponding to the reference entity as the reference image features corresponding to the reference entity in the target knowledge graph;

[0165] An entity alignment module 45 is configured to perform entity alignment on the reference entity and the entity to be aligned according to the reference image features and image features corresponding to the entity to be aligned in the other knowledge graphs.

[0166] Optionally, the model parameters are obtained in the following manner:

[0167] The sample mixed features are processed by using a variational autoencoder to obtain sample probability distribution parameters of a sample entity in each latent feature dimension;

[0168] ​input the sample probability distribution parameter into a to-be-trained model, reparameterize the sample probability distribution parameter by using a to-be-trained model parameter of the to-be-trained model, and obtain a sample predicted image feature corresponding to the sample entity;

[0169] obtain a target loss value, train the to-be-trained model according to the target loss value, update the to-be-trained model parameter, and obtain the model parameter;

[0170] The target loss value includes at least one of the following:

[0171] a first loss value calculated according to a real image feature corresponding to a sample entity and the sample predicted image feature;

[0172] a second loss value calculated according to a reconstructed mixed feature and the sample mixed feature, wherein the reconstructed mixed feature is obtained by decoding the sample predicted image feature by using a decoder of the variational autoencoder;

[0173] a third loss value obtained by performing divergence loss calculation according to the sample predicted image feature;

[0174] a fourth loss value obtained by performing similarity constraint loss calculation according to the sample predicted image feature.

[0175] Optionally, the mixed feature generation module 42 is specifically configured to:

[0176] obtain an entity vocabulary corresponding to the reference entity in the target knowledge graph;

[0177] encode the entity vocabulary, and determine a reference semantic feature corresponding to the reference entity in the target knowledge graph according to a feature vector obtained by encoding.

[0178] Optionally, the mixed feature generation module 42 is specifically further configured to:

[0179] determine at least one neighborhood entity corresponding to the reference entity in the target knowledge graph, and obtain a topological structure feature corresponding to each of the neighborhood entities in the target knowledge graph as a neighborhood topological structure feature;

[0180] determine a reference topological structure feature corresponding to the reference entity in the target knowledge graph according to each of the neighborhood topological structure features.

[0181] Optionally, the apparatus further includes an image feature vector processing module, which is specifically configured to:

[0182] if the reference entity has a reference image in the target knowledge graph, pre-processing the reference image to obtain at least one augmented image; wherein the pre-processing includes at least one of rotation processing, horizontal flip processing and mirror transformation processing;

[0183] performing feature extraction on the reference image to obtain a first image feature vector, and performing feature extraction on the augmented image to obtain a second image feature vector;

[0184] determining a reference image feature corresponding to the reference entity in the target knowledge graph according to the mean of the feature vectors of the first image feature vector and the second image feature vector;

[0185] performing entity alignment on the reference entity and the entity to be aligned according to the reference image feature and the image feature corresponding to the entity to be aligned in the other knowledge graph.

[0186] Optionally, the entity alignment module 45 is specifically configured to:

[0187] obtain a reference entity name corresponding to the reference entity in the target knowledge graph, and obtain a semantic feature to be aligned, a topological structure feature to be aligned, and an entity name to be aligned corresponding to each entity to be aligned in the other knowledge graph;

[0188] determine an image feature similarity corresponding to the reference entity and each entity to be aligned according to the reference image feature and each image feature to be aligned;

[0189] determine a semantic feature similarity corresponding to the reference entity and each entity to be aligned according to the reference semantic feature and each semantic feature to be aligned;

[0190] determine a topological structure feature similarity corresponding to the reference entity and each entity to be aligned according to the reference topological structure feature and each topological structure feature to be aligned;

[0191] determine an entity name character similarity corresponding to the reference entity and each entity to be aligned according to the reference entity name and each entity name to be aligned;

[0192] perform entity alignment on the reference entity and each entity to be aligned according to the image feature similarity, the semantic feature similarity, the topological structure feature similarity and the entity name character similarity.

[0193] Optionally, the entity alignment module 45 is specifically further configured to:

[0194] According to the image feature similarity, the semantic feature similarity, the topology structure feature similarity and the entity name character similarity, a fusion similarity corresponding to each of the reference entity and the to-be-aligned entity is determined by weighted summation;

[0195] According to the fusion similarity, the reference entity and the to-be-aligned entity are aligned.

[0196] Optionally, the entity alignment module 45 is specifically further configured to:

[0197] The fusion similarities are sorted, and a highest fusion similarity is determined according to a sorting result.

[0198] According to the highest fusion similarity, a matched entity is determined from the to-be-aligned entities, and an entity alignment pair between the matched entity and the reference entity is constructed.

[0199] The entity alignment device of the knowledge graph provided in the embodiments of the present application can execute the entity alignment method of the knowledge graph provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0200] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0201] Embodiment five

[0202] Figure 5 A structural schematic diagram of an electronic device 50 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0203] As Figure 5As shown, the electronic device 50 includes at least one processor 51, and a memory, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., connected to the at least one processor 51 in communication. The memory stores computer programs executable by the at least one processor 51, and the processor 51 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 52 or loaded from the storage unit 58 into the random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other through a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0204] Various components in the electronic device 50 are connected to the I / O interface 55, including an input unit 56, such as a keyboard, a mouse, etc., an output unit 57, such as various types of displays, a speaker, etc., a storage unit 58, such as a magnetic disk, an optical disk, etc., and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0205] The processor 51 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 51 performs various methods and processes described above, such as the entity alignment method of the knowledge graph.

[0206] In some embodiments, the entity alignment method of the knowledge graph can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 58. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the entity alignment method of the knowledge graph described above can be performed. Alternatively, in other embodiments, the processor 51 can be configured to perform the entity alignment method of the knowledge graph by any other appropriate means, such as by means of firmware.

[0207] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0208] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program

[0209] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0210] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0211] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), a blockchain network, and the Internet.

[0212] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private service.

[0213] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0214] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Accordingly, the disclosure is not limited to the specific embodiments described above, but only by the scope of the appended claims.

Claims

1. A method for entity alignment in a knowledge graph, characterized in that, The method includes: Obtain a target knowledge graph and other knowledge graphs besides the target knowledge graph, determine a baseline entity from the target knowledge graph, and determine entities to be aligned from the other knowledge graphs; If the benchmark entity does not have a benchmark image in the target knowledge graph, then the benchmark semantic features and benchmark topological features corresponding to the benchmark entity in the target knowledge graph are obtained, and the features are spliced ​​according to the benchmark semantic features and benchmark topological features to generate benchmark hybrid features; The baseline hybrid features are processed by a variational autoencoder to obtain the probability distribution parameters of the baseline entity in each latent feature dimension; wherein, the probability distribution parameters include mean information and variance information; The probability distribution parameters are input into the image feature prediction model, and the probability distribution parameters are reparameterized using the model parameters of the image feature prediction model to obtain the predicted image features corresponding to the benchmark entity, which are used as the benchmark image features corresponding to the benchmark entity in the target knowledge graph. Based on the baseline image features and the corresponding image features of the entity to be aligned in the other knowledge graph, entity alignment is performed on the baseline entity and the entity to be aligned. Wherein, obtaining the benchmark semantic features corresponding to the benchmark entity in the target knowledge graph includes: Obtain the entity vocabulary corresponding to the benchmark entity in the target knowledge graph; The entity vocabulary is encoded, and the baseline semantic features corresponding to the baseline entity in the target knowledge graph are determined based on the encoded feature vector.

2. The method according to claim 1, characterized in that, The model parameters are obtained in the following way: The sample mixture features are processed by variational autoencoder to obtain the sample probability distribution parameters of the sample entity in each latent feature dimension; The sample probability distribution parameters are input into the model to be trained, and the sample probability distribution parameters are reparameterized using the model parameters to be trained to obtain the sample prediction image features corresponding to the sample entity. Obtain the target loss value, and train the model to be trained based on the target loss value to update the parameters of the model to be trained, thereby obtaining the model parameters; The target loss value includes at least one of the following: The first loss value is calculated based on the real image features corresponding to the sample entity and the predicted image features of the sample. The second loss value is calculated based on the reconstructed mixture features and the sample mixture features; wherein, the reconstructed mixture features are obtained by decoding the sample predicted image features using the decoder of the variational autoencoder; The third loss value is obtained by calculating the divergence loss based on the predicted image features of the sample; The fourth loss value is obtained by calculating the similarity constraint loss based on the image features predicted from the sample.

3. The method according to claim 1, characterized in that, The step of obtaining the baseline topological structure features corresponding to the baseline entity in the target knowledge graph includes: Determine at least one neighboring entity corresponding to the benchmark entity in the target knowledge graph, and obtain the topological structure features corresponding to each neighboring entity in the target knowledge graph, as the neighboring topological structure features; The baseline topological structure features corresponding to the baseline entity in the target knowledge graph are determined based on the neighborhood topological structure features of each of the aforementioned neighborhoods.

4. The method according to claim 1, further comprising, after determining the baseline entity from the target knowledge graph: If the reference entity has a reference image in the target knowledge graph, then the reference image is preprocessed to obtain at least one augmented image; wherein, the preprocessing includes at least one of rotation processing, horizontal flipping processing, and mirror transformation processing; A first image feature vector is obtained by performing feature extraction on the reference image, and a second image feature vector is obtained by performing feature extraction on the augmented image; Based on the mean of the feature vectors of the first image feature vector and the second image feature vector, the baseline image features corresponding to the baseline entity in the target knowledge graph are determined; Based on the reference image features and the corresponding image features of the entity to be aligned in the other knowledge graph, entity alignment is performed on the reference entity and the entity to be aligned.

5. The method according to claim 1, characterized in that, The step of aligning the reference entity and the entity to be aligned based on the reference image features and the corresponding image features of the entity to be aligned in the other knowledge graph includes: Obtain the name of the benchmark entity corresponding to the benchmark entity in the target knowledge graph, and obtain the semantic features to be aligned, the topological features to be aligned, and the name of the entity to be aligned corresponding to each of the entities to be aligned in the other knowledge graphs respectively; Based on the reference image features and the features of each of the images to be aligned, determine the image feature similarity between the reference entity and each of the entities to be aligned. Based on the baseline semantic features and each of the semantic features to be aligned, the semantic feature similarity between the baseline entity and each of the entities to be aligned is determined. Based on the baseline topological features and each of the topological features to be aligned, the similarity of the topological features corresponding to the baseline entity and each of the entities to be aligned is determined. Based on the baseline entity name and each of the entity names to be aligned, determine the character similarity of the entity names corresponding to the baseline entity and each of the entity names to be aligned; Based on the image feature similarity, semantic feature similarity, topological feature similarity, and entity name character similarity, entity alignment is performed on the baseline entity and each of the entities to be aligned.

6. The method according to claim 5, characterized in that, The step of aligning the baseline entity and each of the entities to be aligned based on the image feature similarity, the semantic feature similarity, the topological structure feature similarity, and the entity name character similarity includes: The fusion similarity between the baseline entity and each entity to be aligned is determined by weighted summation of the image feature similarity, the semantic feature similarity, the topological structure feature similarity, and the entity name character similarity. The baseline entity and each entity to be aligned are aligned based on the fusion similarity.

7. The method according to claim 6, characterized in that, The step of aligning the baseline entity and each of the entities to be aligned based on the fusion similarity includes: The fusion similarities are sorted, and the highest fusion similarity is determined based on the sorting results; Based on the highest fusion similarity, a matching entity is determined from each of the entities to be aligned, and an entity alignment pair is constructed between the matching entity and the baseline entity.

8. An entity alignment device for a knowledge graph, characterized in that, The device includes: An entity determination module is used to acquire a target knowledge graph and other knowledge graphs besides the target knowledge graph, determine a baseline entity from the target knowledge graph, and determine an entity to be aligned from the other knowledge graphs; The hybrid feature generation module is used to obtain the benchmark semantic features and benchmark topological features corresponding to the benchmark entity in the target knowledge graph if the benchmark entity does not have a benchmark image in the target knowledge graph, and to perform feature concatenation based on the benchmark semantic features and benchmark topological features to generate benchmark hybrid features; The feature encoding module is used to process the benchmark mixed features through a variational autoencoder to obtain the probability distribution parameters of the benchmark entity in each potential feature dimension; wherein, the probability distribution parameters include mean information and variance information; The image feature prediction module is used to input the probability distribution parameters into the image feature prediction model, and to reparameterize the probability distribution parameters using the model parameters of the image feature prediction model to obtain the predicted image features corresponding to the benchmark entity, which are used as the benchmark image features corresponding to the benchmark entity in the target knowledge graph. An entity alignment module is used to perform entity alignment between the reference entity and the entity to be aligned based on the reference image features and the corresponding image features of the entity to be aligned in the other knowledge graph. Specifically, the hybrid feature generation module is used for: Obtain the entity vocabulary corresponding to the benchmark entity in the target knowledge graph; The entity vocabulary is encoded, and the baseline semantic features corresponding to the baseline entity in the target knowledge graph are determined based on the encoded feature vector.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the entity alignment method of the knowledge graph according to any one of claims 1-7.

Images