A computer-constructed embedded alignment method

Abstract

The application designs an embedded alignment method constructed by a computer, through the method, the entity name and attribute information of a knowledge graph are constructed by a large-scale pre-training language model through the computer, semantic information on the knowledge graph can be comprehensively utilized, in addition, first-order neighbors and second-order neighbors of entities are respectively modeled by using a graph attention network, utilization of more distant entity information is realized, so that more complex neighbor structures can be captured, and a feature linear modulation method is used to effectively combine word embedding and structure embedding. Experimental results on different data sets show that the application has strong robustness. At the same time, an ablation experiment is set to verify the effectiveness of the application.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph technology, and specifically to entity alignment and enhancement technology for semantic and structural information on knowledge graphs. Background Technology

[0002] In recent years, with the development of various knowledge graph technologies, the scale of knowledge graphs has been continuously expanding, and various fields have begun to build new knowledge graphs according to their own needs. Today, knowledge graphs are widely used in tasks such as retrieval, question answering, and reasoning, supporting applications in many industries. However, a single knowledge graph can no longer meet the actual application needs, and there are problems such as information redundancy and heterogeneity between different knowledge graphs. Therefore, knowledge fusion has become a topic of concern. Entity alignment is an important component of knowledge graph fusion, aiming to find nodes in different knowledge graphs that point to the same entity in the real world.

[0003] Entity alignment by integrating different knowledge graphs is significant but also presents numerous challenges. The main challenges include computational complexity, data quality, and the acquisition of prior knowledge.

[0004] The challenge of computational complexity stems from the booming development of various knowledge bases in the era of big data, resulting in enormous data volumes. This situation leads to the need for massive computational resources to align entities in two knowledge graphs. The design of various entity alignment methods needs to consider reducing computational complexity while ensuring accuracy, so that the methods can be applied to the ever-growing large-scale knowledge graphs.

[0005] The challenge of data quality stems from the lack of a unified standard for constructing various knowledge graphs. For example, two knowledge graphs may focus on different domains, and even if the domains are the same, different data sources may lead to different descriptive languages ​​and different types of relationships. These problems caused by different construction methods and purposes are directly reflected in data quality: the same entity from two knowledge graphs may have different names, such as "apple" and "Apple"; the same name may refer to different real-world entities, such as the "Apple" company and the fruit "apple"; the same entity may have different domain structures and attributes in different knowledge graphs; the same entity may have inconsistent names due to the lack of standardized naming formats, such as "DeoxyriboNucleic Acid" and "DNA"; there may also be differences in the granularity of entities, for example, "lung cancer" may be represented as "disease" and "cancer" in two knowledge graphs with different patterns. The challenge of acquiring prior knowledge stems from aligning the seed dataset, which is the training set in supervised learning, and obtaining it is quite difficult. In practice, it is often necessary to rely on expert manual annotation or crowdsourcing algorithms to expand the size of the alignment seed dataset and minimize the occurrence of noisy data. This challenge enables entity alignment research to achieve the highest possible accuracy with relatively small training data. Summary of the Invention

[0006] In view of this, the present invention provides a computer-based embedded alignment method that solves or partially solves the above-mentioned problems.

[0007] To achieve the effects of the above technical solution, the technical solution of the present invention is as follows: Step 1: Set up the bidirectional encoder representation unit based on the transformer. The function of the bidirectional encoder representation unit based on the transformer is to construct the word embeddings of entity names into vectors with a mean of 0 and a covariance matrix that is an identity matrix. Word embedding is achieved by using language models and representation learning in natural language processing to embed a high-dimensional space with the number of all words into a low-dimensional continuous vector space. Each word or phrase is mapped to a vector in the real number field. Then, the word embedding set of N entity names of the two knowledge graphs is obtained through the large-scale pre-trained language model in the representation unit of the bidirectional encoder based on the transformer. Where N is a natural number, j is a natural number from 1 to N, and x j This represents a single word embedding. Then, the mean μ and covariance matrix Σ of the word embedding set are calculated. After singular value decomposition, matrix slicing is performed based on a pre-defined value of k, where k is a natural number. The mean μ and covariance matrix Σ are calculated using Formula 1: mean μ μ = μ covariance matrix Σ ... covariance matrix Here, T represents transpose; and to reduce the potential information learned in large-scale pre-trained language models and mitigate the impact of entity name bias, different targeted weight files are used for different datasets;

[0008] Step 2: When polysemy and identical names occur, determine whether the word embedding set obtained by the bidirectional encoder representation unit model based on the transformer is also the same. If no contextual information is provided to the bidirectional encoder representation unit based on the transformer when polysemy and identical names occur, then proceed to Step 3, which uses the structural information of the knowledge graph to further improve the entity alignment effect.

[0009] Step 3: Extract neighbors with a certain degree of similarity from the aligned entities in the two knowledge graphs. Then, use a graph attention network to integrate relational information into the structural embedding model of the representation unit in the bidirectional encoder based on the transformer. Next, determine whether semantic association information might exist among the direct or distant neighbors of the aligned entities, and then begin aggregating the semantic association information of direct and distant neighbors. The graph attention network is a spatial graph convolutional network. When aggregating feature information, the attention mechanism of the graph attention network uses an attention mechanism representation to determine the weights of the node's neighborhood.

[0010] Step 4: To expand the effective neighbors of multiple jumps, a gate mechanism network is used in the representation unit of the transformer-based bidirectional encoder to combine first-order neighbors and second-order neighbors, thereby expanding the neighbor entities from second-order to multi-order range. The structure embedding model in the representation unit of the transformer-based bidirectional encoder is accelerated by connecting the effective neighbors through the jumps, so as to improve the training effect of the structure embedding model in the representation unit of the transformer-based bidirectional encoder after using the graph attention network.

[0011] The role of the gate mechanism network is to aggregate second-order neighbors based on the structural embedding model in the representation unit of the bidirectional encoder of the transformer to reduce the noise caused by the differences in first-order neighbors when the first-order neighbors of the aligned entities are the same or not completely the same in different knowledge graphs.

[0012] When different specific entities are the central entity and one of its first-order neighbor entities in the knowledge graph, and the first attention weight is set to represent the different associations between the central entity and one of its first-order neighbor entities, and through nonlinear transformation, the first result of the output of the structural embedding model in the representation unit of the bidirectional encoder based on the transformer can approximate the nonlinear function, thereby handling more complex tasks; in order to make the first attention weight comparable between different entities, normalization must be performed before comparison.

[0013] When different specific entities, namely the central entity and one of its second-order neighbor entities in the knowledge graph, are used, a second attention weight is set to represent the different associations between the central entity and the second-order neighbor entity. Through nonlinear transformation, the second result of the output of the structural embedding model in the representation unit of the bidirectional encoder based on the transformer can approximate the nonlinear function, thereby handling more complex tasks. In order to make the second attention weights comparable between different entities, normalization is required before comparison.

[0014] Using two matrices respectively sum matrix Two matrices are used for the transformation between the central entity and its neighboring entities. sum matrix Initialization begins with two matrices. sum matrix The elements in the model change during the training process, so that the input and output of the structural embedding model in the representation unit of the transformer-based bidirectional encoder meet the system requirements.

[0015] Furthermore, the embedding h of entity S, which combines information from first-order and second-order neighbors, is... u The calculation is as shown in Formula 2:

[0016] h u =g(h i,2 )·h i,1 +(1-g(h i,2 ))·h i,2 Formula 2;

[0017] Where h i,1 and h i,2 These refer to the embeddings obtained from the aggregation of first-order neighbors and second-order neighbors, respectively, and the functional function g(h) i,2 )=σ(Mh i,2 In +b), σ is the activation function, which is used to introduce nonlinear factors into the graph attention network, so that the function of the graph attention network can arbitrarily approximate any nonlinear function. M and b are the weight matrix and bias vector, respectively. The weight matrix M is initialized with a uniform distribution, and the bias vector b is initialized with all zeros. The elements in both the weight matrix and the bias vector change during the training process, so that the input and output of the structure embedding model in the representation unit of the transformer-based bidirectional encoder meet the requirements.

[0018] Step 5: Use the first feature linear modulation layer to combine word embedding and structure embedding. The specific calculation method of the first feature linear modulation is shown in Formula 3 below:

[0019] h u,e =FiLM(h u,s ) = h u,w W1 ehu,s +h u,w W2, formula three;

[0020] Where h u,s The structural embedding of entity u, h u,w The word embedding represents entity u; ⊙ represents the Hadamard product operation, a matrix operation; W1 and W2 are two matrices initialized using a specific uniform distribution. This specific uniform distribution is used to maintain the consistency of activation values ​​and gradients across layers of the graph attention network during propagation; FiLM(h u,s The ) represents the characteristic linear modulation method in graph attention networks, and the final output h of the characteristic linear modulation method is... u,e This is the final entity embedding, and the aligned entity pairs are obtained by calculating cosine similarity.

[0021] Step Six: In addition, for a given pair of alignment seed entities, when the two entities in the alignment pair come from different knowledge graphs and the neighborhood structures of the different knowledge graphs are different, in order to make the most efficient use of the information in the pair of alignment seed entities, before the alignment task begins, the neighborhood structure of the entities is expanded. The triples in the pair of alignment seed entities in the two knowledge graphs are made complementary, thereby enhancing the graph attention network's ability to identify isomorphic subgraphs. Furthermore, through more similar neighborhood structures, it is easier for the entity pairs in the alignment seed set to obtain the same embedding representation. Thus, after processing and propagation by the graph attention network, the beneficial influence factors are extended to the embedding of the entire knowledge graph, resulting in more accurate results.

[0022] The preconditions for step six are set as follows: An alignment loss function and a relational loss function are defined. The difference between the forward computation result and the true value in each iteration of the graph attention network is calculated using these functions, thus guiding the next training step in the correct direction. Samples are substituted into the large-scale pre-trained language model in the representation unit of the transformer-based bidirectional encoder to calculate the output prediction value. The error between the prediction value and the true value is calculated using the alignment loss function and the relational loss function. Based on the derivatives of the alignment loss function and the relational loss function, the error is backpropagated along the direction of minimum gradient to correct the trainable matrices and vectors in the forward computation formula. Iteration stops when the loss values ​​of the alignment loss function and the relational loss function reach a satisfactory value. At this point, the training of the large-scale pre-trained language model in the representation unit of the transformer-based bidirectional encoder is complete.

[0023] Step 7: Design the alignment loss function L by reducing the distance between positive examples and increasing the distance between negative examples. a :

[0024]

[0025] Among them G + and G - Let (m, n) represent the sets of positive and negative entity pairs, respectively. A positive entity pair refers to two aligned entities in two knowledge graphs. (m, n) represents any positive entity pair where m and n are entities from different knowledge graphs, representing the same thing in the real world but originating from different knowledge graphs. A negative entity pair refers to two misaligned entities in two knowledge graphs. (m', n') represents any negative entity pair where m' and n' are entities from different knowledge graphs, representing different things in the real world. The method for generating negative examples involves randomly replacing one entity in an aligned entity pair in the training set. μ and λ are hyperparameters greater than 0. Hyperparameters are parameters set before the training process of a large-scale pre-trained language model in the representation unit of the bidirectional encoder based on the transformer. Here, μ = 0.1 and λ = 1.5. λ can make the distance of negative examples greater than a suitable value, thus better identifying potentially aligned entities. ||·|| represents the second norm of the vector, and ||h m' -h n' || represents the second norm for extracting entity embeddings into vectors from two different knowledge graphs m' and n', max(λ-||h) m' -h n' ||,0) represents the value λ-||h m' -h n' The maximum value between || and 0. If the value is 0, it indicates that the negative instance does not exist, thus avoiding negative values ​​during calculation.

[0026] Step 8: To introduce relational information into the graph attention network, define the relational loss function:

[0027]

[0028] Where R represents the set of all relations, T r Let |T| represent the set of triples (h, r, t) formed by relation r, where |T| r | represents set T r The number of elements in h h The entity embedding of entity h in the triple, h t The entity embedding and relation embedding h represent the entity embedding of entity t in the triple. r Embedded by entity h h and h t The calculation yielded:

[0029]

[0030] The relation loss function reduces the increase in parameters brought about by the introduction of specific relation embeddings, and when the knowledge graph is used as an entity alignment dataset and does not provide relation alignment-related information, it mitigates the effect of inappropriate relation alignment methods leading to unsatisfactory alignment results.

[0031] During training, it is necessary to optimize both the alignment loss function and the relation loss function simultaneously. This is achieved by setting a hyperparameter α to combine the two loss functions into the final loss function: L = L a +αL r ;

[0032] The ultimate goal of entity alignment is to find a set of aligned entity pairs. Aligned entities are obtained by calculating the similarity of entity vectors. During training, the magnitude of the entity vectors is restricted to 1, so that only inner product operations are needed when calculating cosine similarity. In the matrix formed by the entity vectors, matrix multiplication is required to obtain the cosine similarity of all aligned entity pairs. The specific calculation is as follows:

[0033] Here, sim(h1,h2) represents the calculation of cosine similarity; · is the inner product operation of entity vectors, h1 and h2 represent entity embeddings from different knowledge graphs, and ||h1||||h2|| represents the multiplication of the modulus of h1 and h2, which is 1; then the cosine similarity matrix is ​​constructed. Specific implementation methods

[0034] To make the technical problem to be solved, the technical solution, and the beneficial effects of the present invention clearer, the present invention will be described in detail below with reference to embodiments. It should be noted that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. Products that can achieve the same function are equivalent substitutions and improvements, and are all included within the protection scope of the present invention. The specific method is as follows:

[0035] Example: This example specifically illustrates the content of a computer-built embedded alignment method: In order to maximize the acquisition of structural and semantic information on the knowledge graph, this invention obtains word embeddings of entities, attributes or entity categories through a pre-trained language model, obtains structural embeddings of the knowledge graph through a combination of graph attention mechanism and graph attention network, and finally integrates the two into an iterative alignment model.

[0036] This invention employs a transformer-based bidirectional encoder representation unit as a word embedding method for entity names. However, because the word vectors of the transformer-based bidirectional encoder representation unit exhibit a cone-shaped distribution in space, the representations of high-frequency words are closer to the origin, while the representations of low-frequency words are farther from the origin. This leads to a situation where even if high-frequency and low-frequency words have the same semantics, their representations differ due to their different frequencies, making the distance between word vectors an ineffective representation of semantic relevance. Therefore, this invention ultimately uses a transformer-based bidirectional encoder representation unit, ensuring that all representations have the same mean and variance, allowing the similarity between vectors to represent semantic similarity. For general knowledge graphs and biomedical knowledge graphs, this invention uses transformer-based bidirectional encoder representation unit models to maximize the utilization of rich semantic information in the pre-trained models. For knowledge graphs providing attributes and entity categories, the method for obtaining word embeddings is the same as for entity name embedding.

[0037] This invention designs a knowledge graph alignment method that combines word embedding and structural embedding. It uses a large-scale pre-trained language model with a transformer-based bidirectional encoder representation unit to represent semantic information in the knowledge graph, assisting in entity alignment tasks. Then, by aggregating the first-order and second-order neighbors of nodes using a graph attention network, it achieves the ability to capture second-order neighbors, enabling the model to better utilize information from distant entities and thus improving the alignment of central entities. By combining word embedding and structural embedding, the accuracy of entity alignment is improved.

[0038] Data in knowledge graphs often carries semantic information, appearing in the form of words. Based on these words and some empirical judgments, it's possible to roughly infer whether some entity pairs are aligned. For example, "AcquiredImmune DeficiencySyndrome" and "AIDS". While high-accuracy alignment cannot be achieved entirely through words, the proper use of this information can certainly improve the effectiveness of entity alignment. Based on this consideration, this invention introduces word embeddings for entity names in knowledge graphs.

[0039] This invention uses a large-scale pre-trained language model with a transformer-based bidirectional encoder representation unit for word embedding. However, after extensive research, it was found that the untuned transformer-based bidirectional encoder representation unit model does not perform well in text similarity calculation tasks. Therefore, fine-tuning was performed when setting the transformer-based bidirectional encoder representation unit model. The word embeddings of the transformer-based bidirectional encoder representation unit exhibit a cone-shaped distribution in space, with high-frequency and low-frequency words located in different regions. High-frequency words are closer to the origin (the mean of all word embeddings), while low-frequency words are farther from the origin. This distribution characteristic makes the similarity calculation between high-frequency and low-frequency words unable to represent their semantic similarity. Moreover, low-frequency words are not sufficiently trained and are sparsely distributed, resulting in semantically incomplete areas in this region, which also leads to problems with the calculated similarity.

[0040] This invention possesses a powerful embedding-based entity alignment paradigm and the ability to identify isomorphic subgraphs. However, in different knowledge graphs, corresponding entities often have non-isomorphic neighborhood structures, which can easily lead to different representations. To address this issue, this invention employs a graph attention network with a gating mechanism, designed to capture more diverse neighborhood structure information and highlight useful neighbor nodes. Then, direct and long-distance neighborhood information is aggregated using the gating mechanism. To effectively utilize word embeddings and structural embeddings, this invention combines them into entity embeddings using feature linear modulation. The model also incorporates a relation loss function to improve entity representation.

[0041] To apply the model in practice, this invention implements the process of fusing three published biomedical knowledge graphs. The fused knowledge graph is then published through a knowledge graph application platform. This platform is a web application developed using an open-source web application framework and can be accessed directly through a browser. The platform provides functions such as graph retrieval, intelligent question answering, and entity recommendation.

[0042] Transformer-based bidirectional encoder representation units are also a type of pre-trained language model. The model architecture of this invention is a connection of multiple Transformer encoders, each encoder having two sub-layers: one is a multi-head attention layer, which uses multiple self-attention mechanisms to learn the internal relationships between words in a sentence; the other is a feedforward graph attention network layer, which includes two linear transformations and an activation function. Each sub-layer has a residual connection module.

[0043] The core of the transducer-based bidirectional encoder representation unit is the multi-head attention mechanism. Attention, as the name suggests, is about focusing attention on the important parts of a problem. In practice, attention mechanisms have the advantages of few parameters, parallelizability, and significant effectiveness. The multi-head attention mechanism performs linear transformations on the input matrices Q, K, and V, which are composed of word vectors, using multiple sets of different matrices, and finally concatenates the different attention results. Furthermore, because the three input matrices are identical, this attention mechanism is also called a self-attention mechanism.

[0044] The input of the bidirectional encoder representation unit model based on the transformer consists of three parts: First, the token embedding, where the input sentence or sentence pair is segmented into multiple tokens using the WordPiece algorithm. For classification tasks, a "CLS" token is added at the beginning of the sentence, and for sentence pair tasks, a "SEP" token is added between the two sentences. Then, the segmented tokens and the added tokens are embedded together. Second, the segment embedding, which is used to distinguish between two sentences when performing sentence pair tasks. Third, the position embedding, which is generated by a set of rules in the model.

[0045] The transformer-based bidirectional encoder representation unit model is pre-trained using two unsupervised prediction tasks: masked language modeling and next-sentence prediction. The masked language model, designed for single-sentence tasks, randomly masks parts of the input words within a sentence and then predicts those masked words. During training, a marker is randomly selected in each training sequence with a 15% probability for prediction. The selected marker has an 80% chance of being relabeled as "MASK," a 10% chance of being a random marker, and a 10% chance of being the original marker. This training method enables the model to predict masked words based on context, allowing word vectors to reflect the relevance between words and have different representations in different contexts.

[0046] Many natural language processing tasks require understanding the relationship between two sentences, such as automatic question answering. The next sentence prediction task is designed for sentence-pair level tasks. The specific approach is as follows: each training example consists of sentence A and sentence B. 50% of sentence B is the next sentence of sentence A, and the remaining 50% of sentence B is a random sentence. Then, the training examples are input into the model for binary classification prediction.

[0047] By pre-training the transducer-based bidirectional encoder representation unit on a large-scale corpus using these two tasks, a highly generalizable pre-trained language model can be obtained. When used on different natural language processing tasks, it can achieve excellent results with simple fine-tuning.

[0048] The study of words is a key focus in the field of Natural Language Processing (NLP). Because words are small in granularity, and sentences are composed of words, which in turn form paragraphs, texts, and documents, much research in NLP is built upon this foundation. When studying words, the first problem to solve is their representation. Taking part-of-speech tagging as an example, the traditional machine learning approach typically uses (x, y) to represent a sample, where x is the word and y is the part-of-speech tag, constructing a mapping y = f(x). However, the mathematical model f used in this process (as shown in attention networks and support vector machines) takes numerical inputs, while words are abstract symbols used by humans to express and convey information. These cannot be directly input into the model, requiring a suitable method to convert words into numerical values, or embed them into a vector space. This method of embedding words into a dense vector space is called word embedding. This invention represents words as dense vectors in a low-dimensional space, facilitating use in various downstream tasks.

[0049] The word embedding model in this invention has two forms: one is called inferring the middle word from the context, and the other is called inferring the context word from the middle word. To reduce computational cost, the core idea is to fit the information in the co-occurrence matrix through an objective function, so that words are expressed as word vectors containing statistical information from the co-occurrence matrix. These vectors have certain semantic properties and can be used for natural language processing tasks, such as inferring the semantic similarity between words by calculating the Euclidean distance or cosine similarity between two word vectors. Each word corresponds to a vector, and these word vectors are static and cannot handle polysemous words. For example, the word "Apple" may be the name of a fruit and a company in different contexts.

[0050] Graph attention network (GNN) models extract features from entities and relations using graph attention networks, then perform certain operations on these features to determine the credibility of fact triples. With the development of GNNs, traditional Euclidean graph attention network models have been transferred to graph data modeling, automatically learning and extracting graph data features in an end-to-end manner. Graph convolutional networks (GCNNs) are a popular research direction within GNNs. GCNN models mainly include two categories: spectral methods based on the convolution theorem and spatial methods based on neighbor aggregation. Any GCNN can be written as a nonlinear function. However, knowledge graphs are complex graph structures, typically containing different types of entities and relations, and these relations are directed. GCNNs were proposed based on homogeneous graphs with only one type of node and relation. Clearly, traditional GCNNs cannot fully utilize the information in knowledge graphs to represent their entities and relations.

[0051] This invention uses an attention mechanism to highlight the influence of the most important parts of the data. In the graph attention network of this invention, the attention mechanism is used during the aggregation process to calculate the weight of each neighboring node, thereby highlighting the information of important nodes. Graph attention networks are spatial graph convolutional networks. The attention mechanism of this invention is used to determine the weights of a node's neighborhood when aggregating feature information. Graph attention networks have the following advantages: 1. Computational efficiency; computation can be parallelized. 2. Compared to graph convolutional networks, assigning different weights to nodes in the same neighborhood can expand the model scale. 3. Shared model weights. 4. No need for a fixed sampling size; the algorithm can process the entire neighborhood. 5. Calculating attention weights using node features, rather than the structural characteristics of nodes, allows calculation without knowing the graph structure.

[0052] Different knowledge graphs, with varying methods of knowledge collection, focuses, and sources, result in real-world entities having different names, neighbor structures, or attributes across different knowledge graphs. Entity alignment aims to fuse these heterogeneous knowledge graphs. A transformer-based bidirectional encoder representation unit is set up. This unit constructs word embeddings of entity names into vectors with a mean of 0 and a covariance matrix equal to the identity matrix. Word embeddings utilize language models and representation learning in natural language processing to embed a high-dimensional space containing the number of words into a low-dimensional continuous vector space. Each word or phrase is mapped to a vector in the real number domain. Then, a large-scale pre-trained language model within the representation unit of the transformer-based bidirectional encoder obtains a set of word embeddings for N entity names from the two knowledge graphs. Where N is a natural number, i is a natural number from 1 to N, and x i This represents a single word embedding. Then, the mean μ and covariance matrix Σ of the word embedding set are calculated. Next, singular value decomposition is performed based on a pre-defined value of k, where k is a natural number. The mean μ and covariance matrix Σ are calculated using Formula 1: mean μ... covariance matrix Here, T represents transpose; and to reduce the potential information learned in large-scale pre-trained language models and mitigate the impact of entity name bias, different targeted weight files are used for different datasets;

[0053] When polysemy and identical names occur, it is determined whether the word embedding set obtained by the transformer-based bidirectional encoder representation unit model is also the same. If no contextual information is provided to the transformer-based bidirectional encoder representation unit when polysemy and identical names occur, the structural information of the knowledge graph is used to further improve the entity alignment effect.

[0054] This method extracts neighbors with a certain degree of similarity from aligned entities in two knowledge graphs. A graph attention network is used in the structural embedding model of the representation unit in the transformer-based bidirectional encoder to integrate relational information. Then, it aggregates the semantic association information of direct and distant neighbors, determining whether semantic relationships might exist among the aligned entities' direct or distant neighbors. The graph attention network is a spatial graph convolutional network; its attention mechanism is used to determine the weights of node neighborhoods when aggregating feature information.

[0055] To expand the effective neighbors for more jumps, a gate mechanism network is used in the representation unit of the transformer-based bidirectional encoder to combine first-order and second-order neighbors, thereby expanding the neighbor entities of more orders. This jump connection accelerates the training effect of the structural embedding model in the representation unit of the transformer-based bidirectional encoder after using a graph attention network.

[0056] The role of the gate mechanism network is to aggregate second-order neighbors based on the structural embedding model in the representation unit of the bidirectional encoder of the transformer to reduce the noise caused by the differences in first-order neighbors when the first-order neighbors of the aligned entities are the same or not completely the same in different knowledge graphs.

[0057] When different specific entities are the central entity and one of its first-order neighbor entities in the knowledge graph, and the first attention weight is set to represent the different associations between the central entity and one of its first-order neighbor entities, and through nonlinear transformation, the first result of the output of the structural embedding model in the representation unit of the bidirectional encoder based on the transformer can approximate the nonlinear function, thereby handling more complex tasks; in order to make the first attention weight comparable between different entities, normalization must be performed before comparison.

[0058] When different specific entities, namely the central entity and one of its second-order neighbor entities in the knowledge graph, are used, a second attention weight is set to represent the different associations between the central entity and the second-order neighbor entity. Through nonlinear transformation, the second result of the output of the structural embedding model in the representation unit of the bidirectional encoder based on the transformer can approximate the nonlinear function, thereby handling more complex tasks. In order to make the second attention weights comparable between different entities, normalization is required before comparison.

[0059] Using two matrices respectively sum matrix Two matrices are used for the transformation between the central entity and its neighboring entities. sum matrix Initialization begins with two matrices. sum matrix The elements in the model change during the training process, so that the input and output of the structured embedding model meet the system requirements.

[0060] The embedding h of entity S that combines information from first-order and second-order neighbors u The calculation is as shown in Formula 2:

[0061] h u =g(h i,2 )·h i,1 +(1-g(h i,2 ))·h i,2 Formula 2;

[0062] Where h i,1 and h i,2 These refer to the embeddings obtained from the aggregation of first-order neighbors and second-order neighbors, respectively, and the functional function g(h) i,2 )=σ(Mh i,2 In +b), σ is the activation function, which introduces a nonlinear factor into the graph attention network, allowing the graph attention network to arbitrarily approximate any nonlinear function. M and b are the weight matrix and bias vector, respectively. The weight matrix M is initialized with a uniform distribution, and the bias vector b is initialized with all zeros. The elements in both the weight matrix and the bias vector change during the training process, so that the input and output of the structure embedding model meet the requirements.

[0063] The first feature linear modulation layer is used to combine word embedding and structural embedding. The specific calculation method of the first feature linear modulation is shown in Formula 3 below:

[0064] h u,e =FiLM(h u,s ) = h u,w W1 eh u,s +h u,w W2, formula three;

[0065] Where h u,s The structural embedding of entity u, h u,w The word embedding represents entity u; ⊙ represents the Hadamard product operation, a matrix operation; W1 and W2 are two matrices initialized using a specific uniform distribution. This specific uniform distribution is used to maintain the consistency of activation values ​​and gradients across layers of the graph attention network during propagation; FiLM(h u,s The ) represents the characteristic linear modulation method in graph attention networks, and the final output h of the characteristic linear modulation method is... u,e This is the final entity embedding, and the aligned entity pairs are obtained by calculating cosine similarity.

[0066] In addition, for a given set of alignment seed entity pairs, when the two entities in the alignment entity pair have different neighborhood structures due to coming from different knowledge graphs, in order to make the most of the information in the alignment seed entity pair, the neighborhood structure of the entities is expanded before the alignment task begins. That is, the triples in the alignment seed entity pair contained in the two knowledge graphs complement each other, so as to maximize the ability of the graph attention network to identify isomorphic subgraphs. Furthermore, through more similar neighborhood structures, it is easier for the entity pairs in the alignment seed set to obtain the same embedding representation. Thus, after propagation through the graph attention network, the beneficial influence factors are extended to the embedding of the entire knowledge graph, resulting in more accurate results.

[0067] The preconditions for step six are set as follows: An alignment loss function and a relational loss function are defined. The difference between the forward computation result and the true value in each iteration of the graph attention network is calculated using these functions, thus guiding the next training step in the correct direction. Samples are substituted into the large-scale pre-trained language model in the representation unit of the transformer-based bidirectional encoder to calculate the output prediction value. The error between the prediction value and the true value is calculated using the alignment loss function and the relational loss function. Based on the derivatives of the alignment loss function and the relational loss function, the error is backpropagated along the direction of minimum gradient to correct the trainable matrices and vectors in the forward computation formula. Iteration stops when the loss values ​​of the alignment loss function and the relational loss function reach a satisfactory value. At this point, the training of the large-scale pre-trained language model in the representation unit of the transformer-based bidirectional encoder is complete.

[0068] The alignment loss function L is designed by reducing the distance between positive entities and increasing the distance between negative entities. a :

[0069]

[0070] Among them G + and G -Let (m, n) represent the sets of positive and negative entity pairs, respectively. A positive entity pair refers to two aligned entities in two knowledge graphs. (m, n) represents any positive entity pair where m and n are entities from different knowledge graphs, representing the same thing in the real world but originating from different knowledge graphs. A negative entity pair refers to two misaligned entities in two knowledge graphs. (m', n') represents any negative entity pair where m' and n' are entities from different knowledge graphs, representing different things in the real world. The method for generating negative examples involves randomly replacing one of the aligned entity pairs in the training set. μ and λ are hyperparameters greater than 0. Hyperparameters are parameters set before the training process of a large-scale pre-trained language model in the representation unit of the bidirectional encoder based on the transformer. Here, μ = 0.1 and λ = 1.5. λ can make the distance of negative examples greater than a suitable value, thus better identifying potentially aligned entities. ||·|| represents the second norm of the vector, and ||h m' -h n' || represents the second norm for extracting entity embeddings into vectors from two different knowledge graphs m' and n', max(λ-||h) m' -h n' ||,0) represents the value λ-||h m' -h n' The maximum value between || and 0. If the value is 0, it indicates that the negative instance does not exist, thus avoiding negative values ​​during calculation.

[0071] To incorporate relational information into the graph attention network, a relational loss function is defined as follows:

[0072]

[0073] Where R represents the set of all relations, T r Let |T| represent the set of triples (h, r, t) formed by relation r, where |T| r | represents set T r The number of elements in h h The entity embedding of entity h in the triple, h t The entity embedding and relation embedding h represent the entity embedding of entity t in the triple. r Embedded by entity h h and h t The calculation yielded:

[0074]

[0075] The relation loss function reduces the increase in parameters brought about by the introduction of specific relation embeddings, and when the knowledge graph is used as an entity alignment dataset and does not provide relation alignment-related information, it mitigates the effect of inappropriate relation alignment methods leading to unsatisfactory alignment results.

[0076] During training, it is necessary to optimize both the alignment loss function and the relation loss function simultaneously. This is achieved by setting a hyperparameter α to combine the two loss functions into the final loss function: L = L a +αL r ;

[0077] The ultimate goal of entity alignment is to find a set of aligned entity pairs. Aligned entities are obtained by calculating the similarity of entity vectors. During training, the magnitude of the entity vectors is restricted to 1, so that only inner product operations are needed when calculating cosine similarity. In the matrix formed by the entity vectors, matrix multiplication is required to obtain the cosine similarity of all aligned entity pairs. The specific calculation is as follows: Here, sim(h1,h2) represents the calculation of cosine similarity; · is the inner product operation of entity vectors, h1 and h2 represent entity embeddings from different knowledge graphs, and ||h1||||h2|| represents the multiplication of the modulus of h1 and h2, which is 1; then the cosine similarity matrix is ​​constructed.

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the claims. Furthermore, the above description should be understood and implemented by those skilled in the art; therefore, any equivalent modifications made based on the disclosure of this invention should be included within the scope of these claims.

[0079] Beneficial Results: This invention provides a computer-constructed embedded alignment method that alternately discovers new alignment entities and corrects existing alignment entities through an iterative process. Alignment entities are discovered by comparing the similarity between two entity strings, while alignment entities are corrected using a greedy algorithm and a designed reasoning process. In practice, this invention aligns three publicly available biomedical knowledge graphs and merges them into a larger-scale knowledge graph. By developing a knowledge graph application platform, this knowledge graph can be made available to other researchers.

Claims

1. A computer-implemented method of embedded alignment, the method comprising: Step 1: Set up a transformer-based bidirectional encoder representation unit. The function of the transformer-based bidirectional encoder representation unit is to construct word embeddings of entity names into vectors with a mean of 0 and a covariance matrix that is the identity matrix. Word embedding utilizes language models and representation learning in natural language processing to embed a high-dimensional space with the number of all words into a low-dimensional continuous vector space. Each word or phrase is mapped to a vector in the real number field. Then, the word embedding set of N entity names from two knowledge graphs is obtained through a large-scale pre-trained language model in the representation unit of the transformer-based bidirectional encoder. Where N is a natural number, and j is a natural number from 1 to N. This represents a single word embedding. Then, the mean μ and covariance matrix Σ of the word embedding set are calculated. After singular value decomposition, matrix slicing is performed based on a pre-defined value of k, where k is a natural number. The mean μ and covariance matrix Σ are calculated using Formula 1. ; ;here This indicates transposition; and to mitigate the impact of entity name bias by reducing the potential information learned in large-scale pre-trained language models, different targeted weight files are used for different datasets. Step 2: When polysemy and name homonymy occur, determine whether the word embedding sets obtained through the transformer-based bidirectional encoder representation unit model are also the same. If no contextual information is provided to the transformer-based bidirectional encoder representation unit when polysemy and name homonymy occur, proceed to Step 3, which uses the structural information of the knowledge graph to further improve the entity alignment effect. Step 3: Extract similar neighbors between aligned entities in the two knowledge graphs. Then, use a graph attention network to integrate relational information into the structural embedding model of the representation unit in the transformer-based bidirectional encoder. Next, determine whether semantic association information exists among the direct or distant neighbors of the aligned entities, and then aggregate the semantic association information of direct and distant neighbors. The graph attention network is a spatial graph convolutional network. When aggregating feature information, the attention mechanism of the graph attention network uses an attention mechanism representation to determine the weights of the node's neighborhood. Step 4: To expand the effective neighbors across multiple hops, a gate mechanism network is used in the representation unit of the transformer-based bidirectional encoder to combine first-order and second-order neighbors, thereby expanding the neighbor entities from second-order to multi-order. This connection via the effective neighbors across hops accelerates the structural embedding model in the representation unit of the transformer-based bidirectional encoder, improving the training performance of the structural embedding model after using a graph attention network. The role of the gate mechanism network is to aggregate second-order neighbors when the first-order neighbors of aligned entities are completely or not completely the same in different knowledge graphs, in order to reduce the noise caused by the differences in first-order neighbors. This is based on the structural embedding model in the representation unit of the bidirectional encoder of the transformer to perform second-order neighbor aggregation. When different specific entities are the central entity and one of its first-order neighbor entities in a knowledge graph, and a first attention weight is set to represent the different associations between the central entity and its first-order neighbor entity, a nonlinear transformation is used to make the first result of the output of the structural embedding model in the representation unit of the bidirectional encoder based on the transformer approximate a nonlinear function, thereby handling complex tasks; in order to make the first attention weight comparable between different entities, normalization is performed before comparison; When different specific entities, namely the central entity and one of its second-order neighbor entities in the knowledge graph, are considered, a second attention weight is set to represent the different associations between the central entity and the second-order neighbor entity. Through nonlinear transformation, the second result of the output of the structural embedding model in the representation unit of the bidirectional encoder based on the transformer can approximate the nonlinear function, thereby handling complex tasks. In order to make the second attention weights comparable between different entities, normalization is performed before comparison. Using two matrices respectively and Two matrices are used for the transformation between the central entity and its neighboring entities. and Initialization begins with two matrices. sum matrix The elements in the model change during the training process, thereby ensuring that the input and output of the structural embedding model in the representation unit of the transformer-based bidirectional encoder meet the system requirements.

2. The computer-implemented method of claim 1, wherein Embedding of entities s that combines information of first order neighbors and second order neighbors Compute as Equation Two: Equation Two; in and These refer to the embeddings obtained from the aggregation of first-order neighbors and second-order neighbors, respectively, and the functional groups. middle, It is an activation function used to introduce nonlinearity into graph attention networks, allowing the network's functionality to arbitrarily approximate any nonlinear function. M and b These are the weight matrix and the bias vector, respectively. M Initialize the bias vector using a uniform distribution. b Using all-zero initialization, the elements in both the weight matrix and the bias vector change during training, thus ensuring that the input and output of the structure embedding model in the representation unit of the transformer-based bidirectional encoder meet the requirements; Step 5: Use the first feature linear modulation layer to combine word embedding and structure embedding. The specific calculation method for the first feature linear modulation is shown in Formula 3 below: Equation Three; in Representative Entity u Structural embedding, Representative Entity u Word embedding, The Hadamard product operation is a matrix operation. W 1 and W 2 are two matrices that have been initialized using a specific uniform distribution, which is used to keep the activation values ​​and gradients of each layer of the graph attention network consistent during propagation; The characteristic linear modulation method in a graph attention network, and the final output of the characteristic linear modulation method. This is the final entity embedding, and aligned entity pairs are obtained by calculating cosine similarity. Step Six: In addition, for a given set of alignment seed entity pairs, when the two entities in the alignment entity pair come from different knowledge graphs and the neighborhood structures of the different knowledge graphs are different, in order to make the most efficient use of the information in the alignment seed entity pair, before the alignment task begins, the neighborhood structure of the entities is expanded. This makes the triples in the alignment seed entity pair in the two knowledge graphs complementary, thereby enhancing the graph attention network's ability to identify isomorphic subgraphs. Furthermore, through similar neighborhood structures, the entity pairs in the alignment seed set obtain the same embedding representation, so that after processing and propagation by the graph attention network, the influence factor is extended to the embedding of the entire knowledge graph. The preconditions for step six are set as follows: An alignment loss function and a relational loss function are defined. These functions are used to calculate the difference between the forward computation result and the true value in each iteration of the graph attention network, thus guiding the next training step in the correct direction. Samples are substituted into the large-scale pre-trained language model in the representation unit of the transformer-based bidirectional encoder to calculate the output prediction value. The error between the prediction value and the true value is calculated using the alignment and relational loss functions. Based on the derivatives of the alignment and relational loss functions, the error is backpropagated along the direction of minimum gradient to correct the trainable matrices and vectors in the forward computation formula. Iteration stops when the loss values ​​of the alignment and relational loss functions reach a satisfactory value. At this point, the training of the large-scale pre-trained language model in the representation unit of the transformer-based bidirectional encoder is complete. Step seven, design the alignment loss function by reducing the distance between positive example entities and increasing the distance between negative example entities : ; in and These represent the sets of positive and negative entity pairs, respectively. A positive entity pair refers to two aligned entities in two knowledge graphs. Represents any pair of positive entities. m and n They are entities from different knowledge graphs. m and n Two entities represent the same thing in the real world, but come from different knowledge graphs; a negative entity pair refers to two misaligned entities in two knowledge graphs. Used to represent any pair of negative instance entities. and These are two entities from different knowledge graphs. and The two entities represent different things in the real world; the method for generating negative examples involves randomly replacing one of the entities in an already aligned entity pair in the training set. and These are hyperparameters greater than 0. Hyperparameters are parameters set before the training process of a large-scale pre-trained language model in the representation unit of the bidirectional encoder based on the transformer. , , It can identify aligned entities by recognizing negative examples whose distance is greater than a certain value. This represents the calculation of the second norm of a vector. This indicates the computation of two different knowledge graphs. and The entity embedding is extracted into the second norm of the vector. Indicates the value The maximum value between 0 and 0. If the value is 0, it indicates that the negative instance does not exist, thus avoiding negative values ​​during calculation. Step 8: To introduce relational information into the graph attention network, define the relational loss function: ; in R Represents the set of all relations. Indicates the relation r The triplet formed The set, Represents a set The number of elements in the middle, The entity embedding of entity h in the triple, Represents the entities in the triplet t Entity embedding, relation embedding Embedded by entities and The calculation yielded: ; The relation loss function reduces the increase in parameters introduced by specific relation embeddings, and when the knowledge graph is used as an entity alignment dataset and does not provide relation alignment-related information, it mitigates the effect of inappropriate relation alignment methods leading to unsatisfactory alignment results. In the training process, both alignment loss function and relationship loss function need to be optimized, by setting a hyperparameter The two loss functions are combined to become the final loss function: ; The ultimate goal of entity alignment is to find a set of aligned entity pairs. Aligned entities are obtained by calculating the similarity of entity vectors. During training, the magnitude of the entity vectors is restricted to 1, so that only inner product operations are needed when calculating cosine similarity. In the matrix formed by the entity vectors, matrix multiplication is required to obtain the cosine similarity of all aligned entity pairs. The specific calculation is as follows: ; in, This indicates that cosine similarity calculation is being performed; It is the inner product operation of entity vectors. and Represents entity embeddings from different knowledge graphs. express and Multiply the modulus of the matrix by 1; then construct the cosine similarity matrix.

Images