Semantic-aware embedding based open knowledge graph link prediction method

Abstract

The application relates to an open knowledge graph link prediction method based on semantic perception embedding. The application focuses on how to utilize the internal semantic information of NP and RP, and proposes a SeAE model. Firstly, the application can adaptively select the number of embedding partition blocks according to a data set, and update the embedding of NP in parallel by using the corresponding block embedding of the neighbor NP, so that the entanglement of the multiple semantics contained in the NP is disentangled, and an adaptive disentangled embedding (ADE) is obtained. Then, the application integrates the attention mechanism into the GRU encoder to obtain the focus and importance distribution of the RP coding, so that the semantic information in the RP can be better captured, and the explainability of the model can be improved. Finally, the application also proposes a relationship gate to extract the RP semantic features of the tail NP from the shared edge information, so that the interaction between the entity and the relationship is enhanced, and the semantic constraint of the relationship on the entity is realized. Finally, link prediction is carried out on multiple public open knowledge graph data sets, and the experimental results prove the superiority of the model.

Description

Technical Field

[0001] This invention relates to an open knowledge graph link prediction method based on semantically aware embedding. Background Technology

[0002] Link prediction is the core task of knowledge graph completion, in which missing facts are predicted based on known entities in the knowledge graph, which is crucial for downstream applications. Non-neural models embed entities and relations into the latent space through linear operations. TransE[1] is the pioneering and most representative translation model, which treats relations as translations from head entity to tail entity, i.e., head+relation≈tail. Later, a series of models were proposed, such as TransH[2], TransR[3] and TransD[4]. RotatE[5] extends the translation model to the complex space, defining each relation as a rotation from the source entity to the target entity in the complex space. There are also models that use tensor decomposition to calculate the rationality of triples. For example, RESCAL[6] and DistMult[7] represent each relation with a full-rank matrix and a diagonal matrix, respectively. ComplEx[8] extends DistMult to the complex space. These models are simple and intuitive, but due to their simple operation and limited parameters, they usually only produce low-quality embeddings.

[0003] Neural network-based models have been introduced into the field of knowledge graph completion due to their inherent powerful learning capabilities. ConvE[9] uses 2D convolutions to learn deep features of entities and relations. ConvKB

[10] uses one-dimensional convolutions and inputs the entire triple into a convolutional neural network. InteractE

[11] increases feature interaction by introducing recurrent convolutions. R-GCN

[12] is an extension of graph convolutional networks with relations

[13] . CompGCN

[14] embeds entities and relations together into the knowledge graph by combining operators. NBFNet

[15] and RED-GNN

[16] use GNNs to propagate queries stepwise in the knowledge graph for knowledge graph reasoning. DisenKGAT

[17] combines the idea of ​​disentanglement with graph attention networks and uses mutual information to enhance the independence between embedded components. Recently, capsule networks

[18] , Transformers

[19]

[20] , language models

[21]

[22] , and generative adversarial networks

[23]

[24] have also been introduced into the field of knowledge graph completion.

[0004] Some studies emphasize the improvements that relations can bring to the model, such as embedding quality and entity transformation methods. TransR

[25] uses matrices to project entities into a relation-specific space. TransRHS

[26] improves embedding quality by incorporating relation hierarchy information into the embedding. PairRE

[27] represents relations as two vectors and acts on the head entity and tail entity respectively, and models the connection between the head entity and tail entity by measuring the distance between the head entity and tail entity after the relation-specific transformation. Similarly, QuatRE

[28] adds two relation-specific quaternions to QuatE

[29] , namely the quaternion embedding of the rotated head and tail entities, to better model the correlation between the head and tail. HRQE

[30] represents relations as 3D rotations. TRPE

[31] establishes type-relation pairs to control entity representation transformation. DeepER

[32] introduces relation-specific rotation-reflection through a hypernet architecture to preserve relation-specific information in the embedding while maintaining the rotation and reflection properties of the relation.

[0005] CaRE

[33] was the first to discuss the problem of OpenKnowledge Graph (OpenKG), namely, the existence of multiple noun phrases (NP) representing the same entity or multiple relation phrases (RP) representing the same relation, and the need to learn the embeddings of NP and RP nodes for downstream tasks. This model proposed a normalized embedding model CaRe for OpenKG, which introduces normalized information into the original knowledge graph embedding model to achieve more refined modeling of OpenKG when performing tasks such as link prediction. OKGIT

[34] improved OpenKG link prediction by using a new type compatibility scoring and type regularization based on CaRE, and used BERT

[35] to generate predicted NPs that are compatible with query types. CUVA

[36] is based on variational autoencoders for the normalization of NP and RP in OpenKG, and obtains better phrase vector representations by jointly learning embeddings and cluster assignments. Such a model is more suitable for clustered mentions because its soft clustering ability can explain the different meanings of a given entity mention (polysemous words), and is mainly dedicated to solving the entity normalization problem in open knowledge graphs. OKGSE

[37] aims to fully utilize the differential information of each NP within a cluster in OpenKG, as well as the neighborhood information of the NP. OKGSE divides the embedding of the NP into two parts and learns the embeddings of the two independent parts of the cluster and the NP respectively. At the same time, it integrates the graph context as an independent module into its final scoring function to utilize the neighborhood information of the NP. TernaryCL

[38] is a contrastive learning framework for alleviating the sparsity problem in OpenKG. The framework is based on the ternary relation pattern between head entities, relations and tail entities, and has key ideas such as contrastive entities, contrastive relations, contrastive self, contrastive synonyms and contrastive fusion. It can effectively learn the embedding representation in OpenKG and aims to solve the problems of zero samples, few samples and synonyms.

[0006] OpenKG does not require a pre-defined ontology, making it highly adaptable for downstream tasks such as question answering and document classification. It is typically necessary to learn the embeddings of NPs and RPs that appear as nodes and edges in OpenKG. In recent years, significant efforts have been made in the normalization of OpenKG models, as well as in knowledge graph completion to alleviate sparsity, and in improving NP embeddings and utilizing auxiliary information to enhance link prediction performance. Knowledge graph completion models for OpenKG can perform link prediction tasks in unnormalized scenarios, but research in open knowledge graph link prediction is still far from complete, and model performance remains unsatisfactory.

[0007] Previous work failed to uncover the multiple semantics within NP embeddings, leading to semantic entanglement within the embeddings. Mining more semantic information from NPs is a breakthrough in addressing phrase unnormalization and ambiguous referential information; de-entanglement of different semantics within NPs is crucial for link prediction. Under the influence of (vilnius, is the most important city for, lithuanian), the query (vilnius, became the capital of,·) might be predicted as lithuanian. However, if we can distinguish between the semantics of lithuanian into "language" attributes and "human" attributes, we can eliminate the incorrect answer lithuanian and predict the correct answer lithuannia.

[0008] The impact of semantic information of relational pairs (RPs) on RP and NP embeddings is lacking. Most traditional knowledge graphs are ontology-based knowledge bases, where each entity and relation is represented by a globally unique identifier. However, the number of relation types is far fewer than the number of entity types, and each relation appears frequently as an edge connecting entities in the knowledge graph. Therefore, relations have a much greater chance of being trained, gaining information from numerous entity pairs to achieve better predictive performance. However, in OpenKG, which is obtained from open information extraction tools, the number of RPs and NPs is not significantly different, and each RP appears less frequently. Therefore, more attention needs to be paid to the encoding method of RPs to obtain reliable embeddings. Most existing OpenKG knowledge graph completion models do not focus on the semantic parsing of the RPs themselves, but instead directly initialize them randomly or use common encoders to encode RP embeddings. However, such RP embeddings cannot capture the key semantic information at different input positions. For example, for the triple (mumbai, is also the state capital of,·), generally only the important information "capital" in "is also the state capital of" can be captured. If this patent can capture another descriptive word "state" in the RP that is related to the fact, it can correctly predict "maharashtra" and will not be misled by results such as "india" which are strongly related but incorrectly predicted.

[0009] Furthermore, existing OpenKG knowledge graph completion models neglect the fact that the semantic information carried by the relation (RP) can also influence the features of the NP (NP). For example, the trailing NPs "rome" and "vienna" of the RP "study in" share the same feature "location / city". Trailing NPs of the RP "is author of", such as "earth democracy" and "brave new world", share the feature "work / book". Sets of trailing entities in some relations often exhibit a common feature. If the semantic constraints of relations on entity features can be extracted, NP embeddings can be made more accurate. Summary of the Invention

[0010] The purpose of this invention is to overcome the limitations of existing OpenKG models, which only extract shallow feature information of NPs and ignore the fact that an NP itself can have different semantic information. Furthermore, existing models generally do not make full use of the semantic information of relational phrases (RPs). Therefore, this invention provides an OpenKG link prediction method based on semantically aware embedding, which focuses on exploring how to utilize the inherent semantic information of NPs and RPs and proposes the SeAE model.

[0011] To achieve the above objectives, the technical solution of this invention is: an open knowledge graph link prediction method based on semantically aware embedding, which constructs an open knowledge graph link prediction model SeAE based on semantically aware embedding to achieve link prediction. The open knowledge graph link prediction model SeAE based on semantically aware embedding includes:

[0012] The prediction module embeds entities and relations into d-dimensional continuous vectors, where entities are randomly initialized and relations use pre-trained word vectors; each entity embedding is updated independently in blocks to obtain adaptive unentangled embeddings of entity embeddings; relation embeddings are updated into semantically biased embeddings by a gated recurrent unit (GRU) encoder with attention; the tail entity embeddings predicted by the base model (ConvE or other base models) are used to calculate the prediction loss by comparing the logistic prediction value of the activation function with the true tail entity labels.

[0013] Type compatibility module: The tail entity embeddings predicted by the BERT model and the true tail entities are processed through a linear layer to obtain type vectors; the distance between the two type vectors is used as part of the scoring function; the type compatibility loss is calculated by combining the predicted values ​​of the type vectors after the activation function with the true tail entity labels.

[0014] The gate filtering module: The tail entity embeddings predicted by the base model and the tail entity embeddings predicted by the BERT model interact with the relation embeddings respectively. The relation gate is input, and the distance between the two transformed tail entity embeddings is calculated as the gate filtering loss.

[0015] Loss calculation module: The final loss is obtained by weighted summation of three losses: prediction loss, type compatibility loss, and gate filtering loss.

[0016] In one embodiment of the present invention, the prediction module is specifically implemented as follows:

[0017] First, define: Open Knowledge Graph in It is a set of NP. It is a set of RP. It is a set of fact triples; a fact set The set of all fact triples in a knowledge graph, where (h,r,t) is a fact triple; noun phrase set. Let represent the set of all entities in the knowledge graph, and h represent the head noun phrase. t indicates a suffix noun phrase w is a word element. All lexical units are stored in the vocabulary, h L t L It refers to the number of lexical units in the head NP and tail NP; the set of relational phrases. This represents the set of all relation phrases in the knowledge graph. r L It refers to the number of lexical units in the RP; vocabulary. The lexical units that make up NP and RP are all included in the vocabulary. In the middle; entity embedding e h Represents the embedding of entity h, e h,j The embedding of entity h represents the embedding of the j-th part; the relation embedding r represents the embedding of relation r, r r Represents the embedding of relation r; Open knowledge graph link prediction, i.e., predicting the missing tail NP of triple (h,r,?) in the test set given a training set;

[0018] (1) Adaptive unentangled mode embedding

[0019] Calculate the number of blocks K for NP embeddings based on the average degree (deg) of the knowledge graph dataset:

[0020]

[0021] For all e contains all head NPh and tail NPt, and first randomly initializes all NP embeddings. Let d be the embedding dimension of NP, and then divide it into K parts, where K must be divisible by the embedding dimension. The j-th part = {1,2,…,K} is represented as By projecting each part of the NP embedding into a different subspace, and then updating each part of all NPs in a multi-layer neural network, that is, updating the embedding of the j-th part of the NP embedding in layer l-1, we obtain the j-th part of the NP embedding after l layers:

[0022]

[0023] in Represents the area surrounding NPe. f LAN This represents the neighborhood feature aggregation network (LAN), used to accumulate feature information of multi-hop neighbors. The AGGREGATE function can obtain information collected from the j-th block of the NP embeddings of the neighborhood of e. The function is defined as follows:

[0024]

[0025] The AGGREGATE function normalizes each block after it has been updated by layer l by dividing the NP embedding by the node degree of the NP. When pieced together, the final NPe embedding is as follows:

[0026]

[0027] (2) GRU encoder with attention mechanism

[0028] Every word in all RP The word embedding is mapped to a fixed-size d-dimensional word vector w, and the vectors of all words form a shared word embedding matrix. d is the embedding dimension of RP, which is the same as the embedding dimension of NP. It is the size of the vocabulary, and the word embedding matrix W. v Based on pre-trained word vectors, a GRU encoder is used to encode the RP embedding, and an attention mechanism is introduced to enable the encoder to weight the input according to the importance of each input position, focusing more on the important parts within the RP, thereby improving the information extraction capability; firstly, the word embedding of RP r is obtained through the GRU encoder:

[0029] w r,1 w r,2 , ...,w r，L =unpack(GRUEncoder(w r，1 ,w r，2 ,…,w r，L ))

[0030] Where L represents the sequence length, GRUEncoder represents the GRU encoder, and unpack represents the decompression operation, which decompresses the GRU encoding result into a vector of L words;

[0031] After calculating the attention weight for each encoded word using the attention mechanism, the attention weight is multiplied by each encoded word to obtain the weighted representation of each word. Then, these weighted representations of words are summed to obtain the weighted representation of the entire RP.

[0032]

[0033] Where w r,i At represents the embedding of the i-th word of RP r. r,i The attention weight of the i-th word in RPr is calculated using the following formula:

[0034]

[0035] The activation function σ is tanh, and W... a and b a These are all learnable parameters in a linear layer, r L It represents the number of lexical units in the RP.

[0036] (3) Basic model prediction

[0037] Head NP embedding using adaptive disentanglement RP embeddings encoded with attention-based GRU encoders r To predict tail NP embedding Using ConvE as the basic model for measuring ternary combinatorial rationality, the formula for calculating the predicted tail NP embedding is as follows:

[0038]

[0039] in and represent and r r The 2D reshaping is represented by *, where * represents the convolution operation, ω represents the convolution kernel, f represents a nonlinear function, vec represents the vectorization operation of the 2D tensor, and W is a matrix.

[0040] In one embodiment of the present invention, the type compatibility module is specifically implemented as follows:

[0041] First, based on the pre-trained BERT model, the embedding of the tail NP is returned for a given input triple (h,r,t):

[0042]

[0043] MASK is a special word that represents a placeholder.

[0044] Next, the latent type embedding of the tail NP is learned using the NP embedding projection mechanism:

[0045]

[0046]

[0047] Where Linear is the type encoding conversion function, τ is the type vector of the true tail NP, and τ B It is the type vector of the tail NP predicted by the BERT model.

[0048] In one embodiment of the present invention, the gate filtering module is specifically implemented as follows:

[0049] Design a relation-specific gating filtering mechanism—the relation gate—to capture the common semantic features in relation-specific entity embeddings; given NPt and RPr, the structural NP embeddings predicted by the base model are... The NP and RP embeddings are concatenated, and then a fully connected layer and an activation function are used to obtain a fused embedding vector, allowing for full semantic interaction between NP and RP. This embedding vector is considered a comprehensive representation combining NP and RP, with the output of the activation function and the original NP embedding being concatenated. Multiplication preserves RP-specific information, while entity names are used as auxiliary information to extract RP-specific NP features. Similarly, the tail NP embeddings predicted by BERT are processed in parallel. Thus, RP-specific text embeddings are obtained. RP Feature Enhancement NP Embedding The calculation formula is as follows:

[0050]

[0051]

[0052] Where [;] represents the concatenation operation, W1 and W2 are entity-related weight matrices, b1 and b2 are entity-related biases, ⊙ represents the Hadamard product operation, and the activation function σ is the sigmoid function. and These are the tail NP embeddings predicted by the base model and the tail NP embeddings predicted by the BERT model, respectively.

[0053] Compute the two NP embeddings with RP information after transformation. The distance between the two points is used as the loss, and there are two distance calculation formulas to choose from: Euclidean distance and cosine distance, as shown below:

[0054]

[0055] Where N is the number of all training triples, and ∈ is a small value to avoid division by zero, which is 1e-8.

[0056] In one embodiment of the present invention, the loss calculation module is specifically implemented as follows:

[0057] 1) Scoring function

[0058] The input to the scoring function of the prediction module includes the candidate tails obtained from (h,r). and the real tail NP e t The score φ obtained through the basic model prediction conve (h,r,t) is as follows:

[0059]

[0060] The type compatibility module's scoring function takes as input the type vector τ of the true tail NP and the type vector τ of the tail NP predicted by the BERT model. B Type compatibility score φ type The calculation of (h,r,t) is as follows:

[0061]

[0062] The final score is composed of the scores from the prediction module and the type compatibility module:

[0063] φ final =φ conve (h,r,t)+γ×φ type (h,r,t)

[0064] The relative weight of the γ control type compatibility score;

[0065] 2) Loss Function

[0066] The prediction loss is calculated using the binary cross-entropy (BCE) loss formula, as follows:

[0067]

[0068] Among them, y i It is a triple (h) i ,r i ,t i The tag ) if y iThe probability of a candidate fact being true is assigned a logical value based on the likelihood probability calculated using the scoring function, and this logical value serves as the prediction label. The calculation formula is as follows:

[0069]

[0070] The activation function σ is sigmoid, and φ final (h i ,r i ,t i ) represents (h i ,r i ,t i The final score;

[0071] The type compatibility loss is calculated using the binary cross-entropy (BCE) loss, as shown in the following formula:

[0072]

[0073] Among them, y i Still a triple (h) i ,r i ,t i The tag ) if y i If the value is 1, then it is 0; otherwise, it is the type prediction label. The calculation formula is as follows:

[0074]

[0075] The activation function σ is sigmoid, and φ type (h i ,r i ,t i ) represents (h i ,r i ,t i Type score;

[0076] The formula for calculating gate filter loss is as follows:

[0077]

[0078] By jointly optimizing three objectives, we capture RP-specific NP constraints, distinguish different attributes within NP, and utilize type compatibility information extracted by the BERT model to improve link prediction performance. The final loss calculation formula is as follows:

[0079]

[0080] α and β are hyperparameters used to adjust the proportion of different parts of the loss.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] 1. This invention employs a novel embedding method in the field of open knowledge graph link prediction. Based on the number of NPs and triples in the dataset, the NP embedding is split into multiple parts, and the NP embedding is updated using a local averaging network. An adaptive disentanglement embedding mechanism is used to mine multiple semantic meanings inherent in each NP.

[0083] 2. Capture the importance of different lexical units in the RP through an attention mechanism and generate accurate RP embeddings.

[0084] 3. Design a gate mechanism to filter NP embeddings so that NP embeddings passing through relation gates retain RP-specific semantic information. Attached Figure Description

[0085] Figure 1 This is the overall architecture of the SeAE of this invention.

[0086] Figure 2 This describes the specific process of untangling in this invention. Detailed Implementation

[0087] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0088] This invention provides a method for predicting links in an open knowledge graph based on semantically aware embedding. It constructs a link prediction model SeAE based on semantically aware embedding to achieve link prediction. The SeAE model includes:

[0089] The prediction module embeds entities and relations into d-dimensional continuous vectors, where entities are randomly initialized and relations use pre-trained word vectors; each entity embedding is updated independently in blocks to obtain adaptive unentangled embeddings of entity embeddings; relation embeddings are updated into semantically biased embeddings by a gated recurrent unit (GRU) encoder with attention; the tail entity embeddings predicted by the base model (ConvE or other base models) are used to calculate the prediction loss by comparing the logistic prediction value of the activation function with the true tail entity labels.

[0090] Type compatibility module: The tail entity embeddings predicted by the BERT model and the true tail entities are processed through a linear layer to obtain type vectors; the distance between the two type vectors is used as part of the scoring function; the type compatibility loss is calculated by combining the predicted values ​​of the type vectors after the activation function with the true tail entity labels.

[0091] The gate filtering module: The tail entity embeddings predicted by the base model and the tail entity embeddings predicted by the BERT model interact with the relation embeddings respectively. The relation gate is input, and the distance between the two transformed tail entity embeddings is calculated as the gate filtering loss.

[0092] Loss calculation module: The final loss is obtained by weighted summation of three losses: prediction loss, type compatibility loss, and gate filtering loss.

[0093] The following is a detailed implementation process of the present invention.

[0094] The overall framework of the Open Knowledge Graph Link Prediction with Semantic-Aware Embedding (SeAE) model constructed by the method of this invention is as follows: Figure 1 As shown, it includes four main modules: (1) Prediction module: embeds entities and relations into d-dimensional continuous vectors, where entities are randomly initialized and relations use pre-trained word vectors. Each entity embedding is updated independently in blocks to obtain adaptive unentangled embeddings of entity embeddings. Relation embeddings are updated to semantically biased embeddings by a gate recurrent unit (GRU) encoder with attention. The tail entity embeddings predicted by ConvE or other base models are compared with the logistic prediction value of the activation function and the real tail entity labels to calculate the loss; (2) Type compatibility module: the tail entity embeddings predicted by BERT and the real tail entities are processed by linear layers to obtain type vectors respectively. The distance between the two type vectors is used as part of the scoring function. The type vector is calculated by the predicted value of the activation function and the real tail entity label to obtain the type compatibility loss; (3) Gate filtering module: the tail entity embedding predicted by ConvE and the tail entity embedding predicted by BERT interact with the relation embedding respectively, input the relation gate, and then calculate the distance between the two transformed tail entity embeddings as part of the loss; (4) Loss calculation module: the final loss is obtained by weighted summation of the prediction loss, type compatibility loss and gate filtering loss.

[0095] The relevant definitions of this invention are given below.

[0096] Definition 1 (Open Knowledge Graph) ) in It is a set of NP. It is a set of RP. It is a set of fact triples.

[0097] Definition 2 (Fact Set, ) Let (h,r,t) represent the set of all fact triples in the knowledge graph, where (h,r,t) is a fact triple.

[0098] Definition 3 (set of noun phrases) ) Let represent the set of all entities in the knowledge graph, and h represent the head noun phrase. t indicates a suffix noun phrase w is a word element. All lexical units are stored in the vocabulary. L t L It represents the number of lexical units in the head NP and tail NP.

[0099] Definition 4 (Set of Relational Phrase) ) This represents the set of all relation phrases in the knowledge graph. r L It represents the number of lexical units in the RP.

[0100] Definition 5 (Glossary) The lexical units that make up NP and RP are all included in the vocabulary. middle.

[0101] Definition 6 (Entity Embedding) e h Represents the embedding of entity h, e h,j This represents the embedding of the j-th part of entity h.

[0102] Definition 7 (Relational Embedding) r denotes the embedding of relation r, r r This represents the embedding of relation r.

[0103] Definition 8 (Open Knowledge Graph Link Prediction) predicts the missing tail NP of the triple (h,r,?) in the test set, given a training set.

[0104] 1SeAE Model Design

[0105] 1.1 Prediction Module

[0106] 1.1.1 Adaptive Unentangled Embedding

[0107] Adaptive unentangled module embedding is a method for handling NP embedding problems. Its unique feature is that it selects the number of embedding components K based on the average node degree of the training set. In this way, the method can effectively solve NP embedding problems under different network structures and datasets, exhibiting strong adaptability.

[0108] If the number of segmentation components K is small, the embedding size of each NP block may be too large, making it difficult for the model to handle semantically complex NPs. If K is large, although it is beneficial for the segmentation of latent semantics, some useful information may be lost. In addition, the larger K is, the more parameters of the neural network will increase, which will increase the cost of computation and storage. Therefore, in order to minimize costs while retaining sufficient information, K should be limited to a reasonable range. Since deg represents the relation density around the node, K should not exceed deg to ensure that each embedding block covers at least one relation, so that each embedding block can capture enough semantic information. At the same time, K should not be too small to avoid too much information overlap and redundancy, thereby losing the effect of deentanglement. Therefore, the approach of this patent is to calculate the number of NP embedding blocks K based on the average degree deg of the knowledge graph dataset, as shown in formula (1):

[0109]

[0110] Where deg is the average node degree of the dataset.

[0111] Figure 2 Describe the detailed process of understanding entanglement. For all (e includes all head NPh and tail NPt), this patent first randomly initializes all NP embeddings. Let d be the embedding dimension of NP. Then divide it into K parts, where K needs to be divisible by the embedding dimension. The j-th part = {1,2,…,K} is represented as This process projects each part of the NP embedding into a different subspace. Then, each part of the NP is updated within a multi-layer neural network. For example, the embedding of the j-th part of the NP embedding at layer l-1 is updated in the network, resulting in the j-th part of the NP embedding after l layers.

[0112]

[0113] in Represents the area surrounding NP e. This patent chooses a simple yet efficient Local Aggregation Network (LAN) to accumulate feature information of multi-hop neighbors. Other neural networks, such as graph attention networks or graph convolutional networks, can also be used. Given the arbitrary structure of knowledge graphs, considering only local structures, such as 1-hop neighbors or relational paths, is an effective method. However, it is also feasible to incorporate global graph structure information (e.g., multi-hop neighborhoods) into the fact embedding; this only requires adjusting the number of network layers l that update the NP embedding. The AGGREGATE function can obtain information collected from the j-th block of the NP embedding of the neighborhood of e. The function definition is as follows:

[0114]

[0115] The AGGREGATE function normalizes the NP by embedding the node degree of that NP. It then updates each block after layer l. When concatenated, the final NP e embedding is as follows:

[0116]

[0117] 1.1.2 GRU Encoder with Attention Mechanism

[0118] First, for each word in all the RPs This is mapped to a fixed-size d-dimensional word vector w. The vectors of all words form a shared word embedding matrix. d is the embedding dimension of RP, which is the same as the embedding dimension of NP. This refers to the size of the vocabulary. The word embedding matrix W v This patent uses GloVe, which can be constructed based on pre-trained word vectors. If a word has a pre-trained vector in GloVe, it is used directly; otherwise, it is randomly initialized. GRU improves the RNN's neurons by employing a gating mechanism, reducing computational complexity by controlling the flow and updates of information, and mitigating the gradient vanishing problem to some extent. Because GRU is simple, efficient, and less prone to overfitting, this patent uses a GRU encoder to encode the RP embedding.

[0119] However, this patent notes that if the feature vector output from the last time slice of the GRU encoder is directly used as the RP embedding, the RP embedding will carry more features from the last time slice, potentially diluting the importance of other time slices and failing to accurately reflect the true characteristics of the RP. Therefore, this patent introduces an attention mechanism, allowing the encoder to weight the input based on the importance of each input position, focusing more on the important parts within the RP, thereby improving information extraction capabilities. This patent first obtains the word embedding of RP r through the GRU encoder:

[0120]

[0121] Where L represents the sequence length, and unpack represents the decompression operation, which decompresses the GRU encoding result into a vector of L tokens.

[0122] After calculating the attention weight for each encoded word using the attention mechanism, the attention weight can be multiplied by each encoded word to obtain a weighted representation of each word. Then, these weighted representations are summed to obtain the weighted representation of the entire RP.

[0123]

[0124] Where w r,i This represents the embedding of the i-th word of RP r, att r,i The attention weight of the i-th word in RPr is calculated using the following formula:

[0125]

[0126] 1.1.3 Basic Model Prediction

[0127] This patent uses adaptive disentanglement head NP embedding. RP embeddings encoded with attention-based GRU encoders r To predict tail NP embedding This patent uses ConvE as the basic model for measuring ternary combinatorial rationality, and the formula for calculating the predicted tail NP embedding is as follows:

[0128]

[0129] in and represent and r r The 2D reshaping is represented by *, where * represents the convolution operation, ω represents the convolution kernel, f represents a nonlinear function, vec represents the vectorization operation of the 2D tensor, and W is a matrix.

[0130] 1.2 Type Compatibility Module

[0131] This patent demonstrates that a tail NP with type compatibility information can be obtained through a masked language model (MLM). The MLM in the SeAE model uses BERT. Simply put, given (h, r), although BERT may incorrectly predict the tail NP, the type of the incorrectly predicted tail NP is similar to the type of the correct tail NP.

[0132] To measure the type compatibility between the tail NP t and the NP-RP pair (h,r), this patent uses a scoring function. Specifically, given a (h,r) and t, the type compatibility scoring function is fed into the function. This function assigns a higher score to the tail NP with high type compatibility. Type constraints specific to (h,r) are used to reduce the reasonableness range of the test facts.

[0133] First, based on the pre-trained BERT model, the embedding of the tail NP is returned for a given input triple (h,r,t):

[0134]

[0135] Next, this patent uses an NP embedding projection mechanism to learn the latent type embedding of the tail NP:

[0136]

[0137]

[0138] Where Linear is the type encoding conversion function, τ is the type vector of the true tail NP, and τ B It is the type vector of the tail NP predicted by BERT.

[0139] 1.2 Door Filter Module

[0140] Gating mechanisms are widely used in the field of NLP. By using gating mechanisms, the flow of information in a neural network can be controlled, selectively allowing information to pass through a gate structure, enabling the network to better understand contextual information, thereby helping to improve the performance of language models.

[0141] Following the principle of parameter sharing, this patent designs a relation-specific gating filtering mechanism—the relation gate—to capture the semantic features common to relation-specific entity embeddings. Specifically, the relation gate first examines the semantic relevance between relations and entities. If highly relevant, the activation function outputs a value close to 1, preserving most of the information in the NP embeddings; otherwise, the output is close to 0, filtering out most irrelevant information after the Hadamard product. This gating mechanism enhances the interaction between entities and relations and plays a crucial role in restricting entity types to obtain more suitable NP embeddings, further enhancing the model's discriminative power. The relation gate process is as follows: Figure 1 As shown in the lower left corner.

[0142] Given NPt and RPr, this patent embeds the ConvE-predicted structural NP. The NP and RP embeddings are concatenated, and then passed through a fully connected layer and an activation function to obtain a fused embedding vector, allowing for full semantic interaction between NP and RP. This embedding vector can be viewed as a comprehensive representation combining NP and RP. The output of the activation function is then compared with the original NP embedding. Multiplication preserves RP-specific information. RP-specific structure embedding. The definition is shown in formula (12). This patent relies on entity names as auxiliary information to extract RP-specific NP features. In order to further improve the text embedding predicted by BERT, the tail NP embedding predicted by BERT is also processed in parallel. Thus, RP-specific text embeddings are obtained. RP Feature Enhancement NP Embedding The calculation formula is as follows:

[0143]

[0144]

[0145] Where [;] represents the concatenation operation. W1 and W2 are entity-related weight matrices, b1 and b2 are entity-related biases, ⊙ is the Hadamard product operation, and σ is the sigmoid activation function. and These are the tail NP embeddings predicted by the base model ConvE and the tail NP embeddings predicted by BERT, respectively.

[0146] This patent calculates the two types of NP embeddings with RP information after conversion. The distance between them is used as the loss. Two distance calculation formulas are available: Euclidean distance and cosine distance, as shown below:

[0147]

[0148] Where N is the number of all training triples, and ∈ is a small value to avoid division by zero, with a default value of 1e-8.

[0149] The primary purpose of relation gates is to capture relation-specific semantic features in entity embeddings, which can improve the model's accuracy in recognizing entities with specific relations, thereby enhancing the model's performance. Regardless of whether the dataset contains textual information, entities carry relation-specific semantics, so this component can help ensure consistent performance improvements across different datasets.

[0150] 1.3 Scoring Function

[0151] The input to the scoring function of the prediction module includes the candidate tails obtained from (h,r). and the real tail NP e t The score φ obtained through ConvE prediction conve (h,r,t) is as follows:

[0152]

[0153] The scoring function of the type compatibility module takes as input the type vector τ of the true tail NP and the type vector τ of the tail NP predicted by BERT. B Type compatibility score φt yp The calculation of e(h,r,t) is as follows:

[0154]

[0155] The final score is composed of the scores from the prediction module and the type compatibility module:

[0156] φ final =φ conve (h,r,t)+γ×φ type Formula (17) (h,r,t)

[0157] The relative weight of the γ control type compatibility score.

[0158] 1.4 Loss Function

[0159] This patent uses the following binary cross-entropy (BCE) loss to calculate the accuracy of the base triplet prediction. For binary classification problems, the BCE loss function calculates and minimizes the difference between the target value and the predicted value. For each sample, the BCE loss function converts the predicted value and the target value into probabilities, then sums their logarithms, and finally takes the negative of the result. The formula for calculating the prediction loss is as follows:

[0160]

[0161] Among them, y i It is a triple (h) i ,r i ,t i The tag ) if y i The probability is 1 if true and 0 otherwise. A logical value is assigned to the likelihood of a candidate fact being true based on the probability calculated using the scoring function; this logical value serves as the prediction label. The calculation formula is as follows:

[0162]

[0163] Where σ is the sigmoid activation function, φ final (hi ,r i ,t i ) represents (h i ,r i ,t i The final score.

[0164] In addition to incorporating type compatibility scores into the final scoring function, this patent includes a type matching term in its loss function to further strengthen the type constraint on the prediction results. This forces type vectors between correct triples in the type space to be closer. This patent also uses binary cross-entropy (BCE) loss to calculate type compatibility:

[0165]

[0166] Among them, y i Still a triple (h) i ,r i ,t i The tag ) if y i 1 if true, 0 otherwise. Type prediction label The calculation formula is as follows:

[0167]

[0168] Where σ is the sigmoid activation function, φ type (h i ,r i ,t i ) represents (h i ,r i ,t i (Type score)

[0169] The last part of the loss function is the gate filtering loss. The tail NP embeddings predicted by ConvE and BERT interact with the RP embeddings respectively, input into the relation gate, and then perform distance calculation between the two transformed tail NP embeddings as part of the loss. There are two choices for the distance calculation formula, with little difference in performance, and both can achieve good results. The calculation formula for the gate filtering loss is shown in formula (14).

[0170] By jointly optimizing three objectives, this patent captures RP-specific NP constraints, distinguishes different attributes within NP, and utilizes type compatibility information extracted by BERT, thereby improving link prediction performance. The final loss calculation formula is as follows:

[0171]

[0172] α and β are hyperparameters used to adjust the proportion of different parts of the loss.

[0173] 2 Experiments

[0174] 2.1 Experimental Setup

[0175] 21.1 Dataset

[0176] This patent evaluates the SeAE model on four datasets, including Reverb20K, Reverb45K, Reverb20KF, and Reverb45KF. The statistics for these four datasets are summarized in Table 1. Reverb20K and Reverb45K are two datasets of English open knowledge graphs constructed by the open information extraction tool Reverb

[39] . Predicting NPs composed of multiple lexical units using BERT is very challenging, so filtered subsets Reverb20KF and Reverb45KF were created by filtering multi-lexical NPs. These two filtered subsets contain only noun phrases with a single lexical unit. Table 2 shows the average node degree of the four datasets.

[0177] Table 1 Dataset Details

[0178]

[0179] Table 2 shows the statistics on the average node degree of the training set in the dataset.

[0180]

[0181] 2.1.2 Evaluation Indicators

[0182] This patent uses Mean Reciprocal Rank (MRR), Mean Rank (MR) Hit@1, Hit@3, and Hit@10 as evaluation metrics.

[0183] For each test sample, the model predicts a set of candidate entities (the answer). MRR considers the rank of the correct entity among these candidate entities. The MRR value is obtained by summing the inverse ranks of the correct entities across all test samples and dividing by the total number of test samples. A higher MRR value indicates a better model performance. Similarly, MR is calculated by summing the rank of the correct entities across all test samples and dividing by the total number of test samples. A higher MRR value indicates a better model performance.

[0184] Hit@N is a metric that measures whether the correct entity exists among the top N candidate entities predicted by the model. It is calculated as follows: for each test sample, if the correct entity is among the top N candidate entities predicted by the model, the Hit@N score for that test sample is 1; otherwise, it is 0. The final evaluation metric is the average of the Hit@N scores for all test samples. A higher Hit@N value indicates a better model performance.

[0185] In OpenKG, multiple phrase references may refer to the same entity or relation, making traditional ontology-based knowledge graph evaluation strategies unsuitable for this scenario. Therefore, this patent's evaluation strategy is as follows: To rank clusters, all NPs are first ranked. Secondly, for each cluster, the top-ranked NP is retained as a representative, and the remaining NPs are discarded. Then, these representative NPs are re-ranked, and the new ranking is assigned to the corresponding clusters. Finally, the ranking of clusters containing the true NPs is used to evaluate performance.

[0186] To more accurately evaluate the model's performance, candidate entities need to be filtered. This patent follows previous work, filtering candidate entities under time-aware filtering conditions in all experiments. That is, at each timestamp, all correct entities other than the query object are filtered out from the candidate answers to more fairly evaluate the model's performance.

[0187] 2.1.3 Baseline

[0188] This patent selects the most relevant open knowledge graph completion methods: CaRE, OKGIT, TernaryCL, and the underlying model ConvE included in the model. This patent also compares with a CaRE variant (CaRE[BERT ini]), which uses BERT to initialize phrase embeddings. Because BERT is suitable for NPs containing only a single term, predicting multi-term NPs using BERT can be challenging; therefore, this patent also compares with BERT on Reverb20KF and Reverb45KF. TernaryCL did not report experimental results on the Reverb20KF and Reverb45KF datasets, so reproduced results are used for comparison.

[0189] 2.1.4 Experimental Details

[0190] This patent implements the SeAE model using PyTorch, and all experiments were run on an RTX3090. Regarding the optimizer, an Adam optimizer with a learning rate of 0.001 was used to optimize the loss. Regarding embeddings, the embedding dimension of NP and RP was set to d. e =d r=300, the embedding dimension of the type vector is selected from {100, 300}. Regarding the neural network, the number of layers in the Entity Embedding Update Network (LAN) is set to 1. Regarding the scoring function, the weights of the type regularization term are taken from {10}. -3 10 -2 …,10 1}∪{0}. The type composition weight γ is selected from {0.25,0.5,1.0,2.0,5.0}. The weight of the similarity score between NP embeddings after entity relation gate filtering is set to 0.001, and the function for calculating the similarity score between NP embeddings after entity relation gate filtering is taken from {Euclidean distance, cosine distance}. Regarding the language model, BERT-base is used for the ReVerb20KF and ReVerb45KF datasets, and BERT-large is selected for the ReVerb20K and ReVerb45K datasets. Vocabulary All words in the dataset are initialized using pre-trained 300-dimensional GloVe word vectors, and no further word embeddings are trained. For words not included in the pre-trained vectors, this patent uses uniform distribution initialization. The normalization information generated by CESI

[40] is utilized in SeAE.

[0191] 2.2 Experimental Results

[0192] Link prediction is a core task in knowledge graph completion, aiming to predict the head and tail entities or relationships in missing facts based on existing knowledge in the knowledge graph. Tables 3 and 4 present the results of SeAE and other baselines on the OpenKG link prediction task across four datasets, based on fact plausibility scores. This patent further tests TernaryCL on the ReVerb20KF and ReVerb45KF datasets to obtain MRR, MR, Hit@1, Hit@3, and Hit@10 metrics, facilitating comparison with other models.

[0193] As shown in Tables 3 and 4, compared to the baselines, the patented model SeAE exhibits the best performance in most results, and is generally at the second-best level in the remaining results. Reverb45K and Reverb45KF are the largest datasets before and after filtering, respectively, and all metrics on these two datasets achieve state-of-the-art (SOTA) performance. Specifically, for Reverb45K, it significantly outperforms all baselines across all metrics, demonstrating a considerable advantage in all indicators. Compared to the previous best model, MRR increased by 5.4%, and H@1, 3, and 10 saw improvements of 5.4%, 5.2%, and 6.4%, respectively. For Reverb45KF, MRR increased by 7.4%, and H@1, 3, and 10 saw improvements of 1.8%, 8.6%, and 1.8%. It also outperforms all baselines in 7 out of 10 metrics for Reverb20K and Reverb20KF. In particular, compared to OKGIT, the patented model shows a significant improvement across all metrics. In ReVerb20K, 89.7% of the training nodes have a degree of 2 or less, the highest percentage among the four datasets. TernaryCL, on the other hand, is a model specifically designed to alleviate the sparsity of OpenKG. Its relational and entity contrastive learning methods increase the opportunity for learning from few-shot entities by employing negative sampling. Therefore, this patent's H@1 and H@10 metrics are slightly weaker than TernaryCL in ReVerb20K. However, on other datasets, its metrics are significantly better than TernaryCL.

[0194] The SeAE performance is significantly improved, which this patent attributes to the semantic separation effect brought about by adaptive disentanglement embedding. Using an attention-based GRU encoder to encode the RP representing relations in triples brings benefits in information extraction, semantic representation, and relation representation. The semantic constraints imposed by relation gates on entity features also provide some gains. Overall, SeAE achieves significant performance improvements at a lower cost through innovative embedding encoding methods, making it a more advantageous choice relative to the baseline.

[0195] Table 3 shows the results of SeAE and baseline on-link predictions on the ReVerb20K and ReVerb45K datasets. The best results are indicated in bold, the second best results are indicated by underlining, and "-" indicates that no results were reported in previous work.

[0196]

[0197] Table 4 shows the uplink prediction results for SeAE and baseline on the ReVerb20KF and ReVerb45KF datasets. The best results are indicated in bold, the second best results are indicated by underlining, and "*" indicates that the result was not reported in the original paper and was reproduced in this patent.

[0198]

[0199] 2.3 Ablation Experiment

[0200] This patent conducts ablation experiments on Reverb20KF, Reverb45KF, Reverb20K, and Reverb45K to investigate the impact of each module in SeAE and verify its effectiveness. More specifically, this patent refers to the adaptive unentangled embedding, the attention-based GRU encoder, and the relation gate as ADE, AttGRU, and Gate, respectively, and conducts ablation studies on these three modules. The approach with ADE is to not segment the NP embedding and use the complete NP embedding in the update network. The approach with AttGRU is to remove the attention mechanism from the attention-based GRU encoder. The approach with Gate is to remove the gate filtering loss from the loss function. As shown in Table 5, the experimental results of the complete SeAE model and the three ablation models demonstrate the effectiveness of the improvement.

[0201] Table 5 shows the results of the ablation experiment on Reverb45K.

[0202]

[0203]

[0204] 2.4 Case Study

[0205] Table 6 contains sample studies of the link prediction task on the Reverb45K dataset. As seen in triplet-1 in this patent, for the correct answer "lithuania", SeAE improved its prediction ranking from second to first in OKGIT, while the incorrect answer "lithuanian", which is very similar to the correct answer, dropped from third to eighth in OKGIT prediction. This patent speculates that this is due to adaptive disentangled embeddings. Assuming the number of NP embedding blocks K = 2, the embedding of "lithuanian" is divided into two parts. The first part of the embedding, after being updated by the network, gathers semantic attributes related to "language" from its neighbors "hypertext markip language", "buddha", and "nepali". The second part of the embedding, after being updated by the network, gathers semantic attributes related to "human" from its neighbors "russian" and "vilnius". The embedding of "lithuania" is also divided into two parts. The first part, after updating the network, gathers semantics related to "politics" from its neighbors "russia," "stalin," and "vilnius." The second part, after updating the network, gathers semantics related to "culture" from its neighbors "christianity," "christmas eve," and "lutheranism." The semantics of the two NPs, "lithuania" and "lithuanian," are separated through a deentanglement operation. The semantic differences after deentanglement are significant. When predicting Triple-1, the "politics" semantic of the correct answer "lithuania" meets the requirements of a tail entity, while the two semantic aspects of the incorrect answer "lithuanian"—"language" and "person"—have low matching degrees. Therefore, they can be easily distinguished during the prediction process.

[0206] For triple -2, there are many triples in the training set that are semantically similar to the query (imap, is replacing, ·), such as (imap, is similar to, pop) and (imap, can be used just like, pop). This patent argues that SeAE uses an attention-based GRU encoder to learn the most accurate embeddings of these RPs, perceiving the semantic similarity of "is replacing," "is similar to," and "be used just like," thereby predicting the correct answer. Furthermore, "Perl" is a feature-rich computer programming language, while "POP" and "IMAP" are email-related protocols. Adaptive unentangled embeddings unentangle the semantics of these NPs, allowing "Perl" and "POP" to be distinguished during prediction.

[0207] For triple-3, similar to triple-2, there are many triples in the training set that are similar in meaning to the query (vbscript, is a light version of, ·), such as (vbscript, is essentially a subset of, visual basic), (vbscript, is a subset of, vba), and (vbscript, is a slimmed downversion of, visual basic). This patent argues that by adaptively disentangled embedding and aggregating neighbor NP information, NP semantic disentanglement is achieved, and the attention-based GRU encoder learns better relational semantics. Despite the obvious semantic cues (asp, relieved on, vbscript) and (vbscript, is not, python), OKGIT still predicts "asp" and "python" at the top, indicating that OKGIT's simple aggregation of neighbor information from the LAN only leads to NP semantic confusion. Even when OKGIT fails to predict the correct NP, the correctly predicted NP and other NPs in the cluster are ranked at the top, demonstrating the significant improvement of the SeAE model in semantic mining.

[0208] Table 6 lists several example predictions from the SeAE and OKGIT models, highlighting the top ten NPs. Bold numbers represent correct answers. Results underlined with a wavy line represent incorrect answers illustrated in the text.

[0209]

[0210]

[0211] 2.5 The effects of different numbers of unentanglement blocks K

[0212] This section analyzes the impact of the number of NP embedding blocks, K. In reality, because there is no rigorous proof of the influence of the number of NP embedding blocks on model performance, it is difficult to determine the value of K. Mathematically, each entity can be understood as learning K attributes from a set of objects (deg), which is equivalent to solving K solutions from a set of equations (deg). If K is too large (features are too detailed), the equations become difficult to solve; if K is too small, sufficient information cannot be extracted. Based on experience, this patent finds that setting K to a relatively small value of around 2 to 5 usually yields reasonable and better results.

[0213] For different datasets, the same number of blocks K does not necessarily lead to equally good results. For example, the best-performing K value (K=3) on Reverb20KF does not perform best on Reverb45KF. One possible reason is that not enough neighboring NPs are divided into K groups. This patent addresses this by rounding down the average node degree (deg) of the knowledge graph as K. Test results for different K values ​​on all datasets using SeAE (w / o attentive GRU and Gate) are shown in Tables 7 and 8. Furthermore, this patent found in experiments that disentangled embedding is more robust than NP embedding which is randomly initialized and then updated in a LAN. Even without finding the optimal number of blocks, a reasonable number of blocks consistently improves experimental results.

[0214] Table 7. Test results of SeAE (w / o attentive GRU and Gate) at several different K values ​​on Reverb 20K and Reverb 45K.

[0215]

[0216] Table 8. Test results of SeAE (w / o attentive GRU and Gate) at different K values ​​on Reverb 20KF and Reverb 45KF.

[0217]

[0218] References:

[0219] [1]Bordes A,Usunier N,Garcia-Duran A,et al.Translating embeddings formodeling multi-relational data[J].Advances in neural information processingsystems,2013,26.

[0220] [2]Wang Z, Zhang J, Feng J, et al.Knowledge graph embedding bytranslating on hyperplanes[C] / / Proceedings of theAAAI conference on artificialintelligence.2014,28(1).

[0221] [3]Lin Y,Liu Z,Sun M,et al.Learning entity and relation embeddingsfor knowledge graph completion[C] / / Proceedings ofthe AAAI conference onartificial intelligence.2015,29(1).

[0222] [4]Ji G,He S,Xu L,et al.Knowledge graph embedding via dynamic mappingmatrix[C] / / Proceedings ofthe 53rd annual meeting ofthe association forcomputational linguistics and the 7th international joint conference onnatural language processing(volume 1:Long papers).2015:687-696.

[0223] [5]Sun Z,Deng Z H,Nie J Y,et al.Rotate:Knowledge graph embedding byrelational rotation in complex space[J].arXiv preprint arXiv:1902.10197,2019.

[0224] [6]Nickel M,Tresp V,Kriegel H P.Athree-way model for collectivelearning on multi-relational data[C] / / Icml.2011,11(10.5555):3104482.3104584.

[0225] [7]Yang B,Yih W,He X,et al.Embedding entities and relations forlearning and inference in knowledge bases[J].arXiv preprint arXiv:1412.6575,2014.

[0226] [8]Trouillon T,Welbl J,Riedel S,et al.Complex embeddings for simplelink prediction[C] / / International conference on machine learning.PMLR,2016:2071-2080.

[0227] [9]Dettmers T,Minervini P,Stenetorp P,et al.Convolutional 2dknowledge graph embeddings[C] / / Proceedings ofthe AAAI conference onartificial intelligence.2018,32(1).

[0228]

[10] Nguyen D Q,Nguyen T D,Nguyen D Q,et al.A novel embedding modelfor knowledge base completion based on convolutional neural network[J].arXivpreprint arXiv:1712.02121,2017.

[0229]

[11] Vashishth S,Sanyal S,Nitin V,et al.Interacte:Improvingconvolution-based knowledge graph embeddings by increasing featureinteractions[C] / / Proceedings of the AAAI conference on artificialintelligence.2020,34(03):3009-3016.

[0230]

[12] Schlichtkrull M,Kipf T N,Bloem P,et al.Modeling relational datawith graph convolutional networks[C] / / The Semantic Web:15th InternationalConference,ESWC 2018,Heraklion,Crete,Greece,June 3–7,2018,Proceedings15.Springer International Publishing,2018:593-607.

[0231]

[13] KipfT N,Welling M.Semi-supervised classification with graphconvolutional networks[J].arXiv preprint arXiv:1609.02907,2016.

[0232]

[14] Vashishth S,Sanyal S,Nitin V,et al.Composition-based multi-relational graph convolutional networks[J].arXiv preprint arXiv:1911.03082,2019.

[0233]

[15] Zhu Z,Zhang Z,Xhonneux L P,et al.Neural bellman-ford networks:Ageneral graph neural network framework for link prediction[J].Advances inNeural Information Processing Systems,2021,34:29476-29490.

[0234]

[16] Zhang Y,Yao Q.Knowledge graph reasoning with relational digraph[C] / / Proceedings ofthe ACM web conference 2022.2022:912-924.

[0235]

[17] Wu J,Shi W,Cao X,et al.DisenKGAT:knowledge graph embedding withdisentangled graph attention network[C] / / Proceedings ofthe 30th ACMinternational conference on information&knowledge management.2021:2140-2149.

[0236]

[18] Vu T,Nguyen T D,Nguyen D Q,et al.A capsule network-basedembedding model for knowledge graph completion and search personalization[C] / / Proceedings of the 2019 Conference of the North American Chapter of theAssociation for Computational Linguistics:Human Language Technologies,Volume1(Long and Short Papers).2019:2180-2189.

[0237]

[19] Chen S,Liu X,Gao J,et al.Hitter:Hierarchical transformers forknowledge graph embeddings[J].arXiv preprint arXiv:2008.12813,2020.

[0238]

[20] Liu X,Zhao S,Su K,et al.Mask and reason:Pre-training knowledgegraph transformers for complex logical queries[C] / / Proceedings of the 28thACM SIGKDD Conference on Knowledge Discovery and Data Mining.2022:1120-1130.

[0239]

[21] SaxenaA,KochsiekA,Gemulla R.Sequence-to-sequence knowledge graphcompletion and question answering[J].arXiv preprint arXiv:2203.10321,2022.

[0240]

[22] Yao L,Mao C,Luo Y.KG-BERT:BERT for knowledge graph completion[J].arXiv preprint arXiv:1909.03193,2019.

[0241]

[23] Cai L,Wang W Y.Kbgan:Adversarial learning for knowledge graphembeddings[J].arXiv preprint arXiv:1711.04071,2017.

[0242]

[24] Liu H,Kairong H,Fu-Lee W,et al.Correction to:Aggregatingneighborhood information for negative sampling for knowledge graph embedding[J].Neural Computing&Applications,2021,33(4):1399-1399.

[0243]

[25] Lin Y,Liu Z,Sun M,et al.Learning entity and relation embeddingsfor knowledge graph completion[C] / / Proceedings ofthe AAAI conference onartificial intelligence.2015,29(1).

[0244]

[26] Zhang F,Wang X,Li Z,et al.TransRHS:a representation learningmethod for knowledge graphs with relation hierarchical structure[J].2020.

[0245]

[27] Chao L,He J,Wang T,et al.Pairre:Knowledge graph embeddings viapaired relation vectors[J].arXiv preprint arXiv:2011.03798,2020.

[0246]

[28] Dai Quoc Nguyen T V,Nguyen T D,Phung D.Quatre:Relation-awarequaternions for knowledge graph embeddings[J].arXiv preprint arXiv:2009.12517,2020.

[0247]

[29] Zhang S,Tay Y,Yao L,et al.Quaternion knowledge graph embeddings[J].Advances in neural information processing systems,2019,32.

[0248]

[30] Yang J,Ying X,Shi Y,et al.Learning hierarchy-aware quaternionknowledge graph embeddings with representing relations as 3D rotations[C] / / Proceedings of the 29th International Conference on ComputationalLinguistics.2022:2011-2023.

[0249]

[31] Wang S,Fu K,Sun X,et al.Representation learning of knowledgegraphs with the interaction between entity types and relations[J].Neurocomputing,2022,508:305-314.

[0250]

[32] Zeb A,Saif S,Chen J,et al.Learning knowledge graph embeddings bydeep relational roto-reflection[J].Knowledge-Based Systems,2022,252:109451.

[0251]

[33] Gupta S,Kenkre S,Talukdar P.CaRe:Open knowledge graph embeddings[C] / / Proceedings of the 2019Conference on Empirical Methods in NaturalLanguage Processing and the 9th International Joint Conference on NaturalLanguage Processing(EMNLP-IJCNLP).2019:378-388.

[0252]

[34] Talukdar P P.OKGIT:Open knowledge graph link prediction withimplicit types[J].arXiv preprint arXiv:2106.12806,2021.

[0253]

[35] Devlin J,Chang M W,Lee K,et al.Bert:Pre-training ofdeepbidirectional transformers for language understanding[J].arXiv preprintarXiv:1810.04805,2018.

[0254]

[36] Dash S,Rossiello G,Mihindukulasooriya N,et al.Open knowledgegraphs canonicalization using variational autoencoders[J].arXiv preprintarXiv:2012.04780,2020.

[0255]

[37] Xie T, Peng P, Wang H, et al.Open Knowledge Graph Link Prediction with Segmented Embedding[C] / / 2022International Joint Conference on NeuralNetworks(IJCNN).IEEE,2022:1-8.

[0256]

[38] Li Q,Joty S,Wang D,et al.Alleviating Sparsity of Open KnowledgeGraphs with Ternary Contrastive Learning[J].arXiv preprint arXiv:2211.03950,2022.

[0257]

[39] Fader A, Soderland S, Etzioni O. Identifying relations for open information extraction[C] / / Proceedings of the 2011conference on empiricalmethods in natural language processing.2011:1535-1545.

[0258]

[40] Vashishth S,Jain P,Talukdar P.Cesi: Canonicalizing open knowledgebases using embeddings and side information[C] / / Proceedings of the 2018WorldWide Web Conference.2018:1317-1327..

[0259] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for predicting links in an open knowledge graph based on semantically aware embedding, characterized in that, An open knowledge graph link prediction model (SeAE) based on semantically aware embedding is constructed to achieve link prediction. The SeAE model includes: Prediction module: Embeds entities and relationships into In a continuous vector of dimension, entities are randomly initialized and relations are pre-trained word vectors; each entity embedding is updated independently in blocks to obtain adaptive unentangled embeddings of entity embeddings; relation embeddings are updated to semantically biased embeddings by a gated recurrent unit (GRU) encoder with attention; the prediction loss is calculated by using the logistic prediction value of the activation function and the true tail entity label of the tail entity embedding predicted by the base model. Type compatibility module: The tail entity embeddings predicted by the BERT model and the true tail entities are processed through a linear layer to obtain type vectors; the distance between the two type vectors is used as part of the scoring function; the type compatibility loss is calculated by combining the predicted values ​​of the type vectors after the activation function with the true tail entity labels. The gate filtering module: The tail entity embeddings predicted by the base model and the tail entity embeddings predicted by the BERT model interact with the relation embeddings respectively. The relation gate is input, and the distance between the two transformed tail entity embeddings is calculated as the gate filtering loss. Loss calculation module: The final loss is obtained by weighted summation of three losses: prediction loss, type compatibility loss, and gate filtering loss.

2. The open knowledge graph link prediction method based on semantically aware embedding according to claim 1, characterized in that, The prediction module is implemented as follows: First, define: Open Knowledge Graph ,in It is a set of NP. It is a set of RP. It is a set of fact triples; a fact set This represents the set of all fact triples in a knowledge graph. For a fact triple; a set of noun phrases This represents the set of all entities in the knowledge graph. Indicates a noun phrase, , Noun phrase indicating the end , It is a word element. All lexical elements are stored in the vocabulary. , It refers to the number of lexical units in the head NP and tail NP; the set of relational phrases. This represents the set of all relation phrases in the knowledge graph. , It refers to the number of lexical units in the RP; vocabulary. The lexical units that make up NP and RP are all included in the vocabulary. In the middle; entity embedding Representing entities Embedded, Representing entities No. Partial embedding; relational embedding Represents the embedding of relation r. Represents the embedding of relation r; Open knowledge graph link prediction, i.e., predicting triples in the test set given a training set. Missing tail NP; (1) Adaptive unentangled mode embedding Based on the average degree of the knowledge graph dataset Calculate the number of blocks in the NP embedding. : For all NP , Includes all header NPs NP (Nagi-NP) First, randomly initialize all NP embeddings. , Let NP be the embedding dimension, and then divide it into part, It needs to be divisible by the embedding dimension, NP The Each part is represented as By using this processing method, each part of the NP embedding is projected into a different subspace, and then each part of all NPs is fed into a multi-layer neural network for updating. That is, the embedding of the j-th part of the NP embedding in layer l-1 is fed into the network for updating, resulting in the j-th part of the NP embedding after l layers: in Represents NP Surrounding area, NP , This represents a neighborhood feature aggregation network (LAN), used to accumulate feature information from multi-hop neighbors. The AGGREGATE function can obtain information from... The NP embedding of the neighborhood of The information is collected in blocks, and the function is defined as follows: The AGGREGATE function normalizes each block after it has been updated by layer l by dividing the NP embedding by the node degree of the NP. When pieced together, the final NP Embedded as follows: (2) GRU encoder with attention mechanism Every word in all RP Mapped to a fixed size dimensional word vectors The vectors of all words form a shared word embedding matrix. , The embedding dimension of RP is the same as that of NP. It refers to the size of the vocabulary and the word embedding matrix. Based on pre-trained word vectors, a GRU encoder is used to encode the RP embedding, and an attention mechanism is introduced to enable the encoder to weight the input according to the importance of each input position, focusing more on the important parts within the RP, thereby improving the information extraction capability. First, the RP is obtained through the GRU encoder. Lexical embedding: in Indicates the sequence length. Indicates a GRU encoder. This represents a decompression operation, which decompresses the GRU encoded result into... Vectors of 1 word element; After calculating the attention weight for each encoded word using the attention mechanism, the attention weight is multiplied by each encoded word to obtain the weighted representation of each word. Then, these weighted representations of words are summed to obtain the weighted representation of the entire RP. in Represents RP The Embedding of each word element Representing the first of RP r The attention weight of each word is calculated using the following formula: Where the activation function Tanh is used. and These are all learnable parameters in linear layers. It represents the number of lexical units in the RP. (3) Basic model prediction Head NP embedding using adaptive disentanglement RP embeddings encoded with attention-based GRU encoders To predict tail NP embedding Using ConvE as the basic model for measuring ternary combinatorial rationality, the formula for calculating the predicted tail NP embedding is as follows: in and represent and Two-dimensional reshaping, Represents the convolution operation. Represents the convolution kernel, Represents a nonlinear function. This represents the vectorization operation of 2D tensors. It is a matrix.

3. The open knowledge graph link prediction method based on semantically aware embedding according to claim 2, characterized in that, The type compatibility module is implemented as follows: First, based on the pre-trained BERT model, the given input triples are... Return the embedding of the tail NP: MASK is a special word that represents a placeholder. Next, the latent type embedding of the tail NP is learned using the NP embedding projection mechanism: Among them, Linear is used as a type encoding conversion function. It is a true tail NP type vector. It is the type vector of the tail NP predicted by the BERT model.

4. The open knowledge graph link prediction method based on semantically aware embedding according to claim 3, characterized in that, The gate filtering module is implemented as follows: Design a relation-specific gating filtering mechanism—the relation gate—to capture common semantic features in relation-specific entity embeddings; given NP and RP Embed the structural NP predicted by the basic model and RP embedding The components are concatenated and then passed through a fully connected layer and an activation function to obtain a fused embedding vector, allowing for full semantic interaction between the NP and RP. This embedding vector is considered a comprehensive representation combining the NP and RP, with the output of the activation function and the original NP embeddings used in the process. Multiplication preserves RP-specific information, while entity names are used as auxiliary information to extract RP-specific NP features. Similarly, the tail NP embeddings predicted by BERT are processed in parallel. Thus, RP-specific text embeddings are obtained. RP feature-enhanced NP embedding , The calculation formula is as follows: in It's a splicing operation. , It is a weight matrix related to entities. , It is an entity-related deviation. It is the Hadamard product operation, activation function. It uses sigmoid, here and These are the tail NP embeddings predicted by the base model and the tail NP embeddings predicted by the BERT model, respectively. Compute the two NP embeddings with RP information after transformation. , The distance between the two points is used as the loss, and there are two distance calculation formulas to choose from: Euclidean distance and cosine distance, as shown below: in, It is the total number of training triples. It is a small value to avoid division by zero, which is 1e-8.

5. The open knowledge graph link prediction method based on semantically aware embedding according to claim 4, characterized in that, The loss calculation module is implemented as follows: 1) Scoring function The input to the scoring function of the prediction module includes... The candidate tail NP obtained and the real tail NP The score obtained through the prediction of the basic model as follows: The input to the scoring function of the type compatibility module includes the type vector of the true tail NP. The type vector of the tail NP predicted by the BERT model Type compatibility score The calculation is as follows: The final score is composed of the scores from the prediction module and the type compatibility module: in The relative weight of the control type compatibility score; 2) Loss Function The prediction loss is calculated using the binary cross-entropy (BCE) loss formula, as follows: in, It is a triplet The tag, if , The probability of a candidate fact being true is assigned a logical value based on the likelihood probability calculated using the scoring function, and this logical value serves as the prediction label. , The calculation formula is as follows: in Using sigmoid, represent Final score; The type compatibility loss is calculated using the binary cross-entropy (BCE) loss, as shown in the following formula: in, Still a triplet The tag, if , It is 1 if it is 1, otherwise it is 0. Predict labels for type ; The formula for calculating gate filter loss is as follows: By jointly optimizing three objectives, we capture RP-specific NP constraints, distinguish different attributes within NP, and utilize type compatibility information extracted by the BERT model to improve link prediction performance. The final loss calculation formula is as follows: in These are all hyperparameters, used to adjust the proportion of different parts of the loss.

6. The open knowledge graph link prediction method based on semantically aware embedding according to any one of claims 2-5, characterized in that, The underlying model uses ConvE.

Images