Methods for knowledge graph representation learning models that integrate entity descriptions and path information

Abstract

This invention discloses a method for a knowledge graph representation learning model that integrates entity descriptions and path information. First, two subsets are extracted from the large knowledge graph Freebase as training sets. These are combined with entity descriptions created by Ruobing Xie for the training sets. Both are then input into a finely tuned, powerfully optimized pre-trained language representation model, RoBERT. Training is performed through a self-attention mechanism and a feedforward neural network consisting of four fully connected layers and a ReLU activation function. The final layer outputs entity and relation representation learning vectors that integrate entity descriptions. Then, the knowledge graph is transformed into vectors incorporating ordered relation and path information, and the energy value E of all training set triples is output. P Then, gradient descent is used to optimize the vector representation of the triples, and the knowledge graph representation learning model is trained. This invention improves the efficiency of representation learning path modeling and enhances the multi-hop reasoning capability of the knowledge graph representation learning model.

Description

Technical Field

[0001] This invention belongs to the field of knowledge graph technology, specifically relating to a method for a knowledge graph representation learning model that integrates entity description and path information. Background Technology

[0002] In 2012, Google formally proposed the concept of Knowledge Graph. As a key technology for artificial intelligence applications, Knowledge Graph has a clear triple structure, with each triple containing a head entity, a relation, and a tail entity, making... To represent a knowledge graph, we have ,in Represents the set of all entities. Representing the set of all relations, knowledge graphs are stored in the form of triples to facilitate the integration of knowledge from various domains. Currently, numerous large-scale knowledge graphs have emerged, such as Freebase, WordNet, and the Chinese knowledge graph OpenKG. However, compared to the objective world, these constructed knowledge graphs are often incomplete, missing a significant amount of facts. To alleviate the sparsity problem of data, researchers have proposed knowledge graph representation learning, which aims to project entities and relations in the knowledge graph onto a continuous low-dimensional vector space, thereby improving the reasoning capabilities of the knowledge graph.

[0003] Knowledge graph representation learning can be divided into three categories: geometric models, matrix factorization models, and deep learning models. Among geometric models, the most classic knowledge graph representation learning model is the TransE model. LIN et al. proposed the PTransE model to incorporate path information on top of the Trans series models. In matrix factorization models, the DistMult model forces all relations to be represented as diagonal matrices, which reduces the parameter space to be learned, making the model easier to train. The CompIEx model is a complex number-based knowledge graph representation learning model that captures both symmetric and asymmetric relations in knowledge graphs. In deep learning models, the ConvE model was the first to introduce 2D convolution operations to perform link prediction tasks and has been extended for complex knowledge graphs. The ConvKB model uses a convolutional neural network, representing each triple as a three-column matrix and inputting it into the convolutional layer. Finally, the feature vector and weight vector are multiplied by a dot product to obtain the score.

[0004] Currently, most of the aforementioned knowledge graph representation learning and reasoning models only consider single triplet information in the knowledge graph. However, for complex problems, long paths containing multiple triples are generally required for multi-hop reasoning. Therefore, conducting research on multi-hop knowledge graph question answering is of great significance. Summary of the Invention

[0005] The purpose of this invention is to provide a method for a knowledge graph representation learning model that integrates entity description and path information, thereby improving the efficiency of representation learning path modeling and enhancing the multi-hop reasoning capability of the knowledge graph representation learning model.

[0006] The technical solution adopted in this invention is a method for learning a knowledge graph representation that integrates entity description and path information, characterized by the following steps:

[0007] Step 1: Extract two subsets from the large knowledge graph Freebase as training sets, and combine them with the entity descriptions made by Ruobing Xie for the training sets. Input both into the finely tuned and powerfully optimized pre-trained language representation model RoBERT. Train and learn through a self-attention mechanism and a feedforward neural network with four fully connected layers and ReLU activation function. Output the entity and relation representation learning vectors that fuse entity descriptions in the last layer.

[0008] Step 2: Utilize the OPTransE model, a knowledge graph completion representation learning model based on ordered relational paths, to transform the knowledge graph into vectors incorporating ordered relational path information, and output the energy value E of all training set triples. P ;

[0009] Step 3: Combining the training vectors output from Steps 1 and 2, calculate the energy value E based on entity description and path information. D , and the energy value E based on path information P The total energy of each triplet is defined as E. D +E P Then, gradient descent is used to optimize the vector representation of the triples, with the goal of training a knowledge graph representation learning model by minimizing the total energy value.

[0010] The invention is further characterized in that,

[0011] Step 1 is implemented in the following steps:

[0012] Step 1.1: Extract triples from two subsets of the large knowledge graph Freebase—FB15k-237 and FB15K. Treat each triple as a text sequence containing three sentences and replace the entities with entity descriptions. Using [CLS] classification labels and [SEP] segmentation labels, adjust the original RoBERTa model to accept text sequence input in a specified format: head entity-relation-tail entity. The specific input to the adjusted model is as follows:

[0013] S1.1.1 Assume there are triples: head entity, relation, and tail entity. Set a classification label [CLS] at the head position of each input sequence.

[0014] S1.1.2. Represent the header entity or its corresponding entity description as a sentence containing 'a' tokens. , , , A token is the smallest unit of natural language input into the RoBERTa model, represented as a single English word.

[0015] S1.1.3. Represent the relation as a sentence containing b tokens. , , , ;

[0016] S1.1.4. The tail entity or its corresponding entity description is also represented as a sentence containing c tokens: , , , ;

[0017] S1.1.5. Add a special separator tag [SEP] between entities and relations;

[0018] Step 1.2: After initializing and adjusting the structure of the text sequence transformed in Step 1.1, use it as input to the RoBERTa model, and finally utilize the latent output vector of the special classification label [CLS]. To predict whether a triple is correct, the text is modeled using multi-layer Transformer encoding blocks based on a multi-head self-attention mechanism. This allows each word to incorporate information from the entire sentence. The self-attention mechanism is defined as follows:

[0019] Three vectors are generated from the input vector of each encoded block, i.e., the word vector of each word: query vector. key vector Sum value vector Let the self-attention mechanism vector at the i-th head be... Calculate according to formulas (1), (2) and (3) respectively. and :

[0020]

[0021] Finally, based on and Find the representation learning vector The corresponding normalized exponential function softmax:

[0022]

[0023] for Dimensionality, Attention is self-attention. for transpose,

[0024] Under the multi-head attention mechanism, the above operation is repeated on each attention head to obtain multiple feature representations through different attention heads. The output feature vectors of these multi-head attention are concatenated, multiplied with a randomly initialized weight matrix, and then connected to a fully connected layer for dimensionality reduction.

[0025] After obtaining the multi-head attention mechanism, a LayerNormalize step is followed in each coding block. The layer normalization operation is shown in formulas (5) and (6):

[0026]

[0027] Where Y is the output of the attention head, and ReLU is the activation function. and It is an adjustable parameter. This indicates that multiple outputs are spliced ​​together. The output of one coding block is used as the input of the next coding block, and so on, until the output of the last coding block is used as the output of the entire Transformer.

[0028] Step 1.3: The vector output in step 1.2... The average is calculated as the final model output, resulting in a vector based on the entity description.

[0029] Step 2 is implemented in the following steps:

[0030] Step 2.1: Perform long-path modeling on the triples input in Step 1.1:

[0031] Formula (7) defines the energy function. , Indicates the head entity. Indicates the tail entity. Indicates a relationship. Represents the set of all entities in a knowledge graph. Represents the set of all relations. From arrive One of the relationship paths, for example: , where n is the number of relation paths, if from arrive If the path is reasonable, the resulting energy value will be lower.

[0032]

[0033] and Representing ordered relation paths respectively Middle head entity Tail-end entity The representation of the learned vector, For a given path Regarding the first A sequence matrix of relations, For the i-th relation, the directly connected triples Considered a head entity Tail-end entity There is a single-step path between them. ,therefore, The energy value can be directly related to As Substitute into formula (7) to obtain;

[0034] Step 2.2, establish the relationship Corresponding head entity Tail-end entity Projected onto different hyperplanes using two different projection matrices, let... and Representing the relationship The projection matrices of the head and tail entities, assuming there is a path from... arrive path Define the following formula (10):

[0035]

[0036] in, This represents the i-th node on the path, for a path from... arrive The entities of the relational path are represented after eliminating transitive nodes from equation (10), and the specific forms of the variables in equation (7) are as follows:

[0037]

[0038] Project the tail entity in the path to In the space, Representing a path The projection matrix, and further, assuming the existence of a path , Simultaneously acting as a relationship tail entity and The head entity, in order to connect relationships in different spaces, is defined in formula (14). ,in, Used to transmit nodes from The head entity space is transferred to The tail entity space. k ranges from 1 to i, I represents the identity matrix, and Indicates from The physical space of the head The spatial transformation matrix of the tail entity space;

[0039] Step 2.3: Information from different paths is fused using a two-layer pooling strategy to construct the total energy function of the triples. The first-layer pooling strategy will apply to all... Selecting the path in the step path The first layer selects the best-matching path; the second layer integrates information from paths of different lengths, extracting non-linear features from these paths.

[0040] Step 2.3 is implemented according to the following steps:

[0041] The pooling strategy for the first layer: utilizes the minimum pooling method from... Extract feature information from the path and define the following energy function:

[0042]

[0043] Among them, the energy function is defined. , Represents entities from scratch End-to-end entity Through relationships All The set of steps, in order to obtain Introducing conditional probability Determine the given relation Path in case Feasibility probability:

[0044]

[0045] express and The joint probability, express The marginal probability, in addition. In the knowledge graph and The number of cases where the same entity pair is connected. In the knowledge graph The number of paths, This represents the total number of paths in a knowledge graph, subject to given constraints. filter , to obtain all filtered set ;

[0046] The pooling strategy for the second layer is defined as follows: Total energy function :

[0047]

[0048] Indicates a direct relationship Energy value, By Substituting into formula (16) Calculated.

[0049] Step 3 is implemented in the following steps:

[0050] The energy functions in steps 2.3 and 1.3 of the joint learning process combine triplet information with rich information from path information and entity descriptions, and the overall energy function is defined as shown in formula (18):

[0051]

[0052] It is the total energy function based on path information given in step 2.3; It is an energy function based on entity description, and They are all embedded in the same vector space. Defined as the form of formula (19):

[0053]

[0054] in, It is the transpose matrix. In and Both are head and tail entity representations based on entity descriptions, where r represents a relation. In It is a header entity representation based on entity description. It is a tail entity representation based on path information. In It is a header entity representation based on entity description. It is a tail entity representation based on path information;

[0055] Finally, a marginal loss function is selected for model training, and its definition is shown in formula (23) below:

[0056]

[0057] in These are the boundary parameters of the positive and negative sample sets. It is the total energy function defined in (18). It is a set of correct triples The training set constituted It is a set of incorrect triples The negative sample set constitutes the set of all relations, R represents the set of all relations. The definition is as follows in formula (24):

[0058]

[0059] Knowledge graph representation learning models are processes that represent entities and relationships in a knowledge graph as multi-dimensional vectors using specific rules. The goal is to minimize the total energy value E, and the vector representation of triples is continuously optimized through iterative training. After training, the resulting vectors will have more powerful multi-hop reasoning capabilities.

[0060] The beneficial effect of this invention is that the method of integrating entity description and path information into a knowledge graph representation learning model implicitly captures knowledge of all observed and unobserved links around each node. By designing two encoders to integrate entity description and path information respectively, the first encoder enhances the representation ability of entity description text through a pre-trained language model RoBERTa, while the second encoder selects the OPTransE model to further describe long relational paths in the knowledge graph. Finally, the two are fused, resulting in better long path modeling involving multiple triples and improved multi-hop reasoning capabilities in knowledge graph representation learning. Attached Figure Description

[0061] Figure 1 This is an architecture diagram of the knowledge graph representation learning model that integrates entity description and path information according to the present invention.

[0062] Figure 2 This is a schematic diagram of the fine-tuned RoBERTa structure in the knowledge graph representation learning model that integrates entity description and path information in this invention.

[0063] Figure 3 This is a schematic diagram of the knowledge graph representation learning model architecture based on path information in the knowledge graph representation learning model that integrates entity description and path information in this invention. Detailed Implementation

[0064] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0065] This invention integrates a knowledge graph representation learning model that combines entity description and path information, and combines... Figure 1As shown, please follow these steps:

[0066] Step 1: Extract two subsets from the large knowledge graph Freebase as training sets, and combine them with the entity descriptions created by Ruobing Xie for the training sets. Input both into a finely tuned and powerfully optimized pre-trained language representation model RoBERT (A Robustly Optimized Bidirectional Encoder Representation from Transformers). Train the model through a self-attention mechanism and a feedforward neural network consisting of four fully connected layers and a ReLU (Linear Rectification) activation function. The last layer outputs the entity and relation representation learning vectors that fuse entity descriptions.

[0067] Step 1 is implemented in the following steps:

[0068] Step 1.1: Extract triples from two subsets of the large knowledge graph Freebase—FB15k-237 and FB15K. Treat each triple as a text sequence containing three sentences and replace the entities with entity descriptions. This invention uses [CLS] classification labels and [SEP] segmentation labels to adjust the original RoBERTa model to accept text sequence input in a specified format: head entity-relation-tail entity. The specific input to the adjusted model is as follows:

[0069] Step 1.1: Extract triples from two subsets of the large knowledge graph Freebase—FB15k-237 and FB15K. Treat each triple as a text sequence containing three sentences and replace the entities with entity descriptions. Using [CLS] classification labels and [SEP] segmentation labels, adjust the original RoBERTa model to accept text sequence input in a specified format: head entity-relation-tail entity. The specific input to the adjusted model is as follows:

[0070] S1.1.1 Assume there are triples: head entity, relation, and tail entity. Set a classification label [CLS] at the head position of each input sequence.

[0071] S1.1.2. Represent the header entity or its corresponding entity description as a sentence containing 'a' tokens. , , , A token is the smallest unit of natural language input into the RoBERTa model, represented as a single English word.

[0072] S1.1.3. Represent the relation as a sentence containing b tokens. , , , ;

[0073] S1.1.4. The tail entity or its corresponding entity description is also represented as a sentence containing c tokens: , , , ;

[0074] S1.1.5. Add a special separator tag [SEP] between entities and relations;

[0075] Step 1.2: After initializing and adjusting the structure of the text sequence transformed in Step 1.1, use it as input to the RoBERTa model, and finally utilize the latent output vector of the special classification label [CLS]. To predict whether a triple is correct, RoBERTa's core structure is a bidirectional Transformer model. The Transformer contains multiple encoding blocks, each of which includes a self-attention layer. Note: The above adjustments to the input structure are the same as the adjustments to the Roberta module described in this invention. Fine-tuning of the shape ). Figure 2 This is a schematic diagram of the finely tuned RoBERTa structure of this invention; the text is modeled by multi-layer Transformer encoding blocks based on a multi-head self-attention mechanism, so that each word can incorporate the information of the entire sentence. The self-attention mechanism is defined as follows:

[0076] Three vectors are generated from the input vector of each encoded block, i.e., the word vector of each word: query vector. key vector Sum value vector Let the self-attention mechanism vector at the i-th head be... Calculate according to formulas (1), (2) and (3) respectively. and :

[0077]

[0078] Finally, based on and Find the representation learning vector The corresponding normalized exponential function softmax:

[0079]

[0080] for Dimensionality, Attention is self-attention. for transpose,

[0081] Under the multi-head attention mechanism, the above operation is repeated on each attention head to obtain multiple feature representations through different attention heads. The output feature vectors of these multi-head attention are concatenated, multiplied with a randomly initialized weight matrix, and then connected to a fully connected layer for dimensionality reduction.

[0082] After obtaining the multi-head attention mechanism, a LayerNormalize step is followed in each coding block. The layer normalization operation is shown in formulas (5) and (6):

[0083]

[0084] Where Y is the output of the attention head, and ReLU is the activation function. and It is an adjustable parameter. This indicates that multiple outputs are spliced ​​together. The output of one coding block is used as the input of the next coding block, and so on, until the output of the last coding block is used as the output of the entire Transformer.

[0085] Step 1.3: The vector output in step 1.2... The average is calculated as the final model output, resulting in a vector based on the entity description.

[0086] Step 2: Utilize the OPTransE model (Representation learning with ordered relation paths for knowledge graph completion) to transform the knowledge graph into vectors incorporating ordered relation path information, outputting the energy value E of all training set triples. P ;

[0087] Step 2 is implemented in the following steps:

[0088] like Figure 3 As shown, the head and tail entities of each triple are first projected into different vector spaces to ensure the orderliness of the relation path. Then, the total energy function of the triple is constructed through two pooling layers.

[0089] Step 2.1: Perform long-path modeling on the triples input in Step 1.1:

[0090] Formula (7) defines the energy function. , Indicates the head entity. Indicates the tail entity. Indicates a relationship. Represents the set of all entities in a knowledge graph. Represents the set of all relations. From arrive One of the relationship paths, for example: , where n is the number of relation paths, if from arrive If the path is reasonable, the resulting energy value will be lower.

[0091]

[0092] and Representing ordered relation paths respectively Middle head entity Tail-end entity The representation of the learned vector, For a given path Regarding the first A sequence matrix of relations, For the i-th relation, the directly connected triples Considered a head entity Tail-end entity There is a single-step path between them. ,therefore, The energy value can be directly related to As Substitute into formula (7) to obtain;

[0093] Step 2.2, establish the relationship Corresponding head entity Tail-end entity Projected onto different hyperplanes using two different projection matrices, let... and Representing the relationship The projection matrices of the head and tail entities, assuming there is a path from... arrive path Define the following formula (10):

[0094]

[0095] in, This represents the i-th node on the path, for a path from... arrive The entities of the relational path are represented after eliminating transitive nodes from equation (10), and the specific forms of the variables in equation (7) are as follows:

[0096]

[0097] Project the tail entity in the path to In the space, Representing a path The projection matrix, and further, assuming the existence of a path , Simultaneously acting as a relationship tail entity and The head entity, in order to connect relationships in different spaces, is defined in formula (14). ,in, Used to transmit nodes from The head entity space is transferred to The tail entity space. k ranges from 1 to i, I represents the identity matrix, and Indicates from The physical space of the head The spatial transformation matrix of the tail entity space;

[0098] Step 2.3: Information from different paths is fused using a two-layer pooling strategy to construct the total energy function of the triples. The first-layer pooling strategy will apply to all... Selecting the path in the step path The first layer selects the best-matching path; the second layer integrates information from paths of different lengths, extracting non-linear features from these paths.

[0099] Step 2.3 is implemented according to the following steps:

[0100] The pooling strategy for the first layer: utilizes the minimum pooling method from... Extract feature information from the path and define the following energy function:

[0101]

[0102] Among them, the energy function is defined. , Represents entities from scratch End-to-end entity Through relationships All The set of steps, in order to obtain Introducing conditional probability Determine the given relation Path in case Feasibility probability:

[0103]

[0104] express and The joint probability, express The marginal probability, in addition. In the knowledge graph and The number of cases where the same entity pair is connected. In the knowledge graph The number of paths, This represents the total number of paths in a knowledge graph, subject to given constraints. filter , to obtain all filtered set ;

[0105] The pooling strategy for the second layer: To fuse information from paths of different lengths, the following total energy function is defined. :

[0106]

[0107] Indicates a direct relationship Energy value, By Substituting into formula (16) Calculated.

[0108] Step 3: Combining the training vectors output from Steps 1 and 2, calculate the energy value E based on entity description and path information. D , and the energy value E based on path information P The total energy of each triplet is defined as E. D +E P Then, gradient descent is used to optimize the vector representation of the triples, with the goal of training a knowledge graph representation learning model by minimizing the total energy value.

[0109] Step 3 is implemented in the following steps:

[0110] The energy functions in steps 2.3 and 1.3 of the joint learning process combine triplet information with rich information from path information and entity descriptions, and the overall energy function is defined as shown in formula (18):

[0111]

[0112] It is the total energy function based on path information given in step 2.3; It is an energy function based on entity description, and They are all embedded in the same vector space. Defined as the form of formula (19):

[0113]

[0114] in, It is the transpose matrix. In and Both are head and tail entity representations based on entity descriptions, where r represents a relation. In It is a header entity representation based on entity description. It is a tail entity representation based on path information. In It is a header entity representation based on entity description. It is a tail entity representation based on path information;

[0115] Finally, a marginal loss function is selected for model training, and its definition is shown in formula (23) below:

[0116]

[0117] in These are the boundary parameters of the positive and negative sample sets. It is the total energy function defined in (18). It is a set of correct triples The training set constituted It is a set of incorrect triples The negative sample set constitutes the set of all relations, R represents the set of all relations. The definition is as follows in formula (24):

[0118]

[0119] Knowledge graph representation learning models are processes that represent entities and relationships in a knowledge graph as multi-dimensional vectors using specific rules. The goal is to minimize the total energy value E, and the vector representation of triples is continuously optimized through iterative training. After training, the resulting vectors will have more powerful multi-hop reasoning capabilities.

[0120] Example

[0121] I. Experimental Data

[0122] Experiments were conducted on three widely used benchmark knowledge graph datasets: FB15K, FB15k-237, and WN18RR. WordNet is an English dictionary based on cognitive linguistics and also a database. WN18RR, as a subset of WordNet, has the excellent characteristics of broad coverage and large test volume. FB15k-237 and FB15K are two subsets of Freebase, whose data contain a wealth of knowledge, including information about humans, media, geolocation, and more.

[0123] In the experiment, the FB15K dataset contained 1345 relations and 14951 entities. The WIN18RR dataset contained 11 relations and 4094 entities; the FB15k-237 dataset contained 237 relations and 14541 entities. The remaining data in the three datasets were divided into training, validation, and test sets, respectively. The training set was used for model training, the validation set was used for parameter tuning, and the test set was used to evaluate the performance of the final model.

[0124] Table 1 Statistical data for each dataset

[0125]

[0126] II. Model Training

[0127] Here, different parameter configurations were made for the two encoders of the model.

[0128] (1) Training of Encoder1 with entity description

[0129] The method is based on a RoBERTa pre-trained language model with 12 layers, 12 self-attention heads, and a representation learning dimension of 768. The parameters set during fine-tuning were: batch size = 32, learning rate = 5e-5, and dropout rate = 0.1. Other combinations of these hyperparameters were tested, but no significant differences were observed.

[0130] Table 2 Encoder1 Model Parameters

[0131]

[0132] (2) Training of Encoder2 with fused path information

[0133] Referencing the optimal parameter configuration in OPTransE, the learning rate is set to 0.0001, with a margin. γ1=5.0, γ2=5.5, and the balance factor λ=0.01. During the link prediction training process, 5 negative samples (incorrect triples) were sampled for each positive sample (correct triple), and 1, 3, 5 and 10 were tried. The experimental results of 1:5 were found to be the best.

[0134] Table 3 Encoder2 model parameters

[0135]

[0136] III. Experimental Results

[0137] The link prediction results of our model are compared with those of other benchmark models for knowledge graph representation learning on three datasets: FB15K, FB15k-237, and WN18RR. Specific experimental results are shown in Tables 4, 5, and 6.

[0138] Table 4 Comparison of WN18RR Single-Hop Link Prediction Results

[0139]

[0140] Table 5 Comparison of FB15k-237 Single-Hop Link Prediction Results

[0141]

[0142] Table 6 Comparison of FB15K Single-Hop Link Prediction Results

[0143]

[0144] To evaluate the model's multi-hop reasoning capability, all head entities were traversed in the FB15K knowledge graph. Store all corresponding relationships and tail entities. , and then Starting with the head entity, traversing and storing the data yields the 2-hop data; the 3-hop relationship path follows the same process. After obtaining the corresponding multi-hop data, the next step is to implement multi-hop link prediction. Since there are relatively few models with multi-hop inference capabilities, only PTransE and OPTransE are selected as benchmark comparison methods here.

[0145] Table 7 Comparison of FB15K-2hop Multi-hop Link Prediction Results

[0146]

[0147] Table 8 Comparison of FB15K-3hop Multi-hop Link Prediction Results

[0148]

[0149] IV. Experimental Conclusions

[0150] This invention presents a knowledge graph representation learning model that integrates entity description and path information methods. Experiments have verified the effectiveness of this method. Specific conclusions are as follows:

[0151] (1) The knowledge graph representation learning model that integrates entity description and path information proposed in this invention has achieved good results in single-hop link prediction experiments. Among the three metrics, Hits@10 performed best. Hits@10 refers to the proportion of correct entities ranked in the top 10. This model achieved Hits@10, Hits@10, and Hits@10, respectively, which are the best on all three datasets. This shows that incorporating path information is also a feasible method to improve the accuracy of correctly predicted entities in knowledge graph representation learning.

[0152] (2) The knowledge graph representation learning model that integrates entity description and path information proposed in this invention outperforms PtransE and OPTransE in both 2-hop and 3-hop experiments. This is because the model in this paper uses two external information sources, entity description and path information, to supplement the original graph and expand the knowledge sources. Entity description can help capture hidden details in the knowledge graph. For example, it is impossible to determine the nature of the company "Apple" simply from the entity name Apple Inc., but through the information in the entity description, we can know that it is a high-tech company.

Claims

1. A method for a knowledge graph representation learning model that integrates entity descriptions and path information, characterized in that, The specific steps are as follows: Step 1: Extract two subsets from the large knowledge graph Freebase as training sets, and combine them with the entity descriptions made by Ruobing Xie for the training sets. Input both into the finely tuned and powerfully optimized pre-trained language representation model RoBERT. Train and learn through a self-attention mechanism and a feedforward neural network with four fully connected layers and ReLU activation function. Output the entity and relation representation learning vectors that fuse entity descriptions in the last layer. Step 1 is implemented in the following steps: Step 1.1: Extract triples from two subsets of the large knowledge graph Freebase—FB15k-237 and FB15K. Treat each triple as a text sequence containing three sentences and replace the entities with entity descriptions. Using [CLS] classification labels and [SEP] segmentation labels, adjust the original RoBERTa model to accept text sequence input in a specified format: head entity-relation-tail entity. The specific input to the adjusted model is as follows: S1.1.1 Assume there are triples: head entity, relation, and tail entity. Set a classification label [CLS] at the head position of each input sequence. S1.1.

2. Represent the header entity or its corresponding entity description as a sentence containing 'a' tokens. , , , A token is the smallest unit of natural language input into the RoBERTa model, represented as a single English word. S1.1.

3. Represent the relation as a sentence containing b tokens. , , , ; S1.1.

4. The tail entity or its corresponding entity description is also represented as a sentence containing c tokens: , , , ; S1.1.

5. Add a special separator tag [SEP] between entities and relations; Step 1.2: After initializing and adjusting the structure of the text sequence transformed in Step 1.1, use it as input to the RoBERTa model, and finally utilize the latent output vector of the special classification label [CLS]. To predict whether a triple is correct, the text is modeled using multi-layer Transformer encoding blocks based on a multi-head self-attention mechanism. This allows each word to incorporate information from the entire sentence. The self-attention mechanism is defined as follows: Three vectors are generated from the input vector of each encoded block, i.e., the word vector of each word: query vector. key vector Sum value vector Let the self-attention mechanism vector at the i-th head be... Calculate according to formulas (1), (2) and (3) respectively. and : Finally, according to and Find the representation learning vector The corresponding normalized exponential function softmax: for Dimensionality, Attention is self-attention. for transpose, Under the multi-head attention mechanism, the operation is repeated on each attention head to obtain multiple feature representations through different attention heads. The output feature vectors of these multi-head attention are concatenated, multiplied with a randomly initialized weight matrix, and then connected to a fully connected layer for dimensionality reduction. After obtaining the multi-head attention mechanism, a layer normalization step is followed in each coding block. The layer normalization operation is shown in formulas (5) and (6): Where Y represents the attention head output. relu For activation function, and It is an adjustable parameter. This indicates that multiple outputs are spliced ​​together. The output of one coding block is used as the input of the next coding block, and so on, until the output of the last coding block is used as the output of the entire Transformer. Step 1.3: The vector output in step 1.2... The average is calculated as the final model output, resulting in a vector based on the entity description. Step 2: Utilize the OPTransE model, a knowledge graph completion representation learning model based on ordered relational paths, to transform the knowledge graph into vectors incorporating ordered relational path information, and output the energy value E of all training set triples. P ; Step 2 is implemented in the following steps: Step 2.1: Perform long-path modeling on the triples input in Step 1.1: Formula (7) defines the energy function. , Indicates the head entity. Indicates the tail entity. Indicates a relationship. Represents the set of all entities in a knowledge graph. Represents the set of all relations. From arrive One of the relationship paths, , where n is the number of relation paths, if from arrive If the path is reasonable, the resulting energy value will be lower. and Representing ordered relation paths respectively Middle head entity Tail-end entity The representation of the learned vector, For a given path Regarding the first A sequence matrix of relations, For the i-th relation, the directly connected triples Considered a head entity Tail-end entity There is a single-step path between them. ,therefore, The energy value will be directly related to As Substitute into formula (7) to obtain; Step 2.2, establish the relationship Corresponding head entity Tail-end entity Projected onto different hyperplanes using two different projection matrices, let... and Representing the relationship The projection matrices of the head and tail entities, assuming there is a path from... arrive path Define the following formula (10): in, This represents the i-th node on the path, for a path from... arrive The entities of the relational path are represented after eliminating transitive nodes from equation (10), and the specific forms of the variables in equation (7) are as follows: Project the tail entity in the path to In the space, Representing a path The projection matrix, and further, assuming the existence of a path , Simultaneously acting as a relationship tail entity and The head entity, in order to connect relationships in different spaces, is defined in formula (14). ,in, Used to transmit nodes from The head entity space is transferred to The tail entity space, k takes values ​​from 1 to i, I represents the identity matrix, and Indicates from The physical space of the head The spatial transformation matrix of the tail entity space; Step 2.3: Information from different paths is fused using a two-layer pooling strategy to construct the total energy function of the triples. The first-layer pooling strategy will apply to all... Selecting the path in the step path The best matching path; the second layer will fuse information from paths of different lengths and extract non-linear features from paths of different lengths. Step 2.3 is implemented in the following steps: The pooling strategy for the first layer: utilizes the minimum pooling method from... Extract feature information from the path and define the following energy function: Among them, the energy function is defined. , Represents entities from scratch End-to-end entity Through relationships All The set of steps, in order to obtain Introducing conditional probability Determine the given relation Path in case Feasibility probability: express and The joint probability, express The marginal probability, in addition. In the knowledge graph and The number of cases where the same entity pair is connected. In the knowledge graph The number of paths, This represents the total number of paths in a knowledge graph, subject to given constraints. filter , to obtain all filtered set ; The pooling strategy for the second layer is defined as follows: Total energy function : Indicates a direct relationship Energy value, By Substituting into formula (16) Calculated; Step 3: Combining the training vectors output from Steps 1 and 2, calculate the energy value E based on entity description and path information. D The energy value E based on path information P The total energy of each triplet is defined as E. D +E P Then, gradient descent is used to optimize the vector representation of the triples, with the goal of training a knowledge graph representation learning model by minimizing the total energy value.

2. The method for learning a knowledge graph representation that integrates entity description and path information according to claim 1, characterized in that, Step 3 is implemented in the following steps: The energy functions in steps 2.3 and 1.3 of the joint learning process combine triplet information with rich information from path information and entity descriptions, and the overall energy function is defined as shown in formula (18): It is the total energy function based on path information given in step 2.3; It is an energy function based on entity description, and They are all embedded in the same vector space. Defined as the form of formula (19): in, It is the transpose matrix. In and Both are head and tail entity representations based on entity descriptions, where r represents a relation. In It is a header entity representation based on entity description. It is a tail entity representation based on path information. In It is a header entity representation based on path information. It is a tail entity representation based on entity description; Finally, a marginal loss function is selected for model training, and its definition is shown in formula (23) below: in These are the boundary parameters of the positive and negative sample sets. It is the total energy function defined in (18). It is a set of correct triples The training set constituted It is a set of incorrect triples The negative sample set constitutes the set of all relations, R represents the set of all relations. The definition is as follows in formula (24): Knowledge graph representation learning models are the process of representing entities and relations in a knowledge graph as a multi-dimensional vector using specific rules. The goal is to minimize the total energy value E. Through iterative training, the vector representation of triples is continuously optimized. After training, the resulting vector will have a more powerful multi-hop reasoning capability.

Images