Dynamic Knowledge Graph Completion Method Based on Dual-Stream Embedding and Deep Neural Network

Abstract

The present invention relates to the field of knowledge graph completion, and discloses a dynamic knowledge graph completion method based on dual-stream embedding and deep neural network. Technical solution: Use the text description of knowledge to replace the original name, and then divide the triple into two asymmetric parts: the head entity, the relation, and the tail entity. Apply two identical BERT pre-trained models using the structure of a siamese network for knowledge embedding, saving a large amount of time cost while introducing knowledge semantic information and avoiding the problem of combinatorial explosion; in the inference stage, a residual neural network is used, which has better performance than the traditional CNN residual neural network and gets rid of the problem of model degradation that appears as the number of network layers increases. The present invention has good performance and combines the characteristics of high efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph completion, especially the field of dynamic knowledge graph completion, and involves relations and entities that do not appear in the knowledge graph. Background Art

[0002] A knowledge graph is a special database that usually states a fact in the form of an RDF triple (head entity, relation, tail entity). Currently, knowledge graphs have been widely applied to many fields such as intelligent question answering, machine translation, personalized recommendation, etc. However, many large-scale knowledge graphs still have the problem of data sparsity, and a large number of objectively existing factual relations have not been fully explored.

[0003] Knowledge graph completion is to solve the above problems by predicting the missing relations between entities, thereby generating new triples to supplement the original knowledge base. Knowledge graph completion techniques are divided into two major categories in terms of methods. The first category is the method based on graph embedding, which judges the relations between entities by calculating the structural information of triples. This type of method has a simple model, but does not combine the text information of the knowledge in the knowledge base. Therefore, this type of method can only perform static knowledge completion and cannot be generalized to entities not seen during the training process. The second type of method is to perform link prediction through the text description of triples or the text semantic information of their names. Although this type of method has a relatively high accuracy rate and can predict entities and relations that do not exist in the knowledge graph, the computational time cost of this type of method is relatively high, and the problem of combinatorial explosion is likely to occur, so it cannot be applied to large-scale knowledge graphs. Summary of the Invention

[0004] Object of the Invention: To provide a dynamic knowledge graph completion method based on dual-stream embedding and a deep neural network to solve the problem of dynamic knowledge graph completion, which improves the model effect while reducing the time of entity embedding and reasoning.

[0005] The technical solution adopted by the present invention is: A knowledge graph completion method based on dual-stream embedding and a deep neural network, comprising the following steps:

[0006] Step 1: Obtain the text information of entities and relations in the data;

[0007] Step 2: Perform negative sampling on the data to obtain negative samples;

[0008] Step 3: Combine the head entity and the relation, and at the same time, construct the head entity, the relation, and the tail entity into the input data form of BERT;

[0009] Step 4: Input the text representations of the head entity, the relation, and the tail entity into two BERTs respectively to obtain the embedding vectors of the two parts;

[0010] Step 5: Concatenate the two parts of the embedding vectors according to the original dimensions;

[0011] Step 6: Use one-dimensional convolution to increase the dimension of the matrix and project it onto a two-dimensional plane to generate a feature map;

[0012] Step 7: Input the feature map into multiple residual modules for feature extraction;

[0013] Step 8: Use average pooling on the feature matrix, and then input it into a fully connected layer with a Softmax classifier at the top to obtain the triple scores;

[0014] Step 9: Calculate the triple scores of all triples in the training set through the model, and further train the model using the scores;

[0015] Step 10: Verify the model link prediction effect and the result of completing triples that have not appeared in the training set on the training set.

[0016] Furthermore, for Step 1, in this case, the commonly used FB15K-237, WN18RR datasets in the knowledge graph completion task, and the multi-relation dataset NELL-ONE are used. There are certain descriptions of entities and relations in the datasets. When the description of an entity or relation exists, the description is used instead of its name. When the description of an entity or relation does not exist, the name is directly used as the text information.

[0017] Furthermore, for Step 2, a negative sampling strategy of random replacement is adopted. The head entity or tail entity of the given triple is removed and randomly replaced with other entities. When the recombined triple does not exist in the original knowledge graph, it is added to the negative sample list as a negative sample.

[0018] Furthermore, for Step 3, first, use the tokenizer of BERT to tokenize the descriptions or names of entities and relations and convert them into index form, and then add [CLS] and [SEP] to the head entity, relation, and tail entity respectively to construct the form of the BERT input.

[0019] Furthermore, for Step 4, in order to reduce the time of embedding and reasoning and avoid the problem of combinatorial explosion, a two-branch siamese network structure is adopted in the embedding part. The two parts of the input content obtained in Step 3 are respectively input into two identical BERT pre-trained models, and two BERT weight sharings are set to obtain two parts of embedding vectors.

[0020] Furthermore, for Step 5, stack the two parts of one-dimensional embedding vectors of the head entity, relation, and tail entity obtained in Step 4, which have a length of 768 and a channel number of 1, according to the original dimensions to obtain a one-dimensional embedding vector with a length of 768 and a channel number of 2.

[0021] Further, for step 6, apply a one-dimensional convolutional kernel to the stacked vectors obtained in step 5 for convolution, and project the obtained features onto a two-dimensional plane to generate a feature map.

[0022] Further, for step 7, on the premise that the neural network can converge, as the number of network layers increases continuously, the correlation of gradients will continue to decay, resulting in a gradual deterioration of the network performance. However, the residual neural network can effectively solve the above problems and ensure that during the process of stacking the network, the network will not degenerate due to continuous stacking. Therefore, use the ResNet residual neural network to extract features from the feature map generated in step 6 to generate a new feature matrix.

[0023] Further, for step 8, perform average pooling on the feature matrix generated in step 7, and send it into a fully connected layer with a softmax classifier at the top, and calculate the scores of the triples. Determine whether there is a relationship between entities based on the scores, and convert the link prediction task into a simple binary classification task.

[0024] Further, for step 9, in step 2, positive and negative samples are taken out in a ratio of 1:3 respectively to construct training samples. At the same time, in step 8, the link prediction problem is converted into a binary classification problem of whether there is a relationship between the head and tail entities. Therefore, the labels added to the training data are 0 and 1 labels. Send the data into the model and update the weight parameters in the model in the reverse direction to train the model.

[0025] Further, for step 10, during the testing process, category labels are no longer used, but the scores calculated from the triples are used for ranking, and MR, MRR, and Hits@N are used as evaluation metrics. The testing is mainly divided into two parts. One is the link prediction experiment on the general dataset, and the other is to test the dynamic knowledge completion effect on the NELL-ONE dataset.

[0026] Beneficial effects:

[0027] The dynamic knowledge graph completion method based on dual-stream embedding and deep neural network described in the present invention divides the triples into two asymmetric parts, uses the structure of a siamese network to apply two identical BERT pre-trained models for knowledge embedding, saves a large amount of time cost while introducing knowledge semantic information, and avoids the problem of combinatorial explosion; in the inference stage, a residual neural network is used. Compared with the traditional CNN, the residual neural network has better performance and gets rid of the problem of model degradation that appears as the number of network layers increases. The present invention has good performance and combines the characteristics of high efficiency and accuracy. Description of the Drawings

[0028] Figure 1Schematic diagram of the process of dynamic knowledge graph completion based on dual-stream embedding and deep neural network disclosed in the embodiments of the present invention;

[0029] Figure 2 Structural diagram of the feature extraction ResNet network in the present invention;

[0030] Figure 3 Core framework diagram of the dynamic knowledge graph completion model of dual-stream embedding and deep neural network in the present invention. Detailed implementation manners

[0031] The following will describe in more detail the specific operation steps of a method for dynamic knowledge graph completion based on dual-stream embedding and deep neural network of the present invention with reference to the accompanying drawings. To further clarify the present invention, it should be understood that these examples are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, various equivalent forms of modification of the present invention by those skilled in the art fall within the scope defined by the appended claims of this application.

[0032] A method for dynamic knowledge graph completion based on dual-stream embedding and deep neural network, as Figure 1 shown, includes the following steps:

[0033] Step 1: Obtain the text information of entities and relationships in the data;

[0034] The knowledge graph completion method used in the present invention belongs to the knowledge completion method based on representation learning. In the training and prediction processes, it depends on the semantic information of entities and relationships to infer the relationships between entities. Therefore, first, it is necessary to replace the text information of entities and relationships in the triple.

[0035] For entities and relationships in the knowledge graph, retrieve their corresponding text descriptions in the dataset through their indexes, and use the text descriptions to replace the knowledge in the triple. When the text descriptions of some knowledge do not exist, use their names as the text information.

[0036] (h, r, t) = (T h , T r , T t ) Formula (1)

[0037] where h, r, and t respectively represent the head entity, relationship, and tail entity in the original triple; T h , T r , T t represent the head entity, relationship, and tail entity replaced with the text information of the knowledge.

[0038] Step 2: Perform negative sampling on the data to obtain negative samples;

[0039] In the dataset of the knowledge graph completion task, all triples are considered as objectively existing positive samples. To meet the requirements of model training and performance testing, it is necessary to perform negative sampling on the positive samples in the data to construct corresponding negative samples.

[0040] The present invention constructs negative samples by means of element replacement, that is, randomly replacing the entities in the positive sample triples with other entities. When the replaced triples do not exist in the original knowledge graph, they are used as negative samples and stored in the negative sample list. During the training process, in order to accelerate model convergence and prevent overfitting problems, it is necessary to set the ratio of positive and negative samples according to the model to ensure the balance of positive and negative samples during the training process. After testing, when the ratio of positive and negative samples in the present invention is 1:3, the model converges the fastest.

[0041] Perform negative sampling on the positive sample tp:

[0042]

[0043] Where tp′ represents the negative sample, and h′, r′, t′ represent the replaced head entity, relation, and tail entity respectively, and E represents the set of all entities in the knowledge graph G. Step 3: Combine the head entity and the relation, and at the same time, construct the head entity, relation, and tail entity into the input data form of BERT;

[0044] Divide the triple into two parts: the head entity, relation, and tail entity. This step is one of the core contents of the present invention, which not only meets the data requirements of dual-stream embedding but also preserves the context information between the entity and the relation. The input of BERT has certain format requirements. First, use the tokenizer to perform word segmentation to convert the sentence descriptions of the entity and the relation into tokens, then use the vocabulary to map each token to a single-character id, and finally, add the [CLS] and [SEP] flags to the input sequence at the sentence level.

[0045] For the head entity and the relation:

[0046]

[0047] For the tail entity:

[0048]

[0049] Where, X (h) , X (r) , X (t) represent the head entity, relation, and tail entity respectively. The x [CLS] flag is placed at the beginning of the first sentence, indicating the start of the BERT input sentence; the x [SEP] flag is used to separate the two input sentences.

[0050] Step 4: Input the text representations of the head entity, relation, and tail entity into two BERTs respectively to obtain the embedding vectors of the two parts;

[0051] Use two identical BERTs to perform representation learning on the two parts of the input data constructed in Step 3, and apply max pooling to obtain a vector with a fixed length of 768 as the embedding vector of the knowledge. This vector contains the semantic information of the knowledge and the context information between entity relationships. At the same time, in order to improve the efficiency of knowledge embedding, no fine-tuning of BERT is set.

[0052] The knowledge embedding process is as follows:

[0053]

[0054]

[0055] E h and E r represent the embedding vectors of the head entity, relation, and tail entity respectively. In the experiment, weight sharing of formulas (5) and (6) is set to improve the utilization rate of parameters.

[0056] Step 5: Concatenate the two parts of the embedding vectors according to the original dimensions;

[0057] The present invention abandons the mainstream knowledge graph completion scoring method using vector inner product, but inputs the entire triple into the scoring module. In Step 4, two identical BERTs are used to obtain two one-dimensional embedding vectors with a length of 768 and a channel of 1. In order not to introduce irrelevant information, concatenation is performed using the original dimensions:

[0058] u = (E h :E t ), u ∈ R 2×768 Formula (7)

[0059] u is a one-dimensional vector with a length of 768 and a channel number of 2.

[0060] Step 6: Use one-dimensional convolution to perform matrix upsampling and project it onto a two-dimensional plane to generate a feature map;

[0061] Project the vector at each position i along the combined vector u onto the two-dimensional feature map x i using one-dimensional convolution, where the input channel of the convolution is 2 and the output channel is set to f×f. Therefore, the feature map represents a complete triple.

[0062] Step 7: Input the feature map into multiple residual modules for feature extraction.

[0063] Convolutional neural network CNN has been applied to the field of knowledge graph completion. However, due to the problem of network degradation that occurs when the traditional CNN reaches a certain depth, which severely limits the model's feature extraction ability. To solve the problem of network degradation, we choose to use the ResNet residual neural network to extract features from triples.

[0064] Set 24 residual blocks according to the length of the vector, as Figure 2 shown. Each residual block consists of consecutive 1×1 convolution, 3×3 convolution, and 1×1 convolution. Before each convolutional layer, it is necessary to perform batch normalization on the input features, and then add the ReLU activation function. The first 1×1 convolution is used to reduce the dimension of the features; the second 1×1 convolution is used to restore it to the original dimension.

[0065] If F(X) represents the convolutional operation in the bottleneck block, then the output of the bottleneck block is:

[0066]

[0067] Step 8: Use average pooling on the feature matrix, and then input it into a fully connected layer with a softmax classifier at the top to obtain the triple score.

[0068] The triple features extracted by the fully connected layer do not match the input dimension of the fully connected layer. It is necessary to use average pooling to reduce the dimension of the features. After dimension reduction, the input is fed into the fully connected layer to obtain the score of each triple. Finally, the score is normalized to the 0-1 space through softmax to obtain the final triple score s, which is used to determine the accuracy of a factual triple.

[0069]

[0070]

[0071] Step 9: Calculate the scores of all triples in the training set through the model, and use the scores to further train the model;

[0072] The positive sample label in the training set is 1, and the negative sample label is set to 0. The training data is output to the model in batches. During the training process, the link prediction problem belongs to a binary classification problem. The cross-entropy loss function is used to calculate the loss of the model, and the Adam optimizer is used to update the model parameters in the reverse direction. The optimization goal is to minimize the loss of the positive sample tp and maximize the loss of the negative sample tp'.

[0073] The calculation process of the loss function is:

[0074]

[0075] Among them, D represents the training set containing only positive samples, N(tp) represents the negative samples generated according to tp, and c is the probability that the positive sample tp is predicted as the positive class.

[0076] Step 10: Verify the model link prediction effect and the result of complementing triples that do not appear in the training set on the training set respectively;

[0077] During the testing process, the triple scores are no longer converted into categories, but the calculated triple scores are ranked, and MR, MRR, and Hits@n are used as model evaluation indicators. The smaller the MR value and the larger the MRR and Hits@n values, the better the model performance.

[0078] Two parts of data are used for testing. One part uses the knowledge graph completion general datasets FB15K-237 and WN18RR for link prediction experiments, and the other part tests the complementation effect of entities and relationships that do not appear in the original knowledge graph on the NELL-ONE dataset.

[0079] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A dynamic knowledge graph completion method based on dual-stream embedding and deep neural network, characterized in that The steps are as follows: Step 1: Obtain the text information of entities and relationships in the data; Step 2: Perform negative sampling on the data to obtain negative samples; Step 3: Combine the head entity and the relationship. Meanwhile, construct the head entity, relationship, and tail entity into the input data form of BERT; Step 4: Input the text representations of the head entity, relationship, and tail entity into two BERTs respectively to obtain two parts of embedding vectors; Step 5: Concatenate the two parts of embedding vectors according to the original dimensions; Step 6: Use one-dimensional convolution to increase the dimension of the matrix and project it onto a two-dimensional plane to generate a feature map; Step 7: Input the feature map into several residual modules for feature extraction; Step 8: Use average pooling on the feature matrix, and then input it into a fully connected layer with a Softmax classifier at the top to obtain the triple score; Step 9: Calculate the triple scores of all triples in the training set through the model, and further train the model using the scores; Step 10: Verify the link prediction effect of the model and the result of completing triples that do not appear in the training set on the training set respectively.

2. The dynamic knowledge graph completion method based on dual-stream embedding and deep neural network according to claim 1, wherein Regarding Step 1, when the description of an entity or relationship exists, use the description to replace its name; when the description of an entity or relationship does not exist, directly use the name as the text information.

3. The dynamic knowledge graph completion method based on dual-stream embedding and deep neural network according to claim 2, characterized in that Regarding Step 2, adopt a negative sampling strategy of random replacement. Remove the head entity or tail entity of a given triple and randomly replace it with other entities. When the reorganized triple does not exist in the original knowledge graph, add it to the negative sample list as a negative sample.

4. The dynamic knowledge graph completion method based on dual-stream embedding and deep neural network according to claim 3, wherein Regarding Step 3, use the tokenizer of BERT to tokenize the descriptions or names of entities and relationships, map each word to an id according to the provided dictionary, and then construct the head entity, relationship, and tail entity into the input form of BERT.

5. The dynamic knowledge graph completion method based on dual-stream embedding and deep neural network according to claim 4, characterized in that, Regarding Step 4, input the two constructed parts of the head entity, relationship, and tail entity into two identical BERT pre-trained models respectively, and set the weights of the two BERTs to be shared to obtain two parts of embedding vectors.

6. The dynamic knowledge graph completion method based on dual-stream embedding and deep neural network according to claim 5, wherein, Regarding Step 5, stack the two obtained parts of embedding vectors according to the original dimensions. After stacking, a one-dimensional embedding vector with a length of 768 and a channel number of 2 is obtained.

7. The dynamic knowledge graph completion method based on dual-stream embedding and deep neural network according to claim 6, characterized in that, Regarding Step 6, apply one-dimensional convolution to slide along the stacked vector obtained in Step 5, and project the features obtained by convolution onto a two-dimensional plane to generate a feature map; Regarding Step 7, use the ResNet residual neural network to perform feature extraction on the feature map generated in Step 6 to generate a new feature matrix.

8. The dynamic knowledge graph completion method based on dual-stream embedding and deep neural network according to claim 7, wherein Regarding Step 8, perform average pooling on the generated feature matrix, and send it into a fully connected layer with a softmax classifier at the top, and calculate the score of the triple. Judge whether there is a relationship between entities according to the score, and transform the link prediction problem into a binary classification problem.

9. The dynamic knowledge graph completion method based on dual-stream embedding and deep neural network according to claim 8, characterized in that, Regarding Step 9, add 0 and 1 labels to the training data, send the training data into the model, and update the weight parameters in the model in the reverse direction to train the model.

10. The dynamic knowledge graph completion method based on dual-stream embedding and deep neural network according to claim 9, characterized in that, For step 10, during the testing process, category labels are no longer used, but scores calculated from triples are used for ranking, and MR, MRR, and Hits@N are used as evaluation metrics; the testing is divided into two parts, one is the link prediction experiment on the general dataset, and the other is to test the dynamic knowledge completion effect on the NELL-ONE dataset.

Images