A knowledge graph completion method based on a convolutional neural network

Abstract

The application discloses a kind of knowledge graph completion methods based on convolutional neural network, it includes: by the embedding vector corresponding to head, tail entity and relation is combined to form three different embedding matrix respectively, after being divided into three, it is carried out convolution operation, deep mining the feature information between entity and relation under the same dimension.Judge the score condition of triple according to the length of the continuous vector obtained by dynamic routing process again.Finally, according to the score result, the missing triple is predicted, and the completion task of knowledge graph is completed.The application combines three convolution and dynamic routing algorithm, utilizes the characteristics of the powerful feature extraction capability of convolutional neural network, through three convolution operations, the ability of learning deep association between entity and relation embedding vector can be enhanced.Routing algorithm is used instead of maximum pooling layer to minimize the information loss in feature learning, effectively improve the accuracy and integrity of knowledge graph completion.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning, natural language processing and knowledge graph completion, and relates to a knowledge graph completion method based on a convolutional neural network, which is mainly used for automatically completing a knowledge graph to make the knowledge graph tend to be complete. BACKGROUND

[0002] With the unprecedented growth of data worldwide, the information on the Internet is vast, and in this era of big data, knowledge graph is a very important technology. Knowledge graph describes concepts, entities and their relationships in a structured triple form, which can effectively organize and represent knowledge, such as in search engines, personal assistants, customer service robots, intelligent recommendation systems and other aspects, so that knowledge can be more effectively utilized in advanced applications.

[0003] However, since most of the initial knowledge graphs are manually completed, a large number of hidden entities and relationships have not been mined, and the phenomenon of structure and content missing is almost universal in all knowledge graphs. Therefore, in order to make the knowledge graph provide more comprehensive and reliable services for various fields, the existing knowledge graph needs to be continuously supplemented, improved and updated in time.

[0004] In view of the above problems, i.e. the incompleteness of the relationship and head and tail entities in the current real-world knowledge graph, many scholars at home and abroad have proposed a large number of knowledge graph completion models, which have played a certain role in solving the problem of incomplete knowledge graph, but there is still a lot of room for improvement in the accuracy and efficiency of knowledge graph completion.

[0005] Knowledge graph completion is the prediction of information such as essence, relationship and attribute, and is also an important means of discovering new knowledge. Knowledge graph completion focuses on expanding the knowledge graph, and essentially evaluates the effectiveness of triples. That is, the main purpose of knowledge graph completion is to predict and supplement the missing parts in triples, and to evaluate the rationality of triples that do not exist in the knowledge graph, so that the knowledge graph tends to be complete.

[0006] The main knowledge graph completion methods at present are mainly divided into two categories, one is the traditional knowledge graph completion method, and the other is the knowledge graph completion method based on knowledge representation learning.

[0007] Traditional knowledge graph completion methods have high computational complexity, poor portability and scalability. The main use of knowledge graph inference rules and network structure is greatly limited in knowledge graph completion, resulting in general completion effect. The knowledge graph completion method based on knowledge representation learning performs low-dimensional embedded representation on entities and relations. Commonly used methods include translation model method, which can improve computational efficiency due to simple training. However, the deep correlation between entity and relation embedding vectors cannot be captured, resulting in limited fitting capacity. SUMMARY

[0008] The purpose of the present application is to overcome some of the deficiencies in the prior art, improve the accuracy of knowledge graph completion, and propose a knowledge graph completion method based on convolutional neural network.

[0009] The present application utilizes the powerful nonlinear feature expression and feature extraction capabilities of convolutional neural network, and through three-way convolution operation, it can capture deep feature information between entities and relations. Through the dynamic routing algorithm process, the obtained feature information can be transmitted without loss. Finally, the credibility of the current knowledge graph triple is judged according to the score result. According to the result, the missing triple is predicted to complete the knowledge graph completion task. On the same data set, this method has achieved good results compared with other benchmark models in most cases, which can effectively improve the accuracy and integrity of knowledge graph completion.

[0010] To achieve the above purpose, the present application adopts the following technical solutions. A knowledge graph completion method based on convolutional neural network comprises the following steps:

[0011] Step S1: Process the knowledge graph experimental data set to obtain head and tail entities and existing relations;

[0012] Step S2: Generate corresponding embedding vectors for the head and tail entities and relations obtained in step S1;

[0013] Step S3: Combine the embedding vectors obtained in step S2 and divide them into three paths for convolution operation to obtain feature information;

[0014] Step S4: Encapsulate the feature information of the same dimension in step S3 together, called an initial capsule u i , the initial capsule u i outputs a continuous vector through a dynamic routing process;

[0015] Step S5: Use the length of the continuous vector obtained in step S4 to judge the score of the triple, and then evaluate the correctness of the triple according to the score, and complete the knowledge graph completion task according to the judgment result.

[0016] Further, in step S1, the specific content and method of processing the knowledge graph experimental data set to obtain the head and tail entities and the specific content and method of the existing relationship are as follows: the selected experimental data set data is preprocessed, and the samples in the data are divided into a training set, a test set and an evaluation set according to a 7:2:1 ratio. Each triple is represented as (h, r, t), representing the head entity, the relationship, and the tail entity.

[0017] Further, in step S2, the specific content and method of generating the head and tail entities and the relationship corresponding to the embedding vector obtained in step S1 are as follows: the TransE model with fewer parameters and lower computational complexity is selected to train the low-dimensional embedding vectors v h 、v r 、v t of the head entity, the relationship and the tail entity, wherein the embedding dimensions are all K, and the trained head entity, relationship and tail entity embedding vectors are used as input.

[0018] Further, in step S3, the specific content and method of combining the embedding vectors obtained in step S2 and then performing convolution operation in three ways to obtain feature information are as follows: in the convolution layer, the three embedding vectors of each triple, i.e. the head entity, the relationship and the tail entity v h 、v r 、v t are combined respectively to form M hrt 、M hr 、M rt three different embedding matrices, and different size convolution kernels are used to obtain deep feature information in the embedding vectors. That is, the application uses 1*3 and 1*2 different size convolution kernels, each size has N convolution kernels, and different feature maps are generated by using the convolution kernels for continuous operation, and the deep feature information of the embedding matrix is extracted according to the following formula:

[0019] M’ hrt =WM hrt +b;

[0020] M hr =WM hr +b;

[0021] M’ rt =WM rt +b;

[0022] Wherein, M’ hrt 、M’ gr 、M’ rt are the corresponding feature matrices, W is the weight matrix, and b is the bias parameter.

[0023] More specifically, step S3 further comprises performing an element product, i.e. Hadamard product, on the feature matrix M' gr and the feature matrix M' rt to obtain a feature matrix M" hrt Then, an element product is performed on the feature matrix M" hrt and the feature matrix M' hrt to obtain a final required feature mapping matrix M'''' hrt ,

[0024] M" hrt = M' hr ⊙ M' rt ;

[0025] M hrt = M hrt ⊙ M hrt ;

[0026] wherein, ⊙ represents an element product.

[0027] Further, in step S4, the feature information of the same dimension in step S3 is encapsulated together, referred to as an initial capsule u i The specific content and method of outputting a continuous vector by the dynamic routing process of the initial capsule u i is that, in the dynamic routing process, the feature results of the same dimension extracted by different convolution kernels are encapsulated as an initial capsule, and the length of the vector is obtained by squashing. The output is multiplied by a weight matrix and summed to generate a smaller size capsule s, and the vector length is compressed to a range of 0 to 1 by a compression function squash(), and the formula is as follows:

[0028]

[0029] Further, in step S5, the specific content and method of using the length of the continuous vector to judge the score of the triple, and then judging the correctness of the triple according to the score to complete the knowledge graph completion task is that the score function f(h, r, t) of the triple is:

[0030] f(h, r, t) = ||Caps'(g(Conv1(M hrt )⊙δ(Conv2(M hr )⊙Conv3(M rt ))))||;

[0031] wherein, g represents a ReLu activation function, δ represents a Sigmoid activation function,

[0032] Conv j represents a convolution operation, j∈{1,2,3}, and the specific content is as follows:

[0033] Conv1 represents a convolution operation on the embedding matrix M containing the head entity, the relation and the tail entity at the same time hrt Convolution operation is performed;

[0034] Conv2 represents a convolution operation on the embedding matrix M containing the head entity and the relation at the same time hr Convolution operation is performed;

[0035] Conv3 represents a convolution operation on the embedding matrix M containing the relation and the tail entity at the same time rt Convolution operation is performed;

[0036] For training model parameters, the loss function is minimized using Adam, and the minimization L is taken as the final target, and the loss function L is calculated as follows:

[0037]

[0038] λ is the weight of the regularization term, is the regularization term, which aims to prevent the model from overfitting,

[0039]

[0040] In the model training stage, T is a set of positive triplets, and T' is a set of negative triplets.

[0041] Compared with the prior art, the beneficial effects of the technical solution are:

[0042] I. The feature information of the same dimension of the triplets can be mined more deeply, and the feature variables of the entities and the relations in the knowledge graph can be learned more fully. By using three different types of convolution processes, the strong feature extraction and interpretation capabilities of the convolutional neural network can be utilized to learn the deep association between the entity and relation embedding vectors.

[0043] II. Valuable information can be preserved to the maximum extent, and a routing process is used instead of a pooling layer. The pooling layer can reduce the features of the entity and relation vectors in the knowledge graph completion. The routing process used in the model can preserve all the feature information of the entities and relations in the knowledge graph, and reduce the information loss in feature learning.

[0044] III. The effect of knowledge graph completion can be effectively improved. The convolutional neural network and the dynamic routing algorithm are combined to effectively capture the relationship between the features in the vector, enhance the feature interaction between the entity vector and the relation vector, and solve the information loss problem. On the same data set, the knowledge graph completion method based on the convolutional neural network has achieved good results in most cases compared with many benchmark models. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 Figure 1 is a schematic diagram of a method according to the present application,

[0046] Figure 2 Figure 2 is a schematic diagram of a structure according to the present application,

[0047] Figure 3 Figure 3 is a schematic diagram of a routing process according to the present application,

[0048] Figure 4 Figure 4 is a schematic diagram of a model of a completion algorithm according to the present application. DETAILED DESCRIPTION

[0049] The technical solutions of the present application will be further described below in combination with the drawings and examples. In order to verify that the knowledge graph completion method based on a convolutional neural network according to the present application can effectively complete a knowledge graph, the present application is verified by examples, so as to embody the effectiveness and practicality of the present application.

[0050] The present application uses two public knowledge graph datasets WN18RR and FB15k-237, which are composed of information extracted from WordNet and Freebase knowledge bases, and are subsets of WN18 and FB15k. In order to make the dataset more realistic, the triplets with inverse relations have been deleted in its subset, so that the experimental results are more credible. Table 1 lists the statistical data.

[0051] Table 1 Dataset Statistics

[0052]

[0053] As shown in Figure 1 , using the dataset WN18RR as an example, the specific steps of the knowledge graph completion method based on a convolutional neural network are as follows:

[0054] Step S1: Process the knowledge graph experimental dataset to obtain the head and tail entities and the existing relations. The specific content and method are to perform data preprocessing on the dataset WN18RR, divide the samples in the data into a training set, a test set and an evaluation set according to a 7:2:1 ratio, and mark the existing triplets in the used samples as 1.

[0055] Step S2: Generate the corresponding embedding vectors of the head and tail entities and the relations obtained in step S1. Initialize the entities and relations to obtain the initialized embedding representations of the entities and relations, respectively, and fuse to generate an embedding matrix. Preferably, a translation model is used, and the TransE model with fewer parameters and low computational complexity is selected by the present application to perform knowledge graph embedding. The entity and relation embedding dimensions are both K, and the embedding vectors v h , v r , vt As input.

[0056] Step S3: Combine the embedding vectors obtained in Step S2 and perform convolution operations on three paths to obtain feature information. This is done by converting the three embedding vectors v of each triplet into a single vector. h v r v t Combine them separately to form M hrt M hr M rt Three different embedding matrices are used. In the convolutional layers, the common non-linear activation function ReLU is employed as the activation function to enhance the expressive power of the model in this invention. hrt M hr M rt Three different embedding matrices are used with different convolutional kernels to learn deep feature information from the vectors. Convolutional kernels of different sizes (1x3 and 1x2) are employed, with N kernels for each size. Different feature maps are generated by sequentially operating on these kernels, and the deep feature information of the embedding matrix is ​​extracted according to the following formula:

[0057] M' hrt =WM hrt +b;

[0058] M' hr =WM hr +b;

[0059] M' rt =WM rt +b;

[0060] Among them, M' hrt M' hr M' rt Let W be the corresponding feature matrix, W be the weight matrix, and b be the bias parameter.

[0061] Step S4: Encapsulate the feature information of the same dimension from Step S3 together, called an initial capsule u. i , the initial capsule u i The dynamic routing process outputs a continuous vector. See the routing process diagram of this invention. Figure 3 The feature information of the same dimension extracted by different convolutional kernels is encapsulated into an initial capsule u. i Then, a linear combination is performed; this is the initial capsule u. i The important process of connecting to smaller capsules is formulated as follows:

[0062]

[0063] v = squash(s);

[0064] c i is the coupling coefficient, i.e., the probability value of the initial capsule being connected to the next layer capsule, and v is the output vector.

[0065] The vector length is compressed into the range of 0 to 1 by a compression function squash(), and the formula is as follows:

[0066]

[0067] Step S5: The continuous vector obtained in step S4 is used to judge the score of the triple, and the correctness of the triple is evaluated through the score, and the completion task of the knowledge graph is completed according to the judgment result. According to the three evaluation indexes commonly used in the knowledge graph completion, if correct, the correct triple is added to the knowledge graph, and if incorrect, the incorrect triple is discarded.

[0068] Referring to the method structure diagram of the application Figure 2 , input the triple existing in the knowledge graph, and finally output the score of the triple after training.

[0069] Referring to the model diagram of the completion algorithm of the application Figure 4 The model of the completion algorithm of the application mainly comprises an embedding layer, a convolution layer, a capsule layer and an output layer, and finally outputs the score of the triple through convolution and dynamic routing operations.

[0070] The definition symbols used in the application are as follows:

[0071] The knowledge graph G is defined as G = {E, T, R}, wherein E represents an entity set, T represents a fact in the knowledge graph G, T = {(h, r, t) | h, t E, r R}, and each triple (h, r, t) T, wherein h is a head entity, r is a relationship between the head entity and the tail entity, and t is a tail entity. R is a relationship set.

[0072] The embedding vector v is defined as h , v r , v t , wherein v h is a K-dimensional embedding vector of the head entity, v r is a K-dimensional embedding vector of the relationship, and v t is a K-dimensional embedding vector of the tail entity, wherein the embedding dimension K = 100.

[0073] The embedding matrix M is defined as hrt , M hr , M rt , M hrt represents the head entity embedding vector v h , the relationship embedding vector v r, tail entity embedding vector v t embedding matrix combined together;

[0074] M hr denoted as head entity embedding vector v h , relation embedding vector v r embedding matrix combined together;

[0075] M rt denoted as relation embedding vector v h , tail entity embedding vector v t embedding matrix combined together;

[0076] define feature matrix M' hrt , M' hr , M' rt , M' hrt denoted as M hrt feature matrix obtained after the embedding matrix is convolved by N convolution kernels with a size of 1*3, wherein N=400;

[0077] M' hr denoted as M hr feature matrix obtained after the embedding matrix is convolved by N convolution kernels with a size of 1*2;

[0078] M' rt denoted as M rt feature matrix obtained after the embedding matrix is convolved by N convolution kernels with a size of 1*2.

[0079] define element product feature matrix M" grt =M' hr ⊙M' rt , M''' hrt =M" hrt ⊙M' hrt .

[0080] wherein ⊙ represents element product, also known as Hadamard product;

[0081] final feature mapping matrix M''' hrt is M" hrt and M' hrt obtained through Hadamard product.

[0082] define u i as the i th capsule, wherein i∈{1, 2, 3, …, K}, and the number of neurons in each capsule is equal to the number of convolution kernels.

[0083] In the dynamic routing process, the same dimension feature information extracted by different convolution kernels is encapsulated as an initial capsule u iThe number of iterations of the routing operation is set to 1 time. Among them, the output is multiplied by the weight matrix and summed to generate a smaller size capsule s, and the vector length is compressed to the range of 0 to 1 through the compression function squash(). The formula is as follows:

[0084]

[0085] The score function f(h, r, t) of the triple is:

[0086] f(h, r, t) = ||Caps'(g(Conv1(M hrt )⊙δ(Conv2(M hr )⊙Conv3(M rt ))))||;

[0087] Where g represents the ReLu activation function, and δ represents the Sigmoid activation function.

[0088] Conv j represents convolution operation, j∈{1,2,3}, as follows:

[0089] Conv1 represents convolution operation on the embedding matrix M hrt containing the head entity, the relationship, and the tail entity at the same time;

[0090] Conv2 represents convolution operation on the embedding matrix M hr containing the head entity and the relationship at the same time;

[0091] Conv3 represents convolution operation on the embedding matrix M rt containing the relationship and the tail entity at the same time.

[0092] For training model parameters, the Adam optimizer is used to minimize the loss function, and the minimization L is taken as the final training target, and the loss function L is calculated as follows:

[0093]

[0094] λ is the weight of the regularization term, is the regularization term.

[0095]

[0096] The evaluation results are evaluated by using three evaluation indexes commonly used in the knowledge graph completion task, namely, mean rank MR (Mean Rank), mean reciprocal rank MRR (Mean Reciprocal Rank), and top N percentage (Hits@N). The smaller the MR is, the more front the real triplets in the knowledge graph are ranked, and the MRR is opposite. The Hits@N refers to the percentage of entering the top N, and the higher the Hits@N is, the better the effect is.

[0097] As can be seen from Table 2, compared with other benchmark models such as TransE, DistMult, ConvE, ConvKB, InteractE, etc., the method of the present application has better performance on the three evaluation indexes on the experimental data set WN18RR, and has better completion effect.

[0098] Table 2 experimental results

[0099]

[0100] In the present application, the convolutional neural network is combined with the dynamic routing algorithm, the relationship between the features in the vector is effectively captured, the feature interaction of the entity vector and the relationship vector is enhanced, and the information loss problem is solved. On the same data set, the knowledge graph completion method based on the convolutional neural network and the dynamic routing algorithm achieves good effect in most cases, and can effectively improve the accuracy and integrity of the knowledge graph completion.

[0101] The above only describes the embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for knowledge graph completion based on a convolutional neural network, characterized in that, The method comprises the following steps: Step S1: processing the selected knowledge graph experimental data set to obtain head and tail entities and existing relationships; Step S2: generating corresponding embedding vectors for the head and tail entities and relationships obtained in step S1; Step S3: combining the embedding vectors obtained in step S2 and performing convolution operation in three paths to obtain feature information; The feature information of the same dimension is encapsulated together, and is referred to as an initial capsule u i The initial capsule u i The specific content and method of outputting a continuous vector through the dynamic routing process are that the feature information of the same dimension extracted by different convolution kernels is encapsulated as an initial capsule through the dynamic routing process, and the length of the continuous vector is obtained through squeezing. Wherein the output is multiplied by the weight matrix and summed to generate a smaller size capsule s, the vector length is compressed to the range of 0 to 1 through the compression function squash(), and the formula is as follows: Step S4: encapsulate the feature information of the same dimension in step S3 together, called an initial capsule u i , the initial capsule u i output a continuous vector through a dynamic routing process; The length of the continuous vector is used to judge the score of the triple, and the correctness of the triple is judged by the score, and the completion task of the knowledge graph is completed according to the judgment result, wherein the score function f(h, r, t) of the triple is: f(h, r, t) = ||Caps'(g(Conv1(M hrt ) ⊙ δ(Conv2(M hr ) ⊙ Conv3(M rt ))))||; Wherein, g represents the ReLu activation function, and δ represents the Sigmoid activation function, Conv j represents a convolution operation, j e {1,2,3}, and is specified as follows: Conv1 represents a convolution operation on the embedding matrix M containing the head entity, the relation, and the tail entity at the same time hrt performing a convolution operation; Conv2 is represented as a convolution operation on the embedding matrix M containing both head entities and relations simultaneously hr performing a convolution operation; Conv3 is represented as a convolution operation on the embedding matrix M containing both relations and tail entities rt performing a convolution operation; For training model parameters, use Adam to minimize the loss function, and take the minimized L as the final target, and the loss function L calculation formula is as follows: λ is the weight of the regularization term, is the regularization term, whose purpose is to prevent the model from overfitting, In the model training stage, T is a set of positive example triples, and T' is a set of negative example triples; Step S5: using the length of the continuous vector to judge the score of the triple, and evaluating the correctness of the triple by the score, and completing the completion task of the knowledge graph according to the judgment result. 2.The convolutional neural network based knowledge graph completion method of claim 1, wherein, In step S1, the specific content and method of processing the selected knowledge graph experimental data set to obtain head and tail entities and existing relationships are as follows: preprocessing the selected knowledge graph experimental data set data, and dividing the samples in the data into training set, test set and evaluation set according to the proportion of 7:2:

1. 3.The convolutional neural network based knowledge graph completion method of claim 1, wherein, In step S2, the specific content and method of generating the corresponding embedding vectors of the head-tail entity and the relationship obtained in step S1 are that a TransE model with few parameters and low computational complexity is selected to train low-dimensional embedding vectors v h r t of the head entity, the relationship and the tail entity, where the embedding dimensions are K, and the trained low-dimensional embedding vectors v h r t of the head entity, the relationship and the tail entity are taken as input.​​​​ 4.The convolutional neural network based knowledge graph completion method of claim 1, wherein, In step S3, the obtained embedding vectors in step S2 are combined and then divided into three paths for convolution operation to obtain the specific content and method of feature information. In the convolution layer, the three embedding vectors of the triple, i.e., the head entity, the relation, and the tail entity v h , r , t are combined respectively to form M hrt , M hr , M rt three different embedding matrices, and different sizes of convolution kernels are used to obtain deep feature information in the embedding vectors, i.e., 1*3 and 1*2 different size convolution kernels, each size has N convolution kernels, and different feature maps are generated by using the convolution kernels for continuous operation, and the deep feature information of the embedding matrix is extracted according to the following formula: M' hrt = WM hrt + b; M' hr = WM hr + b; M' rt = WM rt + b; wherein M hrt , M hr , M rt are the corresponding feature matrices, W is a weight matrix, and b is a bias parameter. 5.The convolutional neural network based knowledge graph completion method of claim 4, characterized in that, Step S3 also comprises the element-wise multiplication of the feature matrix M' hr with the feature matrix M' rt The element-wise multiplication, i.e. Hadamard product, of the feature matrix M" hrt with the feature matrix M" hrt and the feature matrix M' hrt results in the final desired feature mapping matrix M ‘’‘ hrt , M" hrt = M' hr ⊙ M' rt ; M''' hrt = M" hrt O M' hrt ; Wherein, ⊙ represents element product.

Images