Chinese knowledge graph completion method, device, computer equipment and storage medium

Abstract

The present application relates to a Chinese knowledge graph completion, device, computer equipment and storage medium. The method uses an average pooling strategy to effectively solve the problem of insufficient sequence representation ability of the BERT model, and strengthens the word position information through the BiGRU network to alleviate the negative impact of weakening position information caused by the parallel operation of the BERT model; then, the head entity, relationship, and tail entity are converted into text sequences, combined with the entity description information as the input of the model, and the knowledge completion task is regarded as a sentence classification task. The pre-trained language model is used for knowledge completion, which overcomes the shortcomings of traditional methods that ignore semantic features and solves the problem of insufficient background knowledge; by superimposing a linear classification layer, the knowledge completion problem is converted into a classification problem. This method can be used to quickly and efficiently achieve Chinese knowledge graph completion.

Description

Technical Field

[0001] The present application relates to the field of knowledge graph technology, and in particular to a Chinese knowledge graph completion method, device, computer equipment and storage medium. Background Art

[0002] Knowledge graphs are usually semantic relationship graphs representing entities and relationships, stored in the form of triples, which consist of a head entity h, a tail entity t, and a relationship r between the head entity and the tail entity. These triples are usually used to describe facts. Compared with traditional databases, knowledge graphs are more intuitive and have better search capabilities. Therefore, knowledge graphs are widely used in finance, military, medical and other fields.

[0003] As the scale of knowledge graphs grows, the amount of data stored increases, and knowledge graphs become sparse and incomplete, resulting in knowledge missing and reducing graph quality. Knowledge completion is a technology that solves the problem of knowledge missing, with the goal of making the knowledge graph more complete. Knowledge completion technology first needs to represent and model the knowledge graph, then predict the missing part of a triple, and predict whether the triple is valid and reasonable.

[0004] Most existing knowledge graph completion methods ignore semantic information, and most knowledge completion studies are based on English datasets, such as FB15K-237Error! Reference source not found., WN18RRError! Reference source not found., etc., lacking high-quality Chinese labeled data. At present, in addition, in terms of the knowledge graph construction process, Chinese knowledge graphs and English knowledge graphs differ in grammatical structure, word formation methods, sentence logic structure, etc., while Chinese grammatical structure and language logic are more flexible and changeable, with stronger context relevance, and more synonyms and polysemous words. Whether the previous knowledge completion Chinese knowledge graph completion model is applicable to Chinese datasets is worth studying.

[0005] In recent years, the pre-trained Chinese knowledge graph completion model has achieved good results in natural language processing (NLP) tasks. For example, ELMo uses bidirectional LSTM to solve the problem of representing polysemous words, and GPT uses the Transformer Decoder architecture to complete tasks such as classification and similarity; while the BERT Chinese knowledge graph completion model comprehensively considers the ideas of ELMo bidirectional encoding and GPT using the Transformer encoder, uses the Transformer bidirectional encoder as a feature extractor, and uses a large amount of unlabeled data to pre-train the Chinese knowledge graph completion model to obtain a general language representation, and then uses a small amount of labeled data for fine-tuning to cope with specific tasks, solving the problem of lacking a large amount of high-quality labeled data and effectively capturing contextual information.

[0006] However, when using the BERT Chinese knowledge graph completion model for knowledge completion tasks, there are the following problems. First, the BERT Chinese knowledge graph completion model generally uses [CLS] output values ​​to represent sequences, but sequences that are too long will accumulate too much information in the [CLS] output value, affecting the representation ability of the Chinese knowledge graph completion model and reducing the efficiency of knowledge completion. Secondly, the BERT Chinese knowledge graph completion model processes text in parallel and only relies on position embedding to represent the positional relationship between texts, which weakens the word position information. Summary of the invention

[0007] Based on this, it is necessary to provide a Chinese knowledge graph completion method, device, computer equipment and storage medium to address the above technical problems.

[0008] A Chinese knowledge graph completion method, the method comprising:

[0009] Obtain a knowledge graph to be processed, where the knowledge graph to be processed is stored as a triple consisting of a head entity, a tail entity, and a relationship between the head entity and the tail entity.

[0010] The triples are input into the input layer, the head entity, the tail entity and the relationship are represented as three paragraphs of text, and the three paragraphs of text are processed to obtain an input sequence.

[0011] The input sequence is input into the Bert model layer, and after the input sequence is encoded, it is calculated by a multi-head self-attention mechanism to obtain a feature matrix containing contextual semantic information.

[0012] The feature matrix is ​​input into the average pooling layer to obtain a new feature matrix.

[0013] The new feature matrix is ​​input into the BiGRU layer, processed by the serialized neural network, and the positional relationship information between words is learned to obtain a sequence feature matrix.

[0014] The sequence feature matrix is ​​input into the output layer to obtain a binary classification score result.

[0015] A Chinese knowledge graph completion device, comprising:

[0016] The module for acquiring the knowledge graph to be processed is used to acquire the knowledge graph to be processed, wherein the storage method of the knowledge graph to be processed is a triple consisting of a head entity, a tail entity, and the relationship between the head entity and the tail entity.

[0017] The Chinese knowledge graph completion module is used to input the triples into the input layer, represent the head entity, the tail entity and the relationship as three paragraphs of text, and process the three paragraphs of text to obtain an input sequence; input the input sequence into the Bert model layer, encode the input sequence, and then calculate it through a multi-head self-attention mechanism to obtain a feature matrix containing contextual semantic information; input the feature matrix into the average pooling layer to obtain a new feature matrix; input the new feature matrix into the BiGRU layer, and through serialized neural network processing, learn the position relationship information between words to obtain a sequence feature matrix; input the sequence feature matrix into the output layer to obtain a binary classification score result.

[0018] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0019] A computer-readable storage medium stores a computer program, which implements the steps of any of the above methods when executed by a processor.

[0020] The above-mentioned Chinese knowledge graph completion, device, computer equipment and storage medium, the method uses the average pooling strategy to effectively solve the problem of insufficient sequence representation ability of the BERT model, and strengthens the word position information through the BiGRU network to alleviate the negative impact of weakening position information caused by the parallel operation of the BERT model; then, the head entity, relationship, and tail entity are converted into text sequences, combined with the entity description information as the input of the model, the knowledge completion task is regarded as a sentence classification task, and the pre-trained language model is used for knowledge completion, which overcomes the shortcomings of traditional methods that ignore semantic features and solves the problem of insufficient background knowledge; by superimposing a linear classification layer, the knowledge completion problem is converted into a classification problem. This method can quickly and efficiently realize the completion of the Chinese knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of a process of completing a Chinese knowledge graph in one embodiment;

[0022] Figure 2 It is a structural block diagram of a Chinese knowledge graph completion model in one embodiment;

[0023] Figure 3 A GRU network structure in one embodiment;

[0024] Figure 4 The Bert core layer architecture in another embodiment;

[0025] Figure 5 This is the experimental result of the English and Chinese public data sets in another embodiment;

[0026] Figure 6 The experimental duration of different neural network modules in one embodiment;

[0027] Figure 7 It is a structural block diagram of a Chinese knowledge graph completion device in one embodiment;

[0028] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] Chinese knowledge graph completion model, referred to as: MpBERT-BiGRU (Mean pooling- Bidirectional encoding layer Representation from Transformers-Bidirectional Gated Recurrent Unit) model.

[0031] In one embodiment, Figure 1 As shown, a Chinese knowledge graph completion method is provided, which includes the following steps:

[0032] Step 100: Obtain the knowledge graph to be processed. The knowledge graph to be processed is stored as a triple consisting of a head entity, a tail entity, and a relationship between the head entity and the tail entity.

[0033] Specifically, a knowledge graph is usually a semantic relationship graph that represents entities and relationships. It is stored in the form of triples, which consist of a head entity h, a tail entity t, and the relationship r between them. These triples are usually used to describe facts. Knowledge graphs are widely used in finance, medicine and other fields.

[0034] The knowledge graph to be processed is a Chinese knowledge completion dataset, which can be an industry knowledge graph based on professional fields such as medicine and finance and / or a general field knowledge graph.

[0035] Step 102: Input the triples into the input layer, represent the head entity, tail entity and relationship as three paragraphs of text, and process the three paragraphs of text to obtain an input sequence.

[0036] Specifically, for example, the triple <Forbidden City, the city it belongs to, Beijing>, such as Figure 2 As shown, the header entity text That is, the description text of "Forbidden City" is "The Forbidden City in Beijing is the royal palace of the Ming and Qing dynasties, formerly known as the Forbidden City, located at the center of the central axis of Beijing...", and the relationship text That is, the text "city", the end entity text That is, the description text of "Beijing" is "Beijing, abbreviated as "Jing", formerly known as Yanjing and Beiping, is located in the northern part of the North China Plain, adjacent to Tianjin in the east, and adjacent to Hebei in the rest of the country..."

[0037] The model converts triples into text sequences as model input and considers the knowledge completion task as a sentence classification task. It aims to use a pre-trained language model for knowledge completion, overcome the shortcomings of traditional methods that ignore semantic features, and use a small amount of labeled data to fine-tune the model to achieve excellent results.

[0038] Step 104: Input the input sequence into the Bert model layer, encode the input sequence, and then calculate it through the multi-head self-attention mechanism to obtain a feature matrix containing contextual semantic information.

[0039] Step 106: Input the feature matrix into the average pooling layer to obtain a new feature matrix.

[0040] Specifically, the average pooling strategy is adopted to alleviate the problem that the traditional BERT model uses the [CLS] label output value as the representation vector and has insufficient sequence representation ability.

[0041] Step 108: Input the new feature matrix into the BiGRU layer, and after being processed by the serialized neural network, learn the positional relationship information between words to obtain a sequence feature matrix.

[0042] Specifically, a BiGRU network is superimposed in the model to fully learn the positional relationship between words and alleviate the negative impact of weakening position information caused by parallel computing of the BERT model.

[0043] GRU uses the update gate to control the degree to which the current time t receives information from the previous time t-1, and controls the degree to which the information from the previous time t-1 is ignored by the reset gate. The unidirectional GRU network structure is as follows: Figure 3 shown.

[0044] The BiGRU layer consists of two GRU sub-networks, which perform forward propagation and backward propagation respectively. The input of the BiGRU layer is the output feature vector of the average pooling layer. The component e at time t t , the main workflow at time t is as follows:

[0045] 1) First splice e t And the hidden layer value h of the GRU network at the previous moment in the same direction t-1 , calculate the reset gate coefficient r t ∈[0,1], selectively forget the hidden layer value h of the previous GRU network t-1 , update to the candidate hidden layer value As shown in equations (1) and (2);

[0046] 2) Calculate the update gate coefficient z again t ∈[0,1], choose e t and h t-1 Important information, use z t Selectively update the hidden layer value h t , as shown in equations (3) and (4);

[0047] 3) After updating the hidden layer values, the output of the BiGRU layer is finally obtained, that is, the feature matrix E with a dimension of L×H, as shown in formula (5).

[0048] r t =σ(h t-1 W r +e t W r +b r ) (1)

[0049]

[0050] z t =σ(h t-1 W z +e t W z +b z ) (3)

[0051]

[0052] E=(h 1 ,h 2 ,...,h t ,...,h H ) (5)

[0053] Where W r , W z is the weight matrix, b r , b z is the bias vector, and ⊙ represents the multiplication of matrix elements.

[0054] Step 110: Input the sequence feature matrix into the output layer to obtain the binary classification score result.

[0055] Specifically, Figure 2 As shown in Figure 1, the Chinese knowledge graph completion model consists of an input layer, a Bert model layer, an average pooling layer, a BiGRU layer, and an output layer. The Chinese knowledge graph completion model uses a binary classification function as the scoring function.

[0056] In the above-mentioned Chinese knowledge graph completion method, the method uses the average pooling strategy to effectively solve the problem of insufficient sequence representation ability of the BERT model, and strengthens the word position information through the BiGRU network to alleviate the negative impact of weakening position information caused by the parallel operation of the BERT model; then, the head entity, relationship, and tail entity are converted into text sequences, combined with the entity description information as the input of the model, and the knowledge completion task is regarded as a sentence classification task. The pre-trained language model is used for knowledge completion, which overcomes the shortcomings of traditional methods that ignore semantic features and solves the problem of insufficient background knowledge; by superimposing a linear classification layer, the knowledge completion problem is converted into a classification problem. This method can quickly and efficiently realize the completion of the Chinese knowledge graph.

[0057] In one embodiment, the input sequence in step 102 is composed of text sequences representing a head entity, a relationship, and a tail entity; the text sequences of the head entity and the tail entity are descriptions of the head entity and the tail entity respectively; the text sequence of the relationship is the name of the relationship; each input sequence starts with a special classification label and consists of three paragraphs of text, each of which is separated by another special label; the input sequence expression is:

[0058]

[0059] Where: Seq(h,r,t) is the input sequence, h is the head entity, r is the relationship between the head entity and the tail entity, t is the tail entity, and are the text sequences of the head entity and the tail entity respectively, is the text sequence of the relation, [CLS] is the classification label, and [SEP] is the special label.

[0060] Entity description can integrate entity knowledge into the model, instill background knowledge into the model, and make BERT a model with a "knowledge base".

[0061] In one embodiment, the Bert model layer includes a Bert embedding layer and a Bert encoding layer; step 104 includes: inputting the input sequence into the Bert embedding layer, converting the input sequence into three embedding matrices, and then summing the three embedding matrices to obtain an output embedding matrix; the three embedding matrices are a word embedding matrix representing the embedding value of each word, a position embedding matrix representing the position information of each word, and a type embedding matrix for distinguishing different parts of a triple; the output embedding matrix is ​​input into the Bert encoding layer, and semantic information is fused through a multi-head attention mechanism to obtain a feature matrix containing contextual semantic information.

[0062] Specifically, the Bert embedding layer is responsible for the initial encoding of the triple input, and the obtained vector is further integrated with semantic information by the Bert encoding layer.

[0063] Define L as the length of the input sequence Seq(h,r,t), and H as the hidden layer size of the model. The Bert embedding layer is responsible for processing the serialized input and converting the input sequence Seq(h,r,t) into three embedding matrices, namely the word embedding matrix E w , position embedding matrix E p and type embedding matrix E t , where E w is the embedding value of each word, E p Represents the position information of each word, E t It is used to distinguish different parts of the triple. Then the three embedding matrices are summed and merged into the output embedding matrix X, as shown in formula (7).

[0064] X=E w +E p +E t =(x 1 ,x 2 ,x 3 ,...,x L ) T (7)

[0065] Among them, x i (i=1,2,3,...,L) is the vector representation of each word, x i The dimensions are H×1, and the X dimensions are L×H.

[0066] In one embodiment, the Bert coding layer includes N Bert core layers, where N is an integer greater than 1; Figure 4 As shown, the Bert core layer includes a Bert attention layer, a Bert middleware layer and a Bert output layer. Steps: Input the output embedding matrix into the Bert encoding layer, perform semantic information fusion through a multi-head attention mechanism, and obtain a feature matrix containing contextual semantic information; including: input the output embedding matrix into the Bert attention layer in the first Bert core layer of the Bert encoding layer, obtain the multi-head attention weight after processing through the multi-head attention mechanism, and obtain the multi-head attention feature after normalizing the multi-head attention weight through a linear layer, a Dropout layer and a LayerNorm layer; input the multi-head attention feature into the Bert middleware layer, process it through a linear layer and an activation function, and input the obtained result into the Bert output layer, and obtain the first layer feature matrix after normalizing through a linear layer, a Dropout layer and a LayerNorm layer; input the first layer feature matrix into the second layer Bert core layer, obtain the second layer feature matrix, and so on, input the output of the N-1th layer Bert core layer into the Nth layer Bert core layer, and obtain the feature matrix containing contextual semantic information.

[0067] Specifically, the Bert attention layer constructs BERT's multi-head self-attention mechanism, which consists of the Bert self-attention layer and the Bert self-attention output layer. Assume that the number of heads of the multi-head self-attention mechanism in the Bert self-attention layer is n, and the input is the embedding matrix X, W Q , W K and W V are the “lookup”, “key” and “value” matrices for the Bert self-attention layer training. For the i-th (i=1,2,...,n) attention mechanism, X and W i Q , W i Q and W i V Matrix multiplication gives the corresponding Q i , K i , V i The matrix is ​​shown in equations (8) to (10).

[0068]

[0069]

[0070]

[0071] Among them, Then Wi Q , W i Q and W i V The matrix dimensions are H×d, Q i , K i and V i The matrix dimension is L×d, is the bias vector with dimension 1×d.

[0072] Then multiply the Q and K matrices, perform a scaled dot product operation, pass the softmax function and multiply with the V matrix to get the attention weight component Y of the i-th head i , as shown in formula (11), connect the n-head attention weight components to obtain the multi-head attention weight Y, as shown in formula (12).

[0073]

[0074] Y=(Y 1 ,Y 2 ,Y 3 ,...,Y n ) (12)

[0075] where Y i The dimensions are H×d, and the Y dimension is L×H.

[0076] The Bert self-attention output layer normalizes the output Y of the Bert self-attention layer through the linear layer, the Dropout layer, and the LayerNorm layer to obtain the output A of the entire Bert attention layer, as shown in Equations (13) and (14).

[0077]

[0078]

[0079] The parameter matrix W drp The dimension is H×H, and the bias vector b drp The dimension is 1×H, α drp ∈[0,1) is the Dropout rate, and the dimension of A is L×H.

[0080] The Bert middleware layer contains a linear layer and an activation function, which expands the dimension of A from L×H to L×S, as shown in formula (15).

[0081] U=gelu(AW Int +b Int ) (15)

[0082] The parameter matrix W IntThe dimension is H×S, and the bias vector b Int The dimension is 1×S, and the U dimension is L×S.

[0083] The Bert output layer is similar to the Bert self-attention output layer, which consists of a linear layer, a Dropout layer, and a LayerNorm layer. The linear layer reduces the matrix U from L×S dimension to L×H dimension, and then normalizes it to obtain the output of the Bert core layer, which is then used as the input to the Bert core layer of the next layer. After N layers of Bert core layers are superimposed, the feature matrix Z that fully integrates the contextual semantic information is obtained, as shown in Equations (16) and (17).

[0084]

[0085]

[0086] The parameter matrix W drp′ The dimension is S×H, the bias vector b drp′ The dimension is 1×H, α drp′ ∈[0,1) is the dropout rate, and the Z dimension is L×H.

[0087] In one embodiment, step 106 includes: inputting the feature matrix into an average pooling layer, adopting an average pooling strategy, and obtaining a new feature matrix; the expression of the new feature matrix is:

[0088]

[0089]

[0090] Among them, h (i,j) is the j-th hidden layer value of the feature matrix in dimension i (i=1,2,3,..,H), j=1,2,3,..,L; Dimension i is all h (i,j) The average value of is the new feature matrix, which is obtained by concatenating each dimension Got it.

[0091] Specifically, the input of the average pooling layer is the feature matrix Z output by the Bert model layer. This layer fuses the [CLS] label value and the feature values ​​of the remaining words, calculates the feature mean, and obtains a new feature matrix

[0092] 1) The idea of ​​traditional BERT to represent sequences with [CLS] tags is:

[0093] Let the hidden value h of the first position of the feature matrix Z in each dimension i (i = 1, 2, 3, .., H) be (i,0)is the [CLS] label value, and the [CLS] label values ​​of each dimension are concatenated into a sequence representation vector E′=(h (1,0) ,h (2,0) ,...,h (H,0) ).

[0094] 2) The main idea of ​​the average pooling strategy adopted in this embodiment is:

[0095] Suppose the hidden value of the feature matrix Z in dimension i (i = 1, 2, 3, .., H) is h (i,j) (j=1,2,3,..,L), calculate h (i,j) The mean Splicing each dimension Composing a new feature matrix value The calculation is shown in formula (19), the new feature matrix As shown in formula (18).

[0096] In one embodiment, the output layer includes: a linear classification layer and a sigmoid activation function; the Chinese knowledge graph completion model consists of an input layer, a Bert model layer, an average pooling layer, a BiGRU layer and an output layer; step 110 includes: inputting the sequence feature matrix into the linear classification layer, and activating the result with a sigmoid activation function to obtain a binary classification score result; the Chinese knowledge graph completion model score function is:

[0097]

[0098] Sco is the score function of the Chinese knowledge graph completion model, which is a two-dimensional vector consisting of two parts Sco 1 ,Sco 2 ∈[0,1], and Sco 1 +Sco 2 =1; W is the parameter matrix of the Chinese knowledge graph completion model; is the sequence feature matrix.

[0099] Specifically, the output layer is mainly composed of a linear classification layer and an activation function, and its input is the sequence feature matrix obtained by the BiGRU network Inspired by the KG-BERT model, the score function Sco of the Chinese knowledge graph completion model is set as shown in formula (20), and then the model parameter W is optimized by the gradient descent algorithm.

[0100] In one of the embodiments, the Chinese knowledge graph completion model consists of an input layer, a Bert model layer, an average pooling layer, a BiGRU layer and an output layer; the method also includes: taking the obtained triples as positive samples, and constructing a negative sample set by replacement method based on the positive samples; according to the positive samples, the negative sample sets, the binary classification score results and the binary cross entropy loss function, the parameters of the Chinese knowledge graph completion model are fine-tuned by the gradient descent method to obtain a trained Chinese knowledge graph completion model.

[0101] Specifically, in the fine-tuning stage, since the triplets in the dataset are all facts, these facts constitute positive samples Therefore, it is necessary to use the replacement method to construct negative samples The specific method is as follows: 1) For the true sample triple T = <h, r, t>, randomly use the entity set Another h′ replaces h, or another t′ replaces t, constructing a negative sample triple T′=<h′,r,t>∪<h,r,t′>; 2) Excluding the constructed negative sample triple T′ is still in the positive sample In the case of As shown in formula (21).

[0102]

[0103] Therefore, given a set of positive and negative samples and Calculate the binary cross entropy loss function As shown in formula (22).

[0104]

[0105] where y T ∈{0,1} is the label of triple T (negative sample or positive sample), Sco 1 ,Sco 2 ∈[0,1] are the probability scores of the triple T belonging to the two labels respectively.

[0106] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this embodiment, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0107] In a verification example, to verify the effect of the model on the Chinese dataset, this example constructs the dataset UMLS+ownthink, which is mainly composed of the Chinese UMLS dataset. Error！Reference source not found. It is composed of a sub-dataset of the ownthink Chinese dataset.

[0108] Among them, UMLS is a knowledge graph mainly based on the medical field. It stores data in the form of triples of <entity, relationship, entity>, including 135 medical field entities and 46 entity relationships, as well as more than 6,000 triple data. The entire graph has a complete structure and good tightness. The original data set is an English data set. This embodiment cuts the English data set and processes it into Chinese to construct a Chinese UMLS data set.

[0109] The ownthink dataset is a general domain knowledge graph with hundreds of millions of triples. However, since the storage format of ownthink is <entity, attribute, value>, it does not fully conform to the triple format of <entity, relationship, entity>. In addition, the original dataset is relatively sparse, the data structure is not standardized, and values ​​are missing. Therefore, this embodiment extracts some high-quality data from the huge ownthink dataset based on attributes such as "country of origin", "domain", and "software language" to form a subset of the ownthink dataset, and adds the subset to the Chinese UMLS dataset to form UMLS+ownthink.

[0110] Table 1 shows the ownthink dataset, ownthink subset, Chinese UMLS dataset, and the dataset UMLS+ownthink used in this embodiment. In addition, in order to study whether the model is effective in the public English context, this embodiment will also conduct experiments on the FB15K-237 and WN18RR datasets. The English dataset is shown in Table 2.

[0111] Table 1 Statistics of Chinese dataset

[0112] Dataset entity relation Data volume ownthink original data / / 140919785 ownthink 5584 5 5350 Chinese UMLS 135 43 2515 UMLS+ownthink 5719 48 7869

[0113] Table 2 Statistics of English dataset

[0114] Dataset entity relation Training set Validation set Test Set WN18RR 40943 11 86835 3034 3134 FB15k-237 14541 237 272115 17535 20466

[0115] (1) Baseline Model

[0116] To verify and test the effectiveness of the MpBERT-BiGRU model, the following models are used as baseline models: TransE model, TransH model, TransD model, DistMult model, ComplEx model, and KG-BERT model.

[0117] (2) Experimental setup

[0118] The experimental part of this embodiment is implemented using the Pytorch framework. The experimental environment is a GeForce RTX3090 GPU with 24G video memory. The Bert model layer of the model selects the bert-base-chinese model. The total number of parameters is 110M, the hidden layer size is 768, the number of layers of the Bert core layer is 12, and the number of multi-head attention mechanism heads is 12.

[0119] After experimental adjustment, the hyperparameters of the model are set as follows: maximum input sequence length is 30, training batch is 5, batch size is 256, activation function is gelu, and learning rate of Adam optimizer is 5×10 -5 , the dropout rate is 0.1.

[0120] (3) Experimental results

[0121] 1) Model comparison experiment

[0122] The link prediction experimental results on the Chinese dataset UMLS+ownthink are shown in Table 3. This example uses the open source knowledge graph embedding toolkit OpenKE to embed the TransE model, TransH model, and TransD model. Error！Reference source not found. , DistMult model, and ComplEx model for link prediction experiments. Based on the original model, the KG-BERT model changes the English BERT model bert-base-cased to the bert-base-chinese model for link prediction experiments.

[0123] Table 3 Experimental results of Chinese dataset

[0124] Model\Indicator Average position Top 10 hit rate Top 3 hit rate Top 1 hit rate TransE 863.29 48.19 32.41 0.93 TransH 847.48 48.73 33.84 1.35 TransD 954.18 47.1 31.35 1.31 DistMult 912.92 36.03 33.88 18.19 ComplEx 964.94 35.27 29.28 17.64 KG-BERT 49.32 56.97 45.53 36.94 MpBERT-BiGRU 38.93 61.6 50.3 41.14

[0125] Among them: MeanRank (MR) refers to the average ranking of target triples. The smaller this indicator is, the better the model performance is. Top-k hit rate (Hit@k) refers to the ratio of target triples ranked in the top k. The larger this indicator is, the better the model performance is.

[0126] In addition, in order to verify the feasibility of the model in this embodiment on English datasets and public datasets, this embodiment compares the MR indicators of different models on two public English datasets FB15K-237 and WN18RR. The link prediction results are as follows: Figure 5 As shown, in order to observe the experimental results clearly and intuitively, the vertical axis data is set to That is, the higher the vertical axis index, the better the model performance.

[0127] Experimental results show that the MpBERT-BiGRU model achieves the best indicator effect on the UMLS+ownthink Chinese dataset and two English datasets FB15K-237 and WN18RR.

[0128] 2) Model ablation experiment

[0129] In order to verify the effectiveness of the proposed average pooling strategy and BiGRU network, this embodiment compares the experimental results of link prediction using the average pooling strategy, the maximum pooling strategy, and the minimum pooling strategy, as well as the experimental results of whether BiGRU is used, as shown in Tables 4 and 5.

[0130] Table 4 Comparative experimental results of BERT pooling strategy

[0131] Model\Indicator MR Hit@10 Hit@3 Hit@1 BERT 49.32 56.97 45.53 36.94 +Max-pooling 105.17 55.57 45.96 36 +Min-pooling 75.51 51.01 42.79 35.15 +Mean-pooling 49 55.95 47 39.79

[0132] Table 5 Comparative experimental results of BERT-BiGRU pooling strategy

[0133] Model\Indicator MR Hit@10 Hit@3 Hit@1 BERT-BiGRU 41.61 60.84 47.43 37.76 +Max-pooling 40.88 63.67 48.65 40.86 +Min-pooling 39.93 60.5 45.02 38.23 +Mean-pooling 38.93 61.6 50.3 41.14

[0134] The experimental results show that the average pooling strategy improves the MR, Hit@3, and Hit@1 indicators regardless of whether there is a BiGRU module.

[0135] 3) Experiments with different neural network modules

[0136] In order to further verify the effectiveness of the BiGRU network, this embodiment performs a link prediction comparison experiment on BERT superimposed with different neural network models (BERT-RNN, BERT-BiLSTM, BERT-BiGRU), and abandons the average pooling strategy to eliminate its influence. The link prediction experimental results are shown in Table 6. At the same time, in order to verify that GRU has higher operating efficiency, this embodiment sets different training batches and compares the experimental duration of the BERT-BiGRU model and the BERT-BiLSTM model. Figure 6 shown.

[0137] Table 6 Experimental results of different neural network modules

[0138] Model\Indicator MR Hit@10 Hit@3 Hit@1 BERT 49.32 56.97 45.53 36.94 BERT+RNN 47.93 54.3 42.15 33.97 BERT+BiLSTM 54.36 57.09 46.37 39.83 BERT+BiGRU 41.61 60.84 47.43 37.76

[0139] Experiments show that the BiGRU network achieves good results in terms of experimental results and running time.

[0140] In one embodiment, Figure 7 As shown, a Chinese knowledge graph completion device is provided, comprising: a knowledge graph acquisition module to be processed and a Chinese knowledge graph completion module, wherein:

[0141] The module for acquiring the knowledge graph to be processed is used to acquire the knowledge graph to be processed. The knowledge graph to be processed is stored as a triple consisting of a head entity, a tail entity, and the relationship between the head entity and the tail entity.

[0142] The Chinese knowledge graph completion module is used to input the triples into the input layer, represent the head entity, tail entity and relationship as three paragraphs of text, and process the three paragraphs of text to obtain the input sequence; input the input sequence into the Bert model layer, encode the input sequence, and then calculate it through the multi-head self-attention mechanism to obtain a feature matrix containing contextual semantic information; input the feature matrix into the average pooling layer to obtain a new feature matrix; input the new feature matrix into the BiGRU layer, and through the serialized neural network processing, learn the positional relationship information between words to obtain a sequence feature matrix; input the sequence feature matrix into the output layer to obtain the binary classification score result.

[0143] In one of the embodiments, the input sequence in the Chinese knowledge graph completion module consists of a text sequence representing a head entity, a relationship, and a tail entity; the text sequence of the head entity and the tail entity are descriptions of the head entity and the tail entity, respectively; and the text sequence of the relationship is the name of the relationship.

[0144] Each input sequence starts with a special classification label and consists of three paragraphs of text, each separated by another special label; the input sequence expression is shown in formula (6).

[0145] In one embodiment, the Bert model layer includes a Bert embedding layer and a Bert encoding layer; the Chinese knowledge graph completion module is also used to input the input sequence into the Bert embedding layer, convert the input sequence into three embedding matrices, and then sum the three embedding matrices to obtain an output embedding matrix; the three embedding matrices are a word embedding matrix representing the embedding value of each character, a position embedding matrix representing the position information of each character, and a type embedding matrix used to distinguish different parts of a triple; the output embedding matrix is ​​input into the Bert encoding layer, and semantic information is fused through a multi-head attention mechanism to obtain a feature matrix containing contextual semantic information.

[0146] In one embodiment, the Bert encoding layer includes N Bert core layers, where N is an integer greater than 1; the Bert core layer includes a Bert attention layer, a Bert middleware layer, and a Bert output layer. The Chinese knowledge graph completion module is also used to input the output embedding matrix into the Bert attention layer in the first Bert core layer of the Bert encoding layer, obtain the multi-head attention weight after processing by the multi-head attention mechanism, and obtain the multi-head attention feature after normalizing the multi-head attention weight through the linear layer, the Dropout layer, and the LayerNorm layer; input the multi-head attention feature into the Bert middleware layer, process it through the linear layer and the activation function, and input the obtained result into the Bert output layer, and obtain the first layer feature matrix after normalizing it through the linear layer, the Dropout layer, and the LayerNorm layer; input the first layer feature matrix into the second layer Bert core layer to obtain the second layer feature matrix, and so on, input the output of the N-1th layer Bert core layer into the Nth layer Bert core layer to obtain a feature matrix containing contextual semantic information.

[0147] In one of the embodiments, the Chinese knowledge graph completion module is also used to input the feature matrix into the average pooling layer, and adopt the average pooling strategy to obtain a new feature matrix; the expression of the new feature matrix is ​​shown in equations (18) and (19).

[0148] In one of the embodiments, the output layer includes: a linear classification layer and a sigmoid activation function; the Chinese knowledge graph completion model consists of an input layer, a Bert model layer, an average pooling layer, a BiGRU layer and an output layer; the Chinese knowledge graph completion module is also used to input the sequence feature matrix into the linear classification layer, and activate the result using the sigmoid activation function to obtain a binary classification score result; the Chinese knowledge graph completion model score function is as shown in formula (20).

[0149] In one of the embodiments, the Chinese knowledge graph completion model consists of an input layer, a Bert model layer, an average pooling layer, a BiGRU layer and an output layer; the device also includes a fine-tuning module, which is used to take the acquired triples as positive samples and construct a negative sample set by replacement method based on the positive samples; according to the positive samples, the negative sample sets, the binary classification score results and the binary cross entropy loss function, the parameters of the Chinese knowledge graph completion model are fine-tuned by the gradient descent method to obtain a trained Chinese knowledge graph completion model.

[0150] For the specific limitations of the Chinese knowledge graph completion device, please refer to the limitations of the Chinese knowledge graph completion method above, which will not be repeated here. Each module in the above-mentioned Chinese knowledge graph completion device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0151] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a Chinese knowledge graph completion method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0152] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0153] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0154] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0155] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0156] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A Chinese knowledge graph completion method, characterized in that: The method comprises: Obtain a knowledge graph to be processed, wherein the knowledge graph to be processed is stored as a triple consisting of a head entity, a tail entity, and a relationship between the head entity and the tail entity; Input the triples into the input layer, represent the head entity, the tail entity and the relationship as three paragraphs of text, and process the three paragraphs of text to obtain an input sequence; The input sequence is input into the Bert model layer, the input sequence is encoded, and then calculated by a multi-head self-attention mechanism to obtain a feature matrix containing contextual semantic information; the Bert model layer includes a Bert embedding layer and a Bert encoding layer; the Bert encoding layer includes N Bert core layers, where N is an integer greater than 1; the Bert core layer includes a Bert attention layer, a Bert middleware layer and a Bert output layer; Input the feature matrix into the average pooling layer to obtain a new feature matrix; The new feature matrix is ​​input into the BiGRU layer, and is processed by a serialized neural network to learn the position relationship information between words to obtain a sequence feature matrix; Input the sequence feature matrix into the output layer to obtain a binary classification score result; The input sequence is input into the Bert model layer, the input sequence is encoded, and then calculated by a multi-head self-attention mechanism to obtain a feature matrix containing contextual semantic information, including: Input the input sequence into the Bert embedding layer, convert the input sequence into three embedding matrices, and then sum the three embedding matrices to obtain an output embedding matrix; the three embedding matrices are respectively a word embedding matrix representing the embedding value of each word, a position embedding matrix representing the position information of each word, and a type embedding matrix for distinguishing different parts of a triple; The output embedding matrix is ​​input into the Bert encoding layer, and semantic information is fused through a multi-head attention mechanism to obtain a feature matrix containing contextual semantic information; The output embedding matrix is ​​input into the Bert encoding layer, and semantic information is fused through a multi-head attention mechanism to obtain a feature matrix containing contextual semantic information; including: The output embedding matrix is ​​input into the Bert attention layer in the first Bert core layer of the Bert encoding layer, and the multi-head attention weight is obtained after being processed by the multi-head attention mechanism, and the multi-head attention weight is normalized by the linear layer, the Dropout layer and the LayerNorm layer to obtain the multi-head attention feature; The multi-head attention features are input into the Bert middleware layer, processed by the linear layer and the activation function, and the obtained results are input into the Bert output layer, and normalized by the linear layer, the Dropout layer and the LayerNorm layer to obtain the first layer feature matrix; The first-layer feature matrix is ​​input into the second-layer Bert core layer to obtain the second-layer feature matrix. Similarly, the output of the N-1th Bert core layer is input into the Nth Bert core layer to obtain a feature matrix containing contextual semantic information.

2. The method according to claim 1, characterized in that Input the triple into the input layer, represent the head entity, the tail entity and the relationship as three paragraphs of text, and process the three paragraphs of text to obtain an input sequence, wherein the input sequence is composed of a text sequence representing the head entity, the relationship and the tail entity; the text sequence of the head entity and the text sequence of the tail entity are the descriptions of the head entity and the tail entity respectively; the text sequence of the relationship is the name of the relationship Each of the input sequences starts with a special classification tag and consists of three paragraphs of text, each of which is separated by another special tag; the input sequence expression is: in: is the input sequence, is the head entity, is the relationship between the head entity and the tail entity, is the tail entity, and are the text sequences of the head entity and the tail entity respectively, is the text sequence of the relation, is the classification label, For special tags.

3. The method according to claim 1, characterized in that The feature matrix is ​​input into the average pooling layer to obtain a new feature matrix, including: The feature matrix is ​​input into the average pooling layer, and the average pooling strategy is adopted to obtain a new feature matrix; the expression of the new feature matrix is: in, is the feature matrix in dimension i ( ) j Hidden layer value, ; Dimensions i For all The average value of is the new feature matrix, which is obtained by concatenating each dimension Got it.

4. The method according to claim 1, characterized in that: The output layer consists of: Linear classification layer and Activation function; the Chinese knowledge graph completion model consists of the input layer, Model layer, average pooling layer, Layer and output layer; The sequence feature matrix is ​​input into the output layer, and the binary classification score results obtained include: The sequence feature matrix is ​​input into the linear classification layer, and the result is obtained using After the activation function is activated, the binary classification score result is obtained; the score function of the Chinese knowledge graph completion model is: in is the score function of the Chinese knowledge graph completion model, which is a two-dimensional vector consisting of two parts Composition, and ; W Complete the model parameter matrix for the Chinese knowledge graph; is the sequence feature matrix.

5. The method according to claim 1, characterized in that The Chinese knowledge graph completion model consists of the input layer, Bert model layer, average pooling layer, BiGRU layer and output layer; The method further comprises: The obtained triples are used as positive samples, and a negative sample set is constructed based on the positive samples by using a replacement method; According to the positive samples, negative sample sets, binary classification score results and binary cross entropy loss function, the gradient descent method is used to fine-tune the parameters of the Chinese knowledge graph completion model to obtain a trained Chinese knowledge graph completion model.

6. A Chinese knowledge graph completion device, characterized in that: The device comprises: A module for acquiring a knowledge graph to be processed, used for acquiring a knowledge graph to be processed, wherein the knowledge graph to be processed is stored as a triple consisting of a head entity, a tail entity, and a relationship between the head entity and the tail entity; The Chinese knowledge graph completion module is used to input the triples into the input layer, represent the head entity, the tail entity and the relationship as three paragraphs of text, and process the three paragraphs of text to obtain an input sequence; input the input sequence into the Bert model layer, encode the input sequence, and then calculate it through a multi-head self-attention mechanism to obtain a feature matrix containing contextual semantic information; input the feature matrix into the average pooling layer to obtain a new feature matrix; input the new feature matrix into the BiGRU layer, and process it through a serialized neural network to learn the position relationship information between words to obtain a sequence feature matrix; input the sequence feature matrix into the output layer to obtain a binary classification score result; the Bert model layer includes a Bert embedding layer and a Bert encoding layer; the Bert encoding layer includes N Bert core layers, where N is an integer greater than 1; the Bert core layer includes a Bert attention layer, a Bert middleware layer and a Bert output layer; Among them, the Chinese knowledge graph completion module is also used to input the input sequence into the Bert embedding layer, convert the input sequence into three embedding matrices, and then sum the three embedding matrices to obtain an output embedding matrix; the three embedding matrices are respectively a word embedding matrix representing the embedding value of each word, a position embedding matrix representing the position information of each word, and a type embedding matrix for distinguishing different parts of a triple; the output embedding matrix is ​​input into the Bert encoding layer, and semantic information is fused through a multi-head attention mechanism to obtain a feature matrix containing contextual semantic information; The Chinese knowledge graph completion module is also used to input the output embedding matrix into the Bert attention layer in the first Bert core layer of the Bert encoding layer, obtain the multi-head attention weight after processing by the multi-head attention mechanism, and obtain the multi-head attention feature after normalizing the multi-head attention weight through the linear layer, the Dropout layer and the LayerNorm layer; input the multi-head attention feature into the Bert middleware layer, process it through the linear layer and the activation function, and input the obtained result into the Bert output layer, obtain the first layer feature matrix after normalizing through the linear layer, the Dropout layer and the LayerNorm layer; input the first layer feature matrix into the second layer Bert core layer to obtain the second layer feature matrix, and so on, input the output of the N-1th layer Bert core layer into the Nth layer Bert core layer to obtain a feature matrix containing contextual semantic information.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images