A knowledge graph construction method based on BTBC model

By combining the BERT model and the improved LSTM-CRF layer, the knowledge graph construction is constructed using the BTBC model, which solves the problems of low efficiency and entity overlap in the existing technology, and achieves more efficient knowledge graph construction and accuracy.

CN114077673BActive Publication Date: 2024-11-08NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110702281.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-21
Publication Date
2024-11-08
Estimated Expiration
2041-06-21

AI Technical Summary

Technical Problem

The existing knowledge graph construction methods are inefficient, costly, and difficult to effectively deal with entity overlap.

Method used

Using a BTBC model-based method, combined with the BERT model and the improved LSTM-CRF layer, entity recognition and relationship extraction layer are used to identify and relationship through word embedding layer, context encoding layer, nested named entity recognition and relationship extraction layer. Extract, and finally store triplets in the Neo4j graph database.

Benefits of technology

It improves the accuracy and efficiency of knowledge graph construction, effectively solves the problem of entity overlap, and reduces the dependence on labeled samples.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114077673B_ABST
    Figure CN114077673B_ABST
Patent Text Reader

Abstract

The present invention is a knowledge graph construction method based on the BTBC model. The context encoding layer adopts the Tiered-BiLSTM-CRF model, and the nested entities are identified by dynamically stacking the plane NER layer. Each plane NER layer is a plane NER model of BiLSTM-CRF. Each layer fuses the entity output of the LSTM layer and outputs it to the next plane NER layer. The relationship extraction layer adopts a multi-head relationship extraction model, and finally stores it in the Neo4j graph database in the format of triples. The present invention uses the BERT pre-trained model with stronger text feature representation ability as the feature representation layer, combined with Tiered-BiLSTM-CRF, to highlight the important features in the context semantic features, and effectively solves the problem of entity overlap on the basis of identifying multiple pairs of relationships, thereby improving the accuracy of graph construction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and natural language processing, and specifically relates to a knowledge graph construction method based on the BTBC model. Background Art

[0002] The construction of knowledge graph refers to the process of extracting and storing information from unstructured text using natural language processing and deep learning techniques. At present, the research on knowledge graph construction involves multiple fields, including natural language processing, machine learning, and information extraction, and has attracted the attention of many researchers. Among the domain knowledge graphs, medical knowledge graphs account for a large proportion. For example, the Linked Life Data project includes 25 public biomedical databases and can access 10 billion RDF statements, including genes, drugs, clinical trials, and side effects. The construction of medical knowledge graphs can be used to assist doctors in diagnosing diseases, reduce the time of consulting materials, improve work efficiency, reduce treatment costs, and explore similar connections between drugs; for users, the application of knowledge graphs can make medical questions and answers more intelligent and professional, and at the same time facilitate users to query diseases or symptoms in search engines.

[0003] The construction of a knowledge graph is essentially the process of building basic data into a set of triples using a variety of methods and techniques, involving multiple technologies such as data acquisition, entity relationship extraction, knowledge representation, knowledge fusion, knowledge storage and knowledge reasoning.

[0004] Context encoders often use convolutional neural networks (CNN) and recurrent neural networks (RNN). Among them, CNN can extract local features of input data, so it is mainly used for feature learning; RNN considers the influence between characters in a sentence and can be used for feature learning and sequence labeling at the same time. The long short-term memory network (LSTM) in RNN has achieved remarkable results in sequence data modeling, especially the bidirectional LSTM (BiLSTM) can process a sentence from two directions. However, there is no word embedding layer, and the random initialization effect is unstable.

[0005] The biggest innovation of BERT is to use Transformer to introduce context information. This model uses the structure of transformer encoder. In order to prevent label leakage, Masked LM pre-training is proposed to introduce context information to participate in training, but the position information is weakened in the calculation process.

[0006] CN112347265A discloses a method for constructing a knowledge graph, which performs word segmentation on a sentence to be processed to obtain multiple individual words; identifies entities in the multiple individual words to obtain entity pairs in groups of two; obtains a sentence vector of the sentence for each entity pair; extracts representation features of the sentence based on the sentence vector; performs feature screening on the representation features to shield noise features therein; predicts entity relationships of the entity pairs based on the representation features; and constructs a knowledge graph based on the entity pairs and corresponding entity relationships. Although this method can filter out noise features, the construction of this knowledge graph requires a lot of manpower and time investment, and the efficiency of constructing the knowledge graph is low and high. Summary of the invention

[0007] In order to solve the above problems, the present invention provides a knowledge graph construction method based on the BTBC model, which improves the existing BERT model, uses LSTM to learn the dependencies on the observation sequence, and finally uses CRF to learn the relationship between the state sequence and obtain the answer.

[0008] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0009] The present invention is a knowledge graph construction method based on the BTBC model, wherein the BTBC model includes a word embedding layer, a context encoding layer, a nested named entity recognition layer, a relationship extraction layer and a knowledge storage layer, and the knowledge graph construction method includes the following steps:

[0010] Step 1: Use the BERT pre-trained language model to convert each word in the sentence into a low-dimensional word vector.

[0011] The pre-training in step 1 is as follows: in the face of a large amount of unlabeled data crawled, use the entry names of Baidu Encyclopedia and the entities in the triples formed by semi-structured data, combine and remove duplicate values ​​to back-label the unstructured data, thereby automatically generating a large amount of training corpus. Because the BERT model has a maximum length requirement for the input sentences, long sentences also need to be truncated according to periods.

[0012] The BERT model in step 1 is a Transformer bidirectional encoder that encodes the input training corpus into word vectors and then inputs the sub-vector sequence into the stacked NER layer, which is a nested named entity recognition layer.

[0013] Step 2: Input the word vector sequence obtained in step 1 into the improved Tiered-BiLSTM layer to encode the sentence and fuse the previous and following information into the context encoding layer;

[0014] The step 2 is specifically as follows:

[0015] Step 2-1: At time t, given input x t The specific calculation process of the hidden layer output representation of Tiered-BiLSTM is obtained as shown in formula (1)(2)(3)(4)(5):

[0016] i t =σ(W xi x t +W hi h t-1 +W ci C t-1 +b i ) (1)

[0017] f t =σ(W xf x t +W hf h t-1 +W cf C t-1 +b f ) (2)

[0018] C t =f t C t-1 +i t tanh(W x C xt +W h Ch t-1 +b C ) (3)

[0019] o t =σ(W xo x t +W ho h t-1 +W co C t +b o ) (4)

[0020] h t =o t tanh(C t ) (5)

[0021] Among them, W is the weight matrix between the two layers, W xi is the weight matrix from the input layer to the inputs, b is the bias vector, b i is the bias vector of the input gate, C is the memory cell state, σ and tanh are two activation functions, i t is the input gate, o t is the output gate, f t is the forget gate, h tis the output vector; this storage unit can effectively filter and memorize the information of the unit, can effectively utilize long-distance information, and solves the gradient diffusion problem that is easy to occur in the RNN model.

[0022] Step 2-2: For a given input sequence (x1, x2, ..., x n ), each word is represented as a d-dimensional vector, and the forward Tiered-BiLSTM calculates the context representation of each word t from left to right in the sentence Correspondingly, the backward Tiered-BiLSTM is used to read the same sequence in reverse to obtain the context representation from right to left. By concatenating the left and right context representations of a word, the word representation using this model is obtained as:

[0023]

[0024] Step 2-3: For a given sequence x = (x1, x2, ..., x n ) and the corresponding label sequence y = (y1, y2, ..., y n ), after passing through the Tiered-BiLSTM layer, the score of each label is obtained through linear mapping:

[0025] P i =W s h t +b s (6)

[0026] where h t is the output of the Tiered-BiLSTM layer at time t, W s and b s is the linear mapping parameter.

[0027] Step 3: Fuse the entity context representation detected in step 2, use the bit-by-bit addition method to fuse the internal entities except the tail character, then perform vector concatenation with the tail character, and finally use linear mapping to convert it into the original dimension;

[0028] In step 3, the specific method of finally converting to the original dimension by linear mapping is as follows:

[0029]

[0030] where z start Indicates the representation of the first word in the entity recognized by the current plane NER, z end is the last word representation in the entity, W is the linear mapping parameter, m iIt is a fused representation of the entity. The fusion starts from the starting point of the entity position and ends at the end point of the entity position. If the area is detected as an entity, this fused table representation allows the detected entity to be processed as a single character; if the area is detected as a non-entity, the representation is passed to the next layer without any processing.

[0031] Step 4: Concatenate the label sequence obtained in step 2 and the predicted label to obtain the label transfer score. The conditional random field (CRF) obtains the globally optimized label sequence by considering the adjacent relationship between labels.

[0032] The specific steps in step 4 are:

[0033] Step 4-1: CRF defines the label transfer score from the input sequence to the label sequence:

[0034]

[0035] Where W is the transformation matrix, represents the label transfer score, Indicates that the character is i In order to solve the problem of label bias, CRF needs to be globally normalized. Specifically, the probability that the label sequence corresponding to input x is y is defined as:

[0036]

[0037] Step 4-2: During the training process, we hope to maximize the logarithmic probability of the correct label sequence, that is:

[0038]

[0039] Among them, λ and θ are regularization parameters.

[0040] Step 5: Concatenate the original dimension of step 3 and the globally optimized label sequence of step 4, calculate the most likely corresponding relationship and head vector of each word through the multi-head relationship extraction model, and obtain multiple relationships through the sigmoid loss function;

[0041] The step 5 is specifically as follows:

[0042] Step 5-1: Calculate the most likely corresponding relationship and head vector of each word to get rel_scores using the following formula

[0043] s (r) (z j , z i , z k )=V (r) f(U (r) z j +W(r) z i +b (r) ) (11)

[0044] The superscript represents relation extraction, f() is the activation function, relu activation function, tanh activation function;

[0045] Step 5-2: Process the score obtained in step 5-1 through the sigmoid function to get token w i With the relationship k The probability of minimizing the loss function of relation extraction during training is used to optimize the parameters of the model:

[0046]

[0047] in Yes i The ground truth head image quantity and the corresponding relationship label; m is the i The number of relations. By default, only one relation is selected for a head; θ is a parameter set. In prediction, as long as the calculated probability value is greater than the lowest probability obtained by all true label values, it is a relation pair.

[0048] Step 6: Store the output of step 5 in the format of triples in the Neo4j graph database. Specifically: use the Neo4j graph database, the basic elements are nodes, edges and attributes, nodes are used to store entities and attributes, edges are used to store the relationship between entities, use the topological structure to store, and display the data in a visual form.

[0049] The beneficial effects of the present invention are as follows: the knowledge graph construction method of the BTBC model of the present invention combines the BERT model with the improved BILSTM-CRF model. The BERT model overcomes the disadvantage that the BiLSTM-CRF requires a large number of labeled samples. The improved LSTM-CRF layer strengthens the recognition of nested entities by strengthening the weight of the tail characters of the entity, highlights the important features in the contextual semantic features, and effectively solves the problem of entity overlap on the basis of identifying multiple pairs of relationships, thereby improving the accuracy of graph construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is the knowledge graph construction method process of the present invention. DETAILED DESCRIPTION

[0051] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is, in some embodiments of the present invention, these practical details are not necessary. In addition, for the purpose of simplifying the drawings, some conventional structures and components will be depicted in a simple schematic manner in the drawings.

[0052] like Figure 1 As shown, the present invention is a knowledge graph construction method based on the BTBC model, characterized in that: the BTBC model includes a word embedding layer, a context encoding layer, a nested named entity recognition, a relationship extraction layer and a knowledge storage layer. Specifically, the knowledge graph construction method includes the following steps:

[0053] Step 1: Use the BERT pre-trained language model to convert each word in the sentence into a low-dimensional word vector.

[0054] Faced with a large amount of crawled unlabeled data, we use the entry names of Baidu Encyclopedia and the entities in the triples formed by semi-structured data to combine and remove duplicate values ​​to back-label the unstructured data, thereby automatically generating a large amount of training corpus. Because the BERT model has a maximum length requirement for the input sentences, long sentences also need to be truncated according to periods.

[0055] Step 2: Input the word vector sequence obtained in step 1 into the improved Tiered-BiLSTM layer, encode the sentence, and fuse the previous and following information into the context encoding layer.

[0056] Specifically:

[0057] Step 2-1: At time t, given input x t The specific calculation process of the hidden layer output representation of Tiered-BiLSTM is obtained as shown in formula (1)(2)(3)(4)(5):

[0058] i t =σ(W xi x t +W hi h t-1 +W ci C t-1 +b i ) (1)

[0059] f t =σ(W xf x t +W hf h t-1 +W cf C t-1 +b f) (2)

[0060] C t =f t C t-1 +i t tanh(W x Cx t +W hC h t-1 +b C ) (3)

[0061] o t =σ(W xo x t +W ho h t-1 +W co C t +b o ) (4)

[0062] h t =o t tanh(C t ) (5)

[0063] Among them, W is the weight matrix between the two layers, W xi is the weight matrix from the input layer to the inputs, b is the bias vector, b i is the bias vector of the input gate, C is the memory cell state, σ and tanh are two activation functions, i t is the input gate, o t is the output gate, f t is the forget gate, h t is the output vector; this storage unit can effectively filter and memorize the information of the unit, can effectively utilize long-distance information, and solves the gradient diffusion problem that is easy to occur in the RNN model.

[0064] Step 2-2: For a given input sequence (x1, x2, ..., x n ), each word is represented as a d-dimensional vector, and the forward Tiered-BiLSTM calculates the context representation of each word t from left to right in the sentence Correspondingly, the backward Tiered-BiLSTM is used to read the same sequence in reverse to obtain the context representation from right to left. By concatenating the left and right context representations of a word, the word representation using this model is obtained as:

[0065]

[0066] Step 2-3: For a given sequence x = (x1, x2, ..., x n ) and the corresponding label sequence y = (y1, y2, ..., yn ), after passing through the Tiered-BiLSTM layer, the score of each label is obtained through linear mapping:

[0067] P i =W s h t +b s (6)

[0068] where h t is the output of the Tiered-BiLSTM layer at time t, W s and b s is the linear mapping parameter.

[0069] Step 3: Fuse the entity context representation detected in step 2, use the bit-by-bit addition method to fuse the internal entities except the tail character, then concatenate the vector with the tail character, and finally use linear mapping to convert it into the original dimension. The specific method is:

[0070]

[0071] where z start Indicates the representation of the first word in the entity recognized by the current plane NER, z end is the last word representation in the entity, W is the linear mapping parameter, m i It is a fused representation of the entity. The fusion starts from the starting point of the entity position and ends at the end point of the entity position. If the area is detected as an entity, this fused table representation allows the detected entity to be processed as a single character; if the area is detected as a non-entity, the representation is passed to the next layer without any processing.

[0072] Step 4: Concatenate the label sequence obtained in step 2 and the predicted label to obtain the label transfer score. The conditional random field (CRF) obtains the globally optimized label sequence by considering the adjacent relationship between labels.

[0073] Calculate the label transfer score from the input sequence to the label sequence, and substitute the output sequence obtained in step 3 into formula (8):

[0074]

[0075] Where W is the transformation matrix, represents the label transfer score, Indicates that the character is i In order to solve the problem of label bias, CRF needs to be globally normalized. Specifically, the probability that the label sequence corresponding to input x is y is defined as:

[0076]

[0077] Step 4-2: During the training process, we hope to maximize the logarithmic probability of the correct label sequence, that is:

[0078]

[0079] Among them, λ and θ are regularization parameters.

[0080] Step 5: Concatenate the original dimension of step 3 and the globally optimized label sequence of step 4, calculate the most likely corresponding relationship and head vector of each word through the multi-head relationship extraction model, and obtain multiple relationships through the sigmoid loss function.

[0081] Step 5-1: Concatenate the outputs of step 3 and step 4 as the input of the relation extraction task in step 4, and calculate the most likely corresponding relation and head vector of each word to get rel_scores using the following formula:

[0082] s (r) (z j , z i , z k )=V (r) f(U (r) z j +W (r) z i +b (r) ) (11)

[0083] The superscript represents relation extraction, f() is the activation function, relu activation function, tanh activation function;

[0084] The above scores are processed by sigmoid function to get token w i With the relationship k The probability of minimizing the loss function of relation extraction during training is used to optimize the parameters of the model:

[0085]

[0086] in Yes i The ground truth head image quantity and the corresponding relationship label; m is the i The number of relations in . By default, only one relation is selected for a head; θ is a parameter set.

[0087] In the prediction, as long as the calculated probability value is greater than the lowest probability obtained by all the real label values, it is a relationship pair. This can not only determine whether there is a relationship, but also identify whether there are multiple pairs of relationships in the sentence, effectively solving the problem of entity overlap.

[0088] Step 6: Store the output of step 5 in the format of triples in the Neo4j graph database, use the topological structure to store, and display the data in a visual form.

[0089] The word embedding layer encodes the medical information dataset into word vectors, the context encoding layer adopts the Tiered-BiLSTM-CRF model, and the nested entities are identified by dynamically stacking the plane NER layers. Each plane NER layer is a plane NER model of BiLSTM-CRF. Each layer fuses the entity output of the LSTM layer and outputs it to the next plane NER layer. The relation extraction layer adopts a multi-head relation extraction model, and finally stores it in the Neo4j graph database in the format of triples. The BTBC model obtained by the present invention combines BERT and BiLSTM-CRF, effectively solves the problem of entity overlap, and improves the accuracy of graph construction.

[0090] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A knowledge graph construction method based on BTBC model, Features: The BTBC model includes a word embedding layer, a context encoding layer, a nested named entity recognition layer, a relationship extraction layer and a knowledge storage layer. The knowledge graph construction method includes the following steps: Step 1: Use the BERT pre-trained language model to convert each word in the sentence into a low-dimensional word vector. Step 2: Input the word vector sequence obtained in step 1 into the Tiered-BiLSTM layer to encode the sentence and fuse the previous and following information into the context encoding layer; Step 3: Fuse the entity context representation detected in step 2, use the bit-by-bit addition method to fuse the internal entities except the tail character, then perform vector concatenation with the tail character, and finally use linear mapping to convert it into the original dimension; Step 4: Concatenate the label sequence obtained in step 2 and the predicted label to obtain the label transfer score. The conditional random field (CRF) obtains the globally optimized label sequence by considering the adjacent relationship between labels. Step 5: Concatenate the original dimension of step 3 and the globally optimized label sequence of step 4, calculate the most likely corresponding relationship and head vector of each word through the multi-head relationship extraction model, and obtain multiple relationships through the sigmoid loss function; Step 6: Store the output of step 5 in the Neo4j graph database in the format of triples.

2. According to the method for constructing a knowledge graph based on the BTBC model according to claim 1, Features: The step 5 is specifically as follows: Step 5-1: Calculate the most likely corresponding relationship and head vector of each word to get rel_scores using the following formula s (r) (z j ,z i ,z k )=V (r) f(U (r) z j +W (r) z i +b (r) ) (11) The superscript represents relation extraction, f() is the activation function, relu activation function, tanh activation function; Step 5-2: Process the score obtained in step 5-1 through the sigmoid function to get token w i With relation k The probability of minimizing the loss function of relation extraction during training is used to optimize the parameters of the model: in Yes i The ground truth head image quantity and the corresponding relationship label; m is the i The number of relationships. By default, only one relationship is selected for a head. θ is a set of parameters. In prediction, as long as the calculated probability value is greater than the lowest probability obtained by all true label values, it is a relationship pair.

3. According to the method for constructing a knowledge graph based on the BTBC model according to claim 1, Features: In step 3, the specific method of finally converting to the original dimension by linear mapping is as follows: where z start Indicates the representation of the first word in the entity recognized by the current plane NER, z end is the last word representation in the entity, W is the linear mapping parameter, m i It is a fused representation of the entity. The fusion starts from the starting point of the entity location and ends at the end point of the entity location. If the area formed by the starting point of the entity location to the end point of the entity location is detected as an entity, this fusion table representation allows the detected entity to be processed as a single character; if the area formed by the starting point of the entity location to the end point of the entity location is detected as a non-entity, the representation is passed to the next layer without any processing.

4. According to the method for constructing a knowledge graph based on the BTBC model according to claim 1, Features: The step 2 is specifically as follows: Step 2-1: At time t, given input x t The specific calculation process of the hidden layer output representation of Tiered-BiLSTM is obtained as shown in formula (1)(2)(3)(4)(5): i t =σ(W xi x t ++W hi h t-1 +W ci C t-1 +b i ) (1) f t =σ(W xf x t +W hf h t-1 +W cf C t-1 +b f ) (2) C t =f t C t-1 +i t tanh(W x C xt +W hC h t-1 +b C ) (3) o t =σ(W xo x t +W ho h t-1 +W co C t +b o ) (4) h t =o t brother(C) t ) (5) Among them, W is the weight matrix between the two layers, W xi is the weight matrix from the input layer to the inputs, b is the bias vector, b i is the bias vector of the input gate, C is the memory cell state, σ and tanh are two activation functions, i t is the input gate, o t is the output gate, f t is the forget gate, h t is the output vector; Step 2-2: For a given input sequence (x 1 , x 2 , ..., x n ), each word is represented as a d-dimensional vector, and the forward Tiered-BiLSTM calculates the context representation of each word t from left to right in the sentence Correspondingly, the backward Tiered-BiLSTM is used to read the same sequence in reverse to obtain the context representation from right to left. By concatenating the left and right context representations of a word, the word representation using this model is obtained as: Step 2-3: For a given sequence x=(x 1 , x 2 , ..., x n ) and the corresponding label sequence y=(y 1 ,y 2 , ..., y n ), after passing through the Tiered-BiLSTM layer, the score of each label is obtained through linear mapping: P i =W S h t +b S (6) where h t is the output of the Tiered-BiLSTM layer at time t, W S and b S is the linear mapping parameter.

5. According to the method for constructing a knowledge graph based on the BTBC model according to claim 1, Features: The specific steps in step 4 are: Step 4-1: CRF defines the label transfer score from the input sequence to the label sequence: Where W is the transformation matrix, represents the label transfer score, Indicates that the character is i The score of the label, the probability that the label sequence corresponding to the input x is y is defined as: Step 4-2: During the training process, maximize the logarithmic probability of the correct label sequence, that is: Among them, λ and θ are regularization parameters.

6. According to the method for constructing a knowledge graph based on the BTBC model according to claim 1, Features: The BERT model in step 1 is a bidirectional encoder of Transformer, which encodes the input training corpus into word vectors.

7. According to the method for constructing a knowledge graph based on the BTBC model according to claim 1, Features: The pre-training in step 1 is specifically as follows: in the face of a large amount of unlabeled data crawled, the entry names of Baidu Encyclopedia and the entities in the triples formed by semi-structured data are used to combine and remove duplicate values ​​to back-label the unstructured data, thereby automatically generating a large amount of training corpus. Because the BERT model has a maximum length requirement for the input sentences, long sentences also need to be truncated according to periods.

8. According to the method for constructing a knowledge graph based on the BTBC model according to claim 1, Features: Step 6 is as follows: Using Neo4j graph database, the basic elements are nodes, edges and attributes. Nodes are used to store entities and attributes, edges are used to store the relationship between entities, and topological structure is used to store and display the data in a visual form.