Triplet forest-based entity relation joint extraction method and system

Abstract

The application provides an entity relation joint extraction method and system based on a triple forest, which comprises the following steps: obtaining corpus to be subjected to entity relation extraction, and obtaining a sentence and a corresponding word sequence thereof; inputting the word sequence into a BERT model, performing word segmentation on the word sequence by the BERT model to obtain a subword sequence, encoding the subword sequence by using the BERT model to obtain a distributed representation of the sentence; inputting the distributed representation into a CRF model to label entities in the sentence and obtain vector representations of the entities; inputting the entity vectors, obtaining hidden layer vectors of interaction information between entities contained in the entity vectors and interaction information between the entities and the input sentence by a multi-head attention mechanism in a TransformerDecoder module; taking the hidden layer vectors as initial states and initial hidden layer units of a Tree-RNN, inputting the entity representations into the Tree-RNN, generating relations participated in by a head entity of a root node of the Tree-RNN, selecting tail entities of the head entity according to the head entity and the corresponding relations, thereby generating an overlapping triple tree, and further decoding to obtain entity relation triplets.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, in particular, to an entity relation joint extraction method for overlapping scene. BACKGROUND

[0002] Entity and relation are the main semantic information carriers and the structured form of natural language, so they are widely used in natural language processing tasks. Entity can be an actual existing thing in reality, such as person, place, company, telephone, animal, etc., or time, date, etc., which is usually the subject or object (in the main sentence or subordinate sentence) in natural language. Relation is the connection between entities, which is the semantic relationship between entities in natural language. Natural language can be structured as overlapping and nested combination of 〈head entity, relation, tail entity〉 triplets. Entity relation joint extraction is a technology that extracts entity and relation from text at the same time. For example, given the text "A is known as the romantic capital of Z country", after entity relation joint extraction, 〈A, capital, Z country〉 can be extracted, the head entity is A, the tail entity is Z country, and the relation between them is capital. However, not all sentences in natural language are simple and can be represented by a 〈head entity, relation, tail entity〉 triplet. There are also complex scenes with multiple triplets overlapping with each other, which cause difficulties in joint extraction. Therefore, entity relation joint extraction in relation overlapping scene is an important problem of entity relation joint extraction, which is called overlapping relation problem.

[0003] The overlapping relation problem refers to the case where the same entity participates in multiple triplets in a sentence. Specifically, it can be divided into entity pair overlap (EPO) and single entity overlap (SEO). Entity pair overlap refers to the existence of multiple relations between entity pairs, while single entity overlap refers to the existence of the same entity in two or more triplets. The problem of overlapping relation is difficult to identify for most entity relation joint extraction methods due to the overlapping of positions, and it is also easy to confuse, as the overlapping of positions causes the similarity of semantic vectors, which makes the model overestimate the existence of relations that should not exist.

[0004] Currently, the joint extraction methods of overlapping entity relationships in the relationship overlapping scenario mainly include three categories: sequence labeling, table filling and sequence generation. The sequence labeling based method can only handle overlapping relationship problems through multiple sequence labeling. The table filling method fills the relationship between the i th word and the j th word into the i th row and the j th column of the table, and fills the entity information in the diagonal position, so it cannot handle the entity pair overlapping problem and needs to be further expanded. The sequence generation method translates the text into a sequence of entity, relationship triplets through a sequence-to-sequence framework. Although it can theoretically handle overlapping relationships, the information order transmission between triplet sequences causes transmission errors, so some works improve the framework to better model the interactive information of entities and relationships, such as sequence-to-set framework and sequence-to-tree framework. However, the existing improvements still cannot balance information transmission and transmission errors, and there is a human parameter limit on the maximum generation length, so further improvement is needed to obtain better entity relationship joint extraction performance. SUMMARY

[0005] In order to balance information transmission and transmission errors, the present application provides a kind of entity relationship joint extraction method based on triplet forest. The method takes sentence text as input, uses BERT encoder for encoding, and uses sequence labeling to obtain entity, then inputs entity information into Transformer Decoder, further learns the interactive information between entities, finally generates triplets in the form of triplet forest, and completes the joint extraction of entity relationship in sentence.

[0006] In view of the deficiencies of the prior art, the present application provides an entity relationship joint extraction method based on triplet forest, which comprises:

[0007] Step 1, obtaining the corpus to be extracted, and performing sentence segmentation and word segmentation processing to obtain the sentence and its corresponding word sequence;

[0008] Step 2, inputting the word sequence into the BERT model, and the BERT model performs word segmentation on the word sequence to obtain a subword sequence, and using the BERT model to encode the subword sequence to obtain a distributed representation of the sentence;

[0009] Step 3, inputting the distributed representation into the CRF model to label the entities in the sentence to obtain the vector representation of the entities;

[0010] Step 4, inputting the entity vector, obtaining the hidden layer vector of the interactive information between entities and the interactive information between entities and input sentence in the entity vector through the multi-head attention mechanism in the Transformer Decoder module;

[0011] Step 5, input the entity representation to the Tree-RNN with the hidden layer vector as the initial state and initial hidden layer unit of the Tree-RNN, generate the relationship participated by the head entity of the root node of the Tree-RNN according to the head entity and its corresponding relationship, select the tail entity according to the head entity and its corresponding relationship, thereby generating an overlapping triple tree, and further decoding to obtain an entity relationship triple.

[0012] The entity relationship joint extraction method based on triple forest, wherein step 2 comprises: using the pre-trained BERT model to encode the word sequence to obtain the distributed representation :

[0013]

[0014] Step 3 comprises:

[0015] Step 31, obtain the probability of each word belonging to each label in the sentence by the following formula to label the entity in the sentence:

[0016]

[0017] Step 32, randomly initialize the state transition matrix for the CRF model , wherein is the number of initial state and end state labels added to the BIOES label of sequence labeling; is the probability of the i-th label transitioning to the j-th label, and the label sequence of sequence labeling is , then the score of the model marking the sentence text X as the label Y is:

[0018]

[0019] In the decoding stage, the Viterbi algorithm is used for decoding to obtain the optimal predicted label sequence, and then the corresponding entity is obtained by BIOES labeling .

[0020] Step 4 comprises: automatically learning the interaction information between entities, the interaction information between entities and input text by the multi-head attention mechanism in the Transformer Decoder module, and subsequently generating a head entity overlapping triple tree from each entity; and the entity vector representation is the vector representation of the entity set E' generated by the sequence labeling module;

[0021] During training, E' = GE|E, GE is the actually labeled entity set, E is the entity set predicted by the sequence labeling module, and | is the set merging;

[0022] During prediction, E' = E; , wherein 、 head and tail indices of the ith entity; entity in the text hidden layer representation vector representation of the corresponding subword aggregation algorithm aggre results in an entity vector representation n+1 entity vector representations combine into a matrix :

[0023]

[0024] entity matrix as input to a Transformer Decoder module, which is internally structured as N layers of Transformers, each containing a multi-head self-attention mechanism module and a multi-head encoder-decoder attention mechanism module; the multi-head self-attention mechanism module is used to learn the connections between entities, while the multi-head encoder-decoder attention mechanism module is used to learn the connections between the sentence text and the entities, obtaining context information specific to the entities; through the following formula, the entity matrix and the sentence text representation are interacted through the Transformer Decoder module, obtaining a hidden layer cell that contains information about the entities and the sentence text ,

[0025]

[0026] This step 5 includes: generating a triple forest from the hidden layer cell , which consists of n+1 generated tree modules, each of which is a Tree-RNN; the input of the ith Tree-RNN is the entity corresponding to the position and the hidden layer cell of the Transformer Decoder module ; the jth branch of the Tree-RNN is a triple with the ith entity as the head entity ; the initial hidden layer state of the Tree-RNN is , the memory cell is initialized as , and the initial output is , and the input at each subsequent time step is the vector representation of the predicted result at the previous time step; if the prediction is an entity , then the input is the vector representation of the entity , and if it is a relation , then the input is the vector representation of the relation​ ; each time step is calculated as follows, where the hidden state, memory cell and input of each time step are , time step :

[0027]

[0028] The output of each time step LSTM, if the relationship is predicted, is passed through a linear layer to obtain the probability of each relationship using sigmoid , as follows, where ;

[0029]

[0030] If the current time step predicts an entity, it is passed through a linear layer to further integrate the sentence text context information , and the probability of selecting each entity is obtained using the sigmoid activation function , where the tail of the entity predicted when the sequence label is , , :

[0031] .

[0032] The entity and relationship joint extraction method based on the triple forest, wherein step 5 comprises:

[0033] In the training phase, when predicting entities and relationships at each time step, negative examples are obtained by negative sampling for each sentence , and its label is NA, which is combined with the positive example to form , which are jointly classified; wherein the entity and relationship prediction is multiple binary classification;

[0034] The loss of the triple forest module is the loss of each time step:

[0035] .

[0036]

[0037] L is the final loss, which is used to jointly train the BERT model, the CRF model, the Transformer Decoder module and the triple forest.

[0038] The entity and relationship joint extraction method based on the triple forest, wherein step 5 comprises, according to the entity and relationship triple, constructing or enriching a knowledge graph for a question and answer model or a search recommendation system.

[0039] The application further provides a triple forest-based entity relation joint extraction system, which comprises the following modules:

[0040] An initial module is configured to acquire a corpus to be subjected to entity relation extraction, and perform sentence segmentation and word segmentation on the corpus to obtain sentences and corresponding word sequences of the sentences;

[0041] An encoding module is configured to input the word sequences into a BERT model, perform word segmentation on the word sequences by using the BERT model to obtain sub-word sequences, encode the sub-word sequences by using the BERT model to obtain distributed representations of the sentences;

[0042] A labeling module is configured to input the distributed representations into a CRF model to label entities in the sentences and obtain vector representations of the entities;

[0043] An interaction information extraction module is configured to input the entity vectors into a Transformer Decoder module to obtain hidden layer vectors of interaction information between entities in the entity vectors and interaction information between the entities and input sentences by using a multi-head attention mechanism in the Transformer Decoder module;

[0044] An entity relation extraction module is configured to input the hidden layer vectors into a Tree-RNN as initial states and initial hidden layer units, input the entity representations into the Tree-RNN, generate relations participated by a head entity of a root node of the Tree-RNN, select tail entities of the head entity according to the head entity and the corresponding relations, generate an overlapping triple tree, and further decode to obtain entity relation triplets.

[0045] The triple forest-based entity relation joint extraction system, wherein the encoding module is configured to encode the word sequences by using the pre-trained BERT model to obtain the distributed representations :

[0046]

[0047] The labeling module is configured to:

[0048] obtain probabilities of each word in a sentence belonging to each label by the following formula to label entities in the sentence:

[0049]

[0050] randomly initialize a state transition matrix of the CRF model , wherein is a number of the BIOES labels for sequence labeling plus numbers of initial states and end states; is a probability of the i-th label transitioning to the j-th label, and a label sequence for sequence labeling is Then the model scores the sentence text X with the label Y as:

[0051]

[0052] In the decoding stage, the Viterbi algorithm is used to decode to obtain the optimal prediction label sequence, and then the BIOES annotation is obtained to obtain the corresponding entity ;

[0053] The interaction information extraction module is used to automatically learn the interaction information between entities, the interaction information between entities and input text through the multi-head attention mechanism in the Transformer Decoder module, and then generate a head entity overlapping triple tree from each entity; the entity vector representation is the vector representation of the entity set E' generated by the sequence labeling module;

[0054] During training, E' = GE|E, GE is the actual annotated entity set, E is the entity set predicted by the sequence labeling module, and | is the set merging;

[0055] During prediction, E' = E; , wherein , are the head and tail indexes of the i-th entity; the entity In the text hidden layer representation , the vector representation of the corresponding subword is obtained by the aggregation algorithm aggre , and n+1 entity vector representations are combined into a matrix :

[0056]

[0057] The entity matrix is input into the Transformer Decoder module, and the Transformer Decoder module is internally an N-layer Transformer structure, each layer containing a multi-head self-attention mechanism module and a multi-head encoder-decoder attention mechanism module; the multi-head self-attention mechanism module is used to learn the relationship between entities, and the multi-head encoder-decoder attention mechanism module is used to learn the relationship between the sentence text and the entity to obtain the context information specific to the entity; through the following formula, the entity matrix and the sentence text representation complete the interaction through the Transformer Decoder module to obtain the hidden layer unit , ; containing entity information and sentence text information

[0058]

[0059] The entity relationship extraction module is configured to generate a triple forest according to the hidden layer units of the first layer of neural network The triple forest is composed of n+1 generated tree modules, and each generated tree module is a Tree-RNN; the input of the i-th Tree-RNN is the vector representation of the entity corresponding to the position And the hidden layer units of the Transformer Decoder module The j-th branch of the Tree-RNN is a triple with the i-th entity as the head entity The initial hidden layer state of the Tree-RNN is The memory unit is initialized as The initial output is The input of each time step thereafter is the vector representation of the predicted result of the previous time step; if the prediction is an entity , the input is the vector representation of the entity If the prediction is a relationship , the input is the vector representation of the relationship The calculation of each time step is as follows, where the hidden layer state, memory unit and input of each time step are Time step :

[0060]

[0061] The output of each time step of the LSTM is the probability of each relationship obtained by passing through a linear layer and using a sigmoid function if the relationship is predicted at the time step , as follows, where ;

[0062]

[0063] If the current time step predicts an entity, the linear layer is further fused with the sentence text context information The probability of selecting each entity is obtained by using a sigmoid activation function , which corresponds to the tail of the predicted entity in sequence labeling, where the weight of the linear layer is , , :

[0064] .

[0065] The entity relationship extraction module is configured to generate a triple forest according to the hidden layer units of the first layer of neural network

[0066] In the training stage, for each time step of entity and relation prediction, negative sampling is performed on each sentence to obtain negative examples , and the label is NA, and the positive examples are merged into , and the two are jointly subjected to multi-classification; wherein the entity and relation prediction is multiple binary classification;

[0067] The loss of the triple forest module is the loss of each time step:

[0068] .

[0069]

[0070] L is the final loss, used for joint training of the BERT model, the CRF model, the Transformer Decoder module and the triple forest.

[0071] The entity and relation extraction system based on the triple forest, wherein the entity and relation extraction module is used to construct or enrich a knowledge graph according to the entity and relation triple, for a question and answer model or a search recommendation system.

[0072] The application further provides a storage medium for storing a program for performing any one of the entity and relation joint extraction based on the triple forest.

[0073] The application further provides a client for the entity and relation joint extraction system based on the triple forest.

[0074] From the above scheme, it can be seen that the application has the following advantages:

[0075] A kind of entity relation joint extraction method based on triplets forest, comprising the following steps: 1) training corpus preprocessing, 2) pre-coding using BERT pre-training language model, 3) sequence labeling, 4) entity interaction, 5) triplets generated by triplets forest.Compared with the existing entity relation joint extraction technology, the present application has the following advantages: one is that the unordered interaction information between triplets is obtained using the entity interaction module;Second, triplets forest is used to further model the unordered interaction between triplets, the ordered interaction within triplets and weaken the transmission error caused by it at the same time, to enhance its recognition ability to overlapping relations;Third, through the optimization of model structure, the inherent maximum length limit problem of sequence generation framework is avoided;Four is to simplify, the multi-task composite processing of one stage in the entity relation joint extraction problem is simple and effective in simple data set, but often performs poorly in complex data set, therefore it is decomposed into entity extraction, entity interaction and entity pair combination link of triplets generation three simple parts, joint training, reduce the difficulty of problem, improve model performance.The F1 value of entity relation joint extraction of the present application on WebNLG and NYT data set is 88.3% and 92.7% respectively. BRIEF DESCRIPTION OF DRAWINGS

[0076] Figure 1 is the overall flowchart of the present application;

[0077] Figure 2 is the detailed introduction diagram of sequence labeling module in the present application;

[0078] Figure 3 is the detailed introduction diagram of entity interaction module in the present application;

[0079] Figure 4 is the detailed introduction diagram of triplets extraction module in the present application;

[0080] Figure 5 is the detailed introduction diagram of Tree-RNN in the present application. DETAILED DESCRIPTION

[0081] The entity relation joint extraction method based on triplets forest proposed by the present application comprises the following steps:

[0082] 1) training corpus preprocessing, the training corpus used by the present application is selected from NYT data set and WebNLG data set, and the processing process includes sentence division and conversion of annotation information;

[0083] 2) Pre-encode using BERT pre-trained language model, which takes the word sequence of a sentence as input, uses the word segmentation module of the BERT pre-trained language model to segment the words, then inputs the BERT to pre-encode the sub-word sequence, and outputs the distributed representation of the sentence text. Compared with traditional static word and word vectors, BERT trained on a large corpus can provide more dynamic semantic expression;

[0084] 3) Sequence labeling, which uses CRF to label entities (head entity and tail entity) in the sentence and obtains the vector representation of the entity; the input of CRF is the distributed representation of the sentence text, but the object of labeling is the sentence itself, which marks which entities in the sentence. CRF is a conditional random field, which is a machine learning model.

[0085] 4) Entity interaction, which inputs the entity vector into the Transformer Decoder module to obtain the hidden layer vector containing the interaction information between entities, the interaction information between entities and the input sentence. The input of this step is the vector representation of the entity and the distributed representation of the sentence, and the interaction between entities and the interaction between the sentence and the entity is obtained.

[0086] 5) Generate triplets through triplet forest, starting from the hidden layer vector obtained in step 4), according to the obtained entity interaction information and the interaction between entity and text, generate a tree of overlapping triplets for each entity, and further decode to obtain the final entity relationship triplet. Each tree is an overlapping triplet, and each branch of the tree is a triplet. The first time step inputs the representation of the entity, if the entity is a head entity, it predicts the relationship, if not, it outputs no relationship and terminates the branch; the second time step inputs the vector representation of the relationship corresponding to the branch, and outputs the possible entity index position (span prediction). When decoding, the structure of the tree is split into a list of triplets according to the branches (which is the output of the final model).

[0087] In order to make the above features and effects of the present application more clear and easy to understand, the following embodiments are specifically described below, and the detailed description is as follows in conjunction with the drawings.

[0088] The present application proposes an entity relationship joint extraction method based on triplet forest, and the overall flowchart of the method is as follows Figure 1The method mainly includes four parts of BERT pre-encoding, sequence labeling, entity interaction and triple forest. The BERT pre-encoding uses the BERT pre-training language model to encode the input word and word sequence to obtain the semantic representation thereof. The sequence labeling takes the semantic representation of the text output by the BERT as the input, combines the CRF, labels the category of each word in the text, and thus predicts the entity. The predicted entity combines the semantic representation of the text to obtain the semantic representation of the entity. The entity interaction part inputs the entity representation and the text representation, captures the interaction information between entities and the interaction information between entities and the text through the attention mechanism, and simplifies the subsequent triple generation task into entity combination and relationship prediction. The triple forest starts from the head entity according to the previous interaction information, predicts all the relationships in which the head entity participates, and then selects the possible tail entity from the entity generated by the sequence labeling under the branches of different relationships, so as to generate the overlapping entity relationship triple. The specific method steps include:

[0089] S1, using the BERT pre-training language model to pre-encode the input word sequence to obtain the distributed semantic representation of each word in the text after the BERT pre-training model encoding.

[0090] S2, according to the semantic representation of the text, combining the CRF to learn the transition information between labels, thus predicting the label of each word, and decoding to obtain the entity. The entity combines the semantic representation of the text to obtain the corresponding entity representation.

[0091] S3, input all the entity representation and the text representation into the entity interaction module, learn the relevance between entities and the relevance between entities and the text through the attention mechanism, and obtain a hidden layer vector containing rich information for each entity.

[0092] S4, input the above hidden layer vector into the triple forest, and each hidden layer vector is used as the initial state and initial hidden layer unit of the tree recursive neural network Tree-RNN, input the entity representation, and generate an overlapping triple tree. From the head entity of the root node, the relationship in which the head entity participates is generated, different relationships form different branches, and under the branches of different relationships corresponding to the head entity, the tail entity is further selected, and thus the overlapping triple is generated.

[0093] The CRF identifies the entity and obtains the position of the entity. The vector corresponding to the position in the distributed vector output by the Bert is the vector of the entity. All the vectors of the entity are input into the triple forest as the root node. If it is a head entity, continue to branch to obtain a triple. If not, the tree only has a root node (head entity).

[0094] Specifically, S1 includes two sub-steps, as shown in Figure 1

[0095] ​S101, training data preprocessing. Tokenizer module in Transormers library is used to segment the input text, and the input of the same batch is padded to the same length according to the longest text length of the batch.

[0096] S102, BERT pre-training model coding. The input word and word sequence are pre-coded by the BERT language model pre-trained by a large amount of corpus. Compared with the traditional static word vector, this method can obtain more rich dynamic semantic expression .

[0097]

[0098] Further, S2 includes two sub-steps, and the specific structure is as shown in Figure 2

[0099] S201, the output of BERT Through a linear layer, the probability of each label of sequence labeling is calculated. The sequence labeling is only five labels of entity and BIOES. BIOES is a labeling method of sequence labeling, including: the beginning of entity B-begin, the middle of entity I-inside, not entity O-Other, the end of entity E-end, and single word entity S-single.

[0100]

[0101] S202, a CRF layer is set after the linear layer to learn the transition relationship between labels to better improve the entity extraction capability. The state transition matrix of the CRF layer is randomly initialized in this paper , wherein is the number of initial state and end state added to the BIOES label of sequence labeling. A sentence has multiple words. If the previous word is marked as the beginning of the entity B, the probability of the current word being marked as the middle of the entity I and the end of the entity E is much greater than other labels, which is called transition probability. is the probability of the i-th label transitioning to the j-th label. Let the label sequence of sequence labeling be , then the score of the CRF model for marking the sentence text X as the label Y is:

[0102]

[0103] In the decoding stage, the Viterbi algorithm is used for decoding to obtain the optimal predicted label sequence, and then the corresponding entity is obtained by BIOES labeling .

[0104] The goal of S3 is to interact information, and obtain more initial hidden layer units and memory units for the triple forest. As shown in​Figure 3 The entity interaction module is essentially a Transformer Decoder module with Position Encoding removed. It automatically learns interaction information between entities and between entities and the input text through its multi-head attention mechanism. Subsequently, it generates a triple tree with overlapping head entities starting from each entity. Therefore, this stage is essentially the interaction between triple trees, that is, the complex, unordered interaction between triples with non-overlapping head entities. The input to this part is the encoder's output. and entity vector representation . It is a vector representation of the entity set E' generated by the sequence labeling module. It should be noted that during training, E' = GE|E, where GE is the actual labeled entity set, and E is the entity set predicted by the sequence labeling module; | represents set merging. During prediction, E' = E. Here, we denote... ,in , Here are the head and tail indices for the i-th entity. Entity In the implicit representation of text Vector representation of corresponding subwords The aggregation algorithm aggre yields entity vector representations. n+1 entity vectors are combined into a matrix The aggregation algorithm here can be implemented using mean, summation, or a linear layer. The specific calculation is as follows:

[0105]

[0106] The resulting entity matrix This serves as the input to the Transformer Decoder module. This module removes positional information, thus essentially taking the entity set as input. This is to ensure that the triplet trees generated from the entities are unordered, further guaranteeing the unordered propagation of errors between triples. Internally, the Transformer Decoder module is an N-layer Transformer structure, each layer containing a multi-head self-attention mechanism module and a multi-head encoder-decoder attention mechanism module. The multi-head self-attention mechanism module learns the relationships between entities, while the multi-head encoder-decoder attention mechanism module learns the relationships between sentence text and entities, obtaining entity-specific contextual information. As shown in the formula below, the entity matrix... and sentence text representation After interacting with the Transformer Decoder module, we obtain the hidden layer units that contain other entity information and sentence text information. .in, .

[0107]

[0108] S4 process as follows Figure 4 As shown, it mainly consists of the previous step Start generating triples. Hidden layer unit. A triplet forest was further generated from the entities. This forest consists of n+1 spanning tree modules, where n+1 is the total number of predicted entities. Each spanning tree module is a Tree-RNN, numbered from 0 to n. Figure 5 As shown, the input of the i-th Tree-RNN is the entity corresponding to this position. and the hidden unit of the Transformer Decoder The j-th branch of this Tree-RNN is a triple with the i-th entity as the head entity. If the input entity and the corresponding hidden layer units In the first time step, it is predicted that the vector will participate in relations r1 and r2. Therefore, in the second time step, two branches are formed, which are input into the relation vectors r1 and r2 respectively, denoted as... The relation vector is obtained through a lookup using randomly initialized relation embeddings. In the first branch, the input is the relation vector of r1. Since h2 does not actually have a relation to r1, this is an overprediction from the previous time step, so it is predicted as NA in this step, terminating this branch. That is, this branch does not have a triple. In the second branch, the relation vector of r2 is input. In the second time step, it is predicted that (h2, r2) may contain a tail entity t2. Therefore, in the third time step, the entity vector of the tail entity t2 is input. Since the triple (h2, r2, t2) exists, the process terminates normally, and the prediction is 1. If it does not exist, the prediction is 0. The initial hidden state of the Tree-RNN is... Its memory cells are initialized to The initial output is The input at each subsequent time step is a vector representation of the prediction result from the previous time step. If the prediction is an entity... Then input Vector representation of the entity If it is a relationship Then input Vector representation of relations The specific calculations for each time step are as follows, where the hidden state, memory unit, and input for each time step are: Time step .

[0109]

[0110] For the output of each time step LSTM, if the time step predicts the relationship, the probability of each relationship is obtained through the linear layer using sigmoid , as shown in the following formula, wherein, .

[0111]

[0112] If the time step predicts the entity, the linear layer is further used to fuse the sentence text context information , and the sigmoid activation function is used to obtain the probability of selecting each entity , which corresponds to the tail of the entity predicted in the sequence labeling. As shown in formula 9, the weight of the linear layer is , , .

[0113]

[0114] In the training phase, when predicting the entity and relationship of each time step, negative examples are obtained by negative sampling for each sentence , and the label thereof is denoted as NA, which is combined with the positive example to form , and the two are jointly classified. The loss of the triple forest module constructed in the present application is the sum of the losses of each time step:

[0115]

[0116]

[0117] L is the final loss. In the training phase, the Bert / CRF / entity interaction / triple forest appearing in the text are trained according to L, and the parameters are updated by joint training. In the above entity and relationship prediction, multiple binary classifications are performed.

[0118] Compared with the existing Chinese event extraction technology, the present application has the following advantages: first, the triple forest model structure is constructed, which weakens or even reduces the transmission error caused by sufficient information transmission, thereby enhancing the performance of joint extraction; second, the error correction mechanism is used to further reduce the transmission error in the triple; third, the entity interaction is used to further enhance the information interaction in the model, and the entity generation in the next triple generation is simplified to entity selection; fourth, the performance of entity and relationship joint extraction is improved, and the F1 value of entity and relationship joint extraction of the present application on the WebNLG and NYT data sets reaches 88.3% and 92.7%, respectively.

[0119] The following is a system embodiment corresponding to the above method embodiment. The present embodiment can be implemented in cooperation with the above embodiments. The related technical details mentioned in the above embodiments are still valid in the present embodiment. In order to reduce repetition, they will not be described here. Correspondingly, the related technical details mentioned in the present embodiment can also be applied in the above embodiments.

[0120] The present application also provides a triple forest-based entity relationship joint extraction system, which comprises:

[0121] An initial module is configured to obtain a corpus to be subjected to entity relationship extraction, and perform sentence segmentation and word segmentation on the corpus to obtain a sentence and a corresponding word sequence of the sentence;

[0122] An encoding module is configured to input the word sequence into a BERT model, perform word segmentation on the word sequence by using the BERT model to obtain a subword sequence, encode the subword sequence by using the BERT model to obtain a distributed representation of the sentence;

[0123] A labeling module is configured to input the distributed representation into a CRF model to label entities in the sentence and obtain a vector representation of the entities;

[0124] An interaction information extraction module is configured to input the vector representation of the entities into a Transformer Decoder module to obtain hidden layer vectors of interaction information between the entities and interaction information between the entities and an input sentence by using a multi-head attention mechanism in the Transformer Decoder module;

[0125] An entity relationship extraction module is configured to input the hidden layer vectors as initial states and initial hidden layer units of a Tree-RNN, input the entity representation into the Tree-RNN, generate a relationship participated by a head entity of a root node of the Tree-RNN according to the head entity, select a tail entity of the head entity according to the head entity and the relationship corresponding to the head entity, generate an overlapping triple tree, and further decode to obtain an entity relationship triple.

[0126] The triple forest-based entity relationship joint extraction system, wherein the encoding module is configured to encode the word sequence by using a pre-trained BERT model to obtain the distributed representation :

[0127]

[0128] The labeling module is configured to:

[0129] The probability of each word in the sentence belonging to each label is obtained by the following formula to label the entities in the sentence:

[0130]

[0131] Initialize the state transition matrix randomly for the CRF model where is the number of initial state and end state added to the BIOES label of sequence labeling; is the probability of the i-th label transition to the j-th label, and the label sequence of sequence labeling is Then the score of the model marking the sentence text X with the label Y is:

[0132]

[0133] In the decoding stage, the Viterbi algorithm is used for decoding to obtain the optimal predicted label sequence, and then the corresponding entity is obtained by BIOES annotation .

[0134] The interaction information extraction module is used to automatically learn the interaction information between entities, the interaction information between entities and input text through the multi-head attention mechanism in the Transformer Decoder module, and subsequently generate a head entity overlapping triple tree from each entity; the entity vector representation is the vector representation of the entity set E' generated by the sequence labeling module;

[0135] During training, E' = GE|E, GE is the actually annotated entity set, E is the entity set predicted by the sequence labeling module, and | is the set merging;

[0136] During prediction, E' = E; where , is the head and tail index of the i-th entity; the entity In the text hidden layer representation , the vector representation of the corresponding subword is The aggregation algorithm aggre obtains the entity vector representation , and the n+1 entity vector representations are combined into a matrix :

[0137]

[0138] The entity matrix is input into the Transformer Decoder module, and the Transformer Decoder module is internally an N-layer Transformer structure, each layer containing a multi-head self-attention mechanism module and a multi-head encoder-decoder attention mechanism module; the multi-head self-attention mechanism module is used to learn the relationship between entities, and the multi-head encoder-decoder attention mechanism module is used to learn the relationship between the sentence text and the entity to obtain entity-specific context information; through the following formula, the entity matrix and sentence text representation After the interaction through the Transformer Decoder module, the hidden layer unit containing the entity information and the sentence text information is obtained , ;

[0139]

[0140] The entity relationship extraction module is configured to generate a triple forest according to the hidden layer unit The triple forest is composed of n+1 generated tree modules, and each generated tree module is a Tree-RNN; the input of the i th Tree-RNN is the vector representation of the entity at the corresponding position and the hidden layer unit of the Transformer Decoder module ; the j th branch of the Tree-RNN is a triple with the i th entity as the head entity ; the initial hidden layer state of the Tree-RNN is , the memory cell is initialized as , and the initial output is , and the input at each subsequent time is the vector representation of the predicted result at the previous time; if the prediction is an entity , the input is the vector representation of the entity , and if the prediction is a relationship , the input is the vector representation of the relationship ; the calculation at each time step is as follows, where the hidden layer state, memory cell and input at each time step are , and the time step :

[0141]

[0142] is the output of each time step LSTM, if the relationship is predicted at this time step, the probability of each relationship is obtained through the linear layer using sigmoid , the formula is as follows, where ;

[0143]

[0144] If the current time step predicts an entity, the linear layer is further used to fuse the sentence text context information , and the sigmoid activation function is used to obtain the probability of selecting each entity , which corresponds to the tail of the predicted entity in the sequence labeling, where the weight of the linear layer is , , :

[0145] .

[0146] The entity relation joint extraction system based on the triple forest, wherein the entity relation extraction module is used for:

[0147] In the training stage, for each time step of entity and relation prediction, negative examples are obtained by negative sampling for each sentence , and the label is NA, and the positive examples are merged into and are subjected to multi-classification; wherein the entity and relation prediction is multi-time binary classification;

[0148] The loss of the triple forest module is the loss of each time step and:

[0149] .

[0150]

[0151] L is the final loss, which is used for joint training of the BERT model, the CRF model, the Transformer Decoder module and the triple forest.

[0152] The entity relation joint extraction system based on the triple forest, wherein the entity relation extraction module is used for constructing or enriching a knowledge graph according to the entity relation triple, so as to be used for a question and answer model or a search recommendation system.

[0153] The application further provides a storage medium for storing a program for executing any one of the entity relation joint extraction based on the triple forest.

[0154] The application further provides a client for the entity relation joint extraction system based on the triple forest.

Claims

1. A method for joint entity relation extraction based on triplet forest, characterized in that, include: Step 1: Obtain the corpus from which entity relations are to be extracted, and perform sentence segmentation and word segmentation to obtain sentences and their corresponding word sequences; Step 2: Input the word sequence into the BERT model. The BERT model segments the word sequence into sub-word sequences. The BERT model is then used to encode these sub-word sequences to obtain a distributed representation of the sentence. Step 3: Input the distributed representation into the CRF model, label the entities in the sentence, and obtain the vector representation of the entities; Step 4: Input the entity vector and use the multi-head attention mechanism in the Transformer Decoder module to obtain the hidden layer vector containing the inter-entity interaction information and the interaction information between the entity and the input sentence. Step 5: Use the hidden layer vector as the initial state and initial hidden layer unit of the Tree-RNN, input the entity representation into the Tree-RNN, generate the relations it participates in from the head entity of the root node of the Tree-RNN, select the tail entity according to the head entity and its corresponding relation, thereby generating an overlapping triple tree, and then further decode to obtain the entity relation triple. Step 2 includes: encoding the word sequence using the pre-trained BERT model to obtain the distributed representation. : Step 3 includes: Step 31: Obtain the probability of each word in the sentence belonging to each tag using the following formula, in order to label entities in the sentence: Step 32: Randomly initialize the state transition matrix for the CRF model. ,in Add the number of initial state and final state tags to the BIOES tags of the sequence label; Let be the probability of moving from the i-th label to the j-th label. The label sequence for sequence labeling is: The model then scores the sentence text X labeled with tag Y as follows: In the decoding stage, the Viterbi algorithm is used to decode and obtain the optimal predicted label sequence, which is then labeled by BIOES to obtain the corresponding entities. .

2. The entity relation joint extraction method based on triplet forest as described in claim 1, characterized in that, Step 4 includes: automatically learning the interaction information between entities and the interaction information between entities and input text through the multi-head attention mechanism in the Transformer Decoder module; subsequently generating a triple tree with overlapping head entities starting from each entity; entity vector representation. It is a vector representation of the entity set E' generated by the sequence labeling module; During training, E' = GE|E, where GE is the actual set of labeled entities, E is the set of entities predicted by the sequence labeling module, and | represents set merging; When making predictions, E' = E; ,in , Here are the head and tail indices for the i-th entity; entity In the implicit layer of text representation Vector representation of corresponding subwords The aggregation algorithm aggre yields entity vector representations. n+1 entity vectors are combined into a matrix : Entity matrix As input to the Transformer Decoder module, the Transformer Decoder module internally consists of an N-layer Transformer structure. Each layer contains a multi-head self-attention mechanism module and a multi-head encoder-decoder attention mechanism module. The multi-head self-attention mechanism module is used to learn the relationships between entities, while the multi-head encoder-decoder attention mechanism module is used to learn the relationships between sentence text and entities, obtaining entity-specific contextual information. The entity matrix is ​​obtained through the following formula: and sentence text representation After interacting with the Transformer Decoder module, the hidden layer units containing entity information and sentence text information are obtained. , ; Step 5 includes: based on the hidden layer unit Generate a triplet forest, which consists of n+1 spanning tree modules, and each spanning tree module is a Tree-RNN; the input of the i-th Tree-RNN is the entity corresponding to that position. Hidden units of the Transformer Decoder module The j-th branch of this Tree-RNN is a triple with the i-th entity as the head entity. The initial hidden state of Tree-RNN is Its memory cells are initialized to The initial output is The input at each subsequent time step is a vector representation of the prediction result from the previous time step; if the prediction is an entity... Then input Vector representation of the entity If it is a relationship Then input Vector representation of relations The specific calculations for each time step are as follows, where the hidden state, memory unit, and input for each time step are: Time step : For each time step, the output of the LSTM, if a relation is predicted at that time step, is passed through a linear layer and then subjected to a sigmoid function to obtain the probability of each relation. The formula is as follows, where, ; If the entity is predicted at the current time step, it goes through a linear layer and is further fused with sentence text context information. The probability of selecting each entity is obtained by using the sigmoid activation function. This corresponds to the predicted entity tail during sequence labeling, where the weights of the linear layer are... , , : 。 3. The entity relation joint extraction method based on triplet forest as described in claim 2, characterized in that, Step 5 includes: During the training phase, at each time step for entity and relation prediction, negative samples are obtained by negative sampling for each sentence. Let its label be NA, and the positive example be... merged into Then, multiple classifications are performed together; the entity and relation predictions are multiple binary classifications. The loss for constructing the triplet forest module is the sum of losses at each time step: L is the final loss, used to jointly train the BERT model, the CRF model, the Transformer Decoder module, and the triplet forest.

4. The entity relation joint extraction method based on triple forest as described in claim 1, characterized in that, Step 5 includes constructing or enriching a knowledge graph based on the entity relationship triples for use in question-answering models or search recommendation systems.

5. A joint entity relation extraction system based on triple forest, characterized in that, include: The initial module is used to acquire the corpus from which entity relations are to be extracted, and to perform sentence segmentation and word segmentation on it to obtain sentences and their corresponding word sequences; The encoding module is used to input the word sequence into the BERT model. The BERT model segments the word sequence into sub-word sequences, and then uses the BERT model to encode the sub-word sequences to obtain a distributed representation of the sentence. The annotation module is used to input the distributed representation into the CRF model, annotate entities in the sentence, and obtain vector representations of the entities; The interaction information extraction module is used to input the entity vector and obtain the hidden layer vector containing the interaction information between entities and the interaction information between entities and the input sentence through the multi-head attention mechanism in the Transformer Decoder module. The entity relation extraction module is used to take the hidden layer vector as the initial state and initial hidden layer unit of the Tree-RNN, input the entity representation into the Tree-RNN, generate the relation it participates in from the head entity of the root node of the Tree-RNN, select the tail entity according to the head entity and its corresponding relation, thereby generating an overlapping triple tree, and then further decode to obtain the entity relation triple. The encoding module is used to encode the word sequence using the pre-trained BERT model to obtain the distributed representation. : This annotation module is used for: The probability of each word in a sentence belonging to each tag is obtained using the following formula, which is used to label entities in the sentence: Randomly initialize the state transition matrix of the CRF model. ,in Add the number of initial state and final state tags to the BIOES tags of the sequence label; Let be the probability of moving from the i-th label to the j-th label. The label sequence for sequence labeling is: The model then scores the sentence text X labeled with tag Y as follows: In the decoding stage, the Viterbi algorithm is used to decode and obtain the optimal predicted label sequence, which is then labeled by BIOES to obtain the corresponding entities. .

6. The entity relation joint extraction system based on triplet forest as described in claim 5, characterized in that, This interaction information extraction module is used to automatically learn the interaction information between entities and the interaction information between entities and input text through the multi-head attention mechanism in the Transformer Decoder module. Subsequently, a triple tree with overlapping heads is generated from each entity. Entity vector representation It is a vector representation of the entity set E' generated by the sequence labeling module; During training, E' = GE|E, where GE is the actual set of labeled entities, E is the set of entities predicted by the sequence labeling module, and | represents set merging; When making predictions, E' = E; ,in , Here are the head and tail indices for the i-th entity; entity In the implicit representation of text Vector representation of corresponding subwords The aggregation algorithm aggre yields entity vector representations. n+1 entity vectors are combined into a matrix : Entity matrix As input to the Transformer Decoder module, the Transformer Decoder module internally consists of an N-layer Transformer structure. Each layer contains a multi-head self-attention mechanism module and a multi-head encoder-decoder attention mechanism module. The multi-head self-attention mechanism module is used to learn the relationships between entities, while the multi-head encoder-decoder attention mechanism module is used to learn the relationships between sentence text and entities, obtaining entity-specific contextual information. The entity matrix is ​​obtained through the following formula: and sentence text representation After interacting with the Transformer Decoder module, the hidden layer units containing entity information and sentence text information are obtained. , ; This entity relationship extraction module is used to extract relationships based on hidden layer units. Generate a triplet forest, which consists of n+1 spanning tree modules, and each spanning tree module is a Tree-RNN; the input of the i-th Tree-RNN is the entity corresponding to that position. Hidden units of the Transformer Decoder module The j-th branch of this Tree-RNN is a triple with the i-th entity as the head entity. The initial hidden state of Tree-RNN is Its memory cells are initialized to The initial output is The input at each subsequent time step is a vector representation of the prediction result from the previous time step; if the prediction is an entity... Then input Vector representation of the entity If it is a relationship Then input Vector representation of relations The specific calculations for each time step are as follows, where the hidden state, memory unit, and input for each time step are: Time step : For each time step, the output of the LSTM, if a relation is predicted at that time step, is passed through a linear layer and then subjected to a sigmoid function to obtain the probability of each relation. The formula is as follows, where, ; If the entity is predicted at the current time step, it goes through a linear layer and is further fused with sentence text context information. The probability of selecting each entity is obtained by using the sigmoid activation function. This corresponds to the predicted entity tail during sequence labeling, where the weights of the linear layer are... , , : 。 7. The entity relation joint extraction system based on triple forest as described in claim 6, characterized in that, This entity relationship extraction module is used for: During the training phase, at each time step for entity and relation prediction, negative samples are obtained by negative sampling for each sentence. Let its label be NA, and the positive example be... merged into Then, multiple classifications are performed together; the entity and relation predictions are multiple binary classifications. The loss for constructing the triplet forest module is the sum of losses at each time step: #imgpt110# #imgpt111# L is the final loss, used to jointly train the BERT model, the CRF model, the Transformer Decoder module, and the triplet forest.

8. The entity relation joint extraction system based on triplet forest as described in claim 5, characterized in that, The entity relationship extraction module is used to construct or enrich a knowledge graph based on the entity relationship triples, for use in question answering models or search recommendation systems.

9. A storage medium for storing a program that performs joint entity relation extraction based on a triple forest as described in any one of claims 1 to 4.

10. A client for the entity relation joint extraction system based on triple forest as described in any one of claims 5 to 8.

Images