A joint entity recognition and relation extraction method based on hierarchical encoding

Abstract

The present invention relates to the field of natural language processing technology, and in particular to a joint entity recognition and relationship extraction method based on hierarchical encoding, comprising adopting a pre-trained language model BERT to encode word vectors; constructing a gating mechanism similar to LSTM; constructing an entity recognition task feature vector and a relationship extraction task feature vector; passing the entity recognition task feature vector and the relationship extraction task feature vector through a linear layer respectively to perform entity recognition and relationship extraction score calculation; summing the binary cross entropy loss, the binary cross entropy loss, and the relationship symmetry loss according to coefficients to obtain a total loss; and comparing the predicted score of the entity type and the predicted score of the relationship type 1 between the subject and the object with a threshold value to obtain a final triple. The present invention improves the problems of weak interaction and one-way interaction in current joint extraction, and solves the problem of being unable to recognize entity nesting and relationship nesting in text information.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a joint entity recognition and relationship extraction method based on hierarchical coding. Background Art

[0002] Entity recognition (NER) and relation extraction (RE) are important tasks of information extraction in natural language processing. There are two main solutions: joint extraction and step-by-step extraction. Compared with the step-by-step extraction method, the joint extraction method effectively alleviates the problem of error propagation and takes into account the close connection between entity recognition and relation extraction tasks.

[0003] However, the interactions used in most current joint extraction methods are still weak and one-way. For example, the text word vector is first converted into the entity recognition task vector, and then the relation extraction task feature vector is constructed. This does not take into account the impact of the relation task signal on entity recognition.

[0004] In addition, there may be nested entities and nested relationships in the text information. The span-based marking method marks the head and tail information of the text, allowing all overlapping entities / nested relationships to be listed in a table form. Summary of the Invention

[0005] In response to the shortcomings of existing algorithms, the present invention takes into account the certain connection between the outputs of the entity recognition model and the relationship extraction model, and adopts a joint extraction method to identify the semantic relationship between entities and entity pairs; considering the interaction between the feature encoding ends of the entity recognition and relationship extraction tasks, a gating mechanism similar to LSTM and an activation function cumsoftmax are proposed to divide and update the entity recognition and relationship extraction task feature vectors; in addition, based on the classification form of table filling, the learning of global features is strengthened and the problems of entity overlap and relationship nesting can be effectively solved; using entity recognition and relationship extraction methods, structured triples can be accurately and efficiently identified from unstructured text.

[0006] The technical solution adopted by the present invention is: a joint entity recognition and relationship extraction method based on hierarchical coding includes the following steps:

[0007] Step 1: Use the pre-trained language model BERT to encode word vectors;

[0008] Step 2: Build a gating mechanism similar to LSTM;

[0009] Further, it specifically includes:

[0010] Step 21, initialize the hidden layer vector h0 and cell unit vector c0 at the 0th moment in the LSTM model structure;

[0011] Step 22: Fusion word vector x at time t t and the hidden layer vector h at time t-1 t-1 , the entity gates at time t-1 and time t are obtained through the cumsoftmax activation function and relationship gate The calculation formula is:

[0012]

[0013]

[0014]

[0015]

[0016] Among them, W m / n , U m / n , b m / n are the trainable parameters of the model.

[0017] Furthermore, cumsoftmax activation means that the vector is first softmaxed and then each value in the vector is accumulated. The formula is:

[0018] d1′, d2′, ..., d n ′=softmax(d1,d2,...,d n )

[0019] cumsum(d1′,d2′,...,d n ′)=d1′,d1′+d2′,...,d1′+d2′+...+d n '

[0020] Among them, d1, d2, ..., d n is a vector value.

[0021] Step 3: Construct the entity recognition task feature vector and the relationship extraction task feature vector, and calculate the cell unit vector and hidden layer vector at the next moment;

[0022] Further, it specifically includes:

[0023] Step 31: Fusion word vector x at time t t and the hidden layer vector h at time t-1 t-1 , the cell intermediate vector c at time t is obtained through the activation function tanh t ', the calculation formula is:

[0024] c t '=tanh(W c xt +U c h t-1 +b c )

[0025] Among them, W c , U c , b c are the trainable parameters of the model;

[0026] Step 32: Construct entity recognition task feature vector h ner , the calculation formula is:

[0027]

[0028] Step 33: Construct the relation extraction task feature vector h re , the calculation formula is:

[0029]

[0030] Step 34: Fusion ner and h re The vector is passed through the linear layer to calculate the cell vector c at the current time t t , c t After the activation function tanh, h is obtained t , c t and h t Then as the input of the layered encoder at the next moment, the calculation formula is:

[0031] c t =w ner h ner +w re h re +b t

[0032] h t =tanh(c t )

[0033] Among them, w ner , w re , b t are the trainable parameters of the model.

[0034] Step 4: Pass the obtained entity recognition task feature vector and relationship extraction task feature vector through the linear layer respectively, and then perform the entity recognition and relationship extraction score calculation in the form of table filling;

[0035] Furthermore, the form of table filling uses a biaffine classifier model for score calculation.

[0036] Furthermore, step four specifically includes:

[0037] Step 41: For entity recognition tasks, h ner After the linear layer and Get the entity's head vector representation h es and the entity's tail vector representation h ee , the calculation formula is:

[0038]

[0039]

[0040] Step 42: The head vector is represented as h es and the tail vector represents h ee The biaffine classifier for the entity recognition task is fed into the table. The i-th row represents the beginning of the entity span, and the j-th column represents the end of the entity span. Each table represents an entity category k. The predicted score of entity type k is obtained through the biaffine classifier model and the activation function sigmoid. The calculation formula is:

[0041]

[0042] Among them, U e and W e Represents the trainable parameters of the model, k represents the type of entity, and ε represents the set of entity types;

[0043] Step 43: Calculate the binary cross entropy loss for the entity recognition task The calculation formula is:

[0044]

[0045] Where N represents the number of samples, y ij If it belongs to the current category, it is 1, otherwise it is 0;

[0046] Step 44: For the relation extraction task, similarly set h re After the linear layer and Get the head vector representation h of the subject entity ss and the head vector representation h of the object entity os , the calculation formula is:

[0047]

[0048]

[0049] Step 45: The subject's head vector is represented as h ssand the object's head vector representation h os The table classifier for the relation extraction task is fed into the table. The ath row in the table represents the beginning of the subject, and the bth column represents the beginning of the object. Each table represents a type of relation. The predicted score for the relation type l between subject a and object b is obtained through the biaffine classifier model and the activation function sigmoid. The calculation formula is:

[0050]

[0051] Among them, U r and W r represents the trainable parameters, l represents the type of relationship, Represented as a collection of relation types;

[0052] Step 46: Calculate the binary cross entropy loss for the relation extraction task (RE) The calculation formula is:

[0053]

[0054] Where N represents the number of samples, y ab If it belongs to the current category, it is 1, otherwise it is 0.

[0055] Step 47: Add relational symmetry loss of learning, The relationship between the subject and the object should be symmetrical about the diagonal of the table. The calculation formula is:

[0056]

[0057] Where N represents the number of samples, abs(x,y)=|xy|, is the predicted score of the relationship type between subjects b and a in the reverse relationship.

[0058] Step 5: Binary cross entropy loss of entity recognition task Binary cross entropy loss for relation extraction tasks and relational symmetric loss Add the coefficients to get the total loss Through loss Train the model;

[0059] Furthermore, the total loss The formula is:

[0060]

[0061] Among them, α, β, and γ represent weight ratios.

[0062] Step 6: Compare the predicted score of the entity type and the predicted score of the relationship type l between the subject and the object with the threshold to obtain the final triple.

[0063] Beneficial effects of the present invention:

[0064] 1. Utilizing a gating mechanism similar to LSTM, the cell state vector is divided into entity and relationship levels. By continuously updating the cell state during model training, the encoding of the feature vectors for entity recognition and relationship extraction tasks interacts continuously, achieving mutual assistance between entity recognition and relationship extraction.

[0065] 2. Design the cumsoftmax function to reorder the variables within the vector, thereby achieving the division of feature vectors for entity recognition and relationship extraction tasks;

[0066] 3. The classification method based on table filling strengthens the learning of global features and effectively solves the problems of entity overlap and relationship nesting;

[0067] 4. Add learning of symmetric labeling tasks to further improve the performance of model recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of the hierarchical coding-based joint entity recognition and relationship extraction method of the present invention;

[0069] Figure 2 It is a flow chart of the hierarchical encoder of the present invention;

[0070] Figure 3 Schematic diagram of the biaffine classifier model of the present invention;

[0071] Figure 4 It is a practical diagram of the joint extraction process of the present invention. DETAILED DESCRIPTION

[0072] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore only shows the components related to the present invention.

[0073] like Figure 1 As shown, a joint entity recognition and relationship extraction method based on hierarchical coding includes the following steps:

[0074] Step 1: Use the pre-trained language model BERT to encode the word vector, and output the text word vector sequence {x1, x2, x i ..., xn}; where x i It is represented as the word vector of the i-th word after the text sentence is segmented;

[0075] Step 2: Figure 2 As shown, a gating mechanism similar to LSTM is used to perform entity gate g e and relationship gate r In the LSTM structure, each step contains the cell state vector c at time t-1 t-1 , the hidden state vector h at time t-1 t-1 and the current t moment (the i-th word vector in the word vector sequence) x t Operation; where the entity gate g e and relationship gate r , including a physical gate for updating the cell vector at time t-1 The relation gate used to update the cell vector at time t-1 The entity gate used to update the cell vector at time t and the relation gate used to update the cell vector at time t

[0076] Specifically include:

[0077] Step 21: Initialize the hidden state vector h0 and cell state vector c0 at the 0th moment in the LSTM model structure to vectors of all zeros.

[0078] Step 22: Fusion word vector x at time t (i-th in the sequence) t and the hidden layer vector h at time t-1 t-1 , after the cumsoftmax activation function, the entity gate at time t-1 and time t are obtained and relationship gate The construction of is calculated as follows:

[0079]

[0080]

[0081]

[0082]

[0083] Among them, W m / n , U m / n , b m / nis a trainable parameter of the model, and the cumsoftmax function represents the vector D(d1~d n To form a vector), first perform softmax, and then perform accumulation operation on each value in the vector, that is:

[0084] d1′, d2′, ..., d n ′=softmax(d1,d2,...,d n )

[0085] cumsum(d1′,d2′,...,d n ′)=d1′,d1′+d2′,...,d1′+d2′+...+d n '

[0086] The encoding vectors generated using the traditional LSTM gating mechanism cannot be divided into the feature vectors for entity recognition and relation extraction tasks. Introducing a custom cumsoftmax function reorders the variables within the vectors, thereby enabling the division of the feature vectors for entity recognition and relation extraction tasks. By continuously updating the variables during model training, the encodings for the two tasks interact, achieving mutual support between the two tasks.

[0087] Step 3: Construct entity recognition task feature vector h ner And the relation extraction task feature vector h re , and calculate the cell vector c at the next moment t and the hidden layer vector h t ;

[0088] Specifically include:

[0089] Step 31: First, fuse the word vector x at time t t and the hidden layer vector h at time t-1 t-1 , the cell intermediate vector c at time t is obtained through the activation function tanh t ', the calculation formula is:

[0090] c t '=tanh(W c x t +U c h t-1 +b c )

[0091] Among them, W c , U c , b c are the trainable parameters of the model.

[0092] Step 32: Construct entity recognition task feature vector hner , the calculation formula is:

[0093]

[0094] Step 33: Construct the relation extraction task feature vector h re , the calculation formula is:

[0095]

[0096] Step 34: Fusion ner and h re The vector is passed through the linear layer to calculate the cell vector c at the current time t t , c t After the activation function tanh, the hidden layer vector h at the tth moment is obtained t , c t and h t Then as the input of the layered encoder at the next moment, the calculation formula is:

[0097] c t =w ner h ner +w re h re +b t

[0098] h t =tanh(c t )

[0099] Among them, w ner , w re , b t are the trainable parameters of the model.

[0100] Step 4: Get the h ner 、h re After the linear layer Linear e and Linear r Then, the scoring calculation of entity recognition and relationship extraction is performed in the form of table filling. The scoring calculation is performed using the biaffine classifier model. The biaffine classifier model is as follows Figure 3 As shown;

[0101] Step 41: For entity recognition tasks, h ner After the linear layer and Get the entity's head vector representation h es and the entity's tail vector representation h ee , the calculation formula is:

[0102]

[0103]

[0104] Step 42: The head vector is represented as h es and the tail vector represents h ee The biaffine classifier for entity recognition (NER) is used. The i-th row in the classifier table represents the beginning of the entity span, and the j-th column represents the end of the entity span. Each table represents a category k of an entity. The predicted score of entity type k is obtained through the biaffine classifier model and the activation function sigmoid. The calculation formula is:

[0105]

[0106] Among them, U e and W e Represents the trainable parameters of the model, k represents the type of entity, and ε represents the set of entity types;

[0107] Step 43: Calculate the binary cross entropy loss for the entity recognition task (NER) The calculation formula is:

[0108]

[0109] Where N represents the number of samples, y ij If it belongs to the current category, it is 1, otherwise it is 0.

[0110] Step 44: For the relation extraction task (RE), similarly, h re After the linear layer and Get the head vector representation h of the subject entity ss and the head vector representation h of the object entity os , the calculation formula is:

[0111]

[0112]

[0113] Step 45: The subject's head vector is represented as h ss and the object's head vector representation h os The table classifier for the relation extraction task (RE) is fed into the table. The ath row in the table represents the beginning of the subject, and the bth column represents the beginning of the object. Each table represents a type of relationship. The predicted score of the relationship type l between subject a and object b is obtained through the biaffine classifier model and the activation function sigmoid. The calculation formula is:

[0114]

[0115] Among them, U r and W r represents the trainable parameters, l represents the relationship type, Represented as a collection of relation types;

[0116] Step 46: Calculate the binary cross entropy loss for the relation extraction task (RE) The calculation formula is:

[0117]

[0118] Where N represents the number of samples, y ab If it belongs to the current category, it is 1, otherwise it is 0.

[0119] Step 47: Add relational symmetry loss of learning, The relationship between the subject and the object should be symmetrical about the diagonal of the table. The calculation formula is:

[0120]

[0121] Where N represents the number of samples and abs(x,y)=|xy|.

[0122] Step 5: The loss obtained in step 43 The loss obtained in step 46 and the loss obtained in step 47 Add the coefficients to get the total loss Through loss Train the model;

[0123] Total loss The formula is:

[0124]

[0125] Among them, α, β, and γ represent weight ratios, which are set to 1 in this experiment.

[0126] Step 6: Save the model with the best performance on the validation set, test it on the sample to be predicted, and get the final test result. Threshold filter the result obtained in step 42. If the result is greater than 0.3, it means that the predicted entity segment ij belongs to a certain entity type. Threshold filter the result obtained in step 45. If the result is greater than 0.3, it means that the predicted subject and object a and b belong to a certain relationship label. Combine the results of entity recognition and relationship extraction tasks after threshold filtering to get the final triple.<s,r,o> .

[0127] like Figure 4 For example, for the text "Rome is in Lazio province", the word vector of each word is obtained through the pre-trained language model BERT, and then the task feature vector h of entity recognition extraction is obtained through the designed encoder. ner And the relation extraction task feature vector h re ,The feature vector is passed through the table classifier to obtain the final recognition score, which is above a certain threshold indicating that a specific entity or relationship is identified.

[0128] Table 1 is a comparison of the experimental results of the proposed model and the existing model on the SciERC dataset.

[0129]

[0130] Table 1 shows the experimental results of the proposed model on the public dataset SciERC and compares the experimental results of some of the latest models. The experimental results include the precision (Prec.), recall (Rec.) and F1 value of entity recognition (NER) and relation extraction (RE); RE indicates that the entity span is correct and the relationship type is correct, and RE+ indicates that the entity span and type are correct and the relationship type is also correct; the bold part is the best experimental result after comparison.

[0131] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A joint entity recognition and relationship extraction method based on hierarchical coding, characterized in that: The following steps are involved: Step 1: Use the pre-trained language model BERT to encode word vectors and output the text word vector sequence through the BERT model; Step 2: Build a gating mechanism similar to LSTM; Step 2 specifically includes: Step 21, initialize the hidden layer vector h0 and cell unit vector c0 at the 0th moment in the LSTM model structure; Step 22: Fusion word vector x at time t t and the hidden layer vector h at time t-1 t-1 , the entity gates at time t-1 and time t are obtained through the cumsoftmax activation function and relationship gate The calculation formula is: Among them, W m / n , U m / n , b m / n are the trainable parameters of the model; Step 3: Construct the entity recognition task feature vector and the relationship extraction task feature vector, and calculate the cell unit vector and hidden layer vector at the next moment; Step three specifically includes: Step 31: Fusion word vector x at time t t and the hidden layer vector h at time t-1 t-1 , the cell intermediate vector c at time t is obtained through the activation function tanh t ', the calculation formula is: c t '=tanh(W c x t +U c h t-1 +b c ) Among them, W c , U c , b c are the trainable parameters of the model; Step 32: Construct entity recognition task feature vector h ner , the calculation formula is: Step 33: Construct the relation extraction task feature vector h re , the calculation formula is: Step 34: Fusion ner and h re The vector is passed through the linear layer to calculate the cell vector c at the current time t t , c t After the activation function tanh, the hidden layer vector h at the tth moment is obtained t , c t and h t Then as the input of the layered encoder at the next moment, the calculation formula is: c t =w ner h ner +w re h re +b t h t =tanh(c t ) Among them, w ner , w re , b t are the trainable parameters of the model; Step 4: Pass the obtained entity recognition task feature vector and relationship extraction task feature vector through the linear layer respectively, and then perform entity recognition and relationship extraction score calculation in the form of table filling; Step 5: Add the binary cross entropy loss of the entity recognition task, the binary cross entropy loss of the relationship extraction task, and the relationship symmetry loss according to the coefficients to obtain the total loss, and train the model through the loss; Step 6: Compare the predicted score of the entity type and the predicted score of the relationship type l between the subject and the object with the threshold to obtain the final triple.

2. The hierarchical coding-based joint entity recognition and relationship extraction method according to claim 1, characterized in that: The cumsoftmax activation means that the vector is first softmaxed and then each value in the vector is accumulated. The formula is: d1',d2',...,d n '=softmax(d1,d2,...,d n ) cumsum(d1',d2',...,d n ')=d1',d1'+d2',...,d1'+d2'+...+d n ' Among them, d1, d2, ..., d n is the vector value of vector D.

3. The hierarchical coding-based joint entity recognition and relationship extraction method according to claim 1, characterized in that: The form of table filling is to use the biaffine classifier model for score calculation.

4. The method for joint entity recognition and relationship extraction based on hierarchical coding according to claim 1, characterized in that: Step 4 specifically includes: Step 41: For entity recognition tasks, h ner After the linear layer and Get the entity's head vector representation h es and the entity's tail vector representation h ee , the calculation formula is: Step 42: The head vector is represented as h es and the tail vector represents h ee The biaffine classifier is sent to the entity recognition task; the prediction score of entity type k is obtained through the biaffine classifier model and the activation function sigmoid. The calculation formula is: Among them, U e and W e represents the model training parameters, k is the type of entity, and ε is the set of entity types; Step 43: Calculate the binary cross entropy loss for the entity recognition task The calculation formula is: Where N represents the number of samples, y ij Indicates whether it belongs to the current category; Step 44: For the relation extraction task, similarly set h re After the linear layer and Get the head vector representation h of the subject entity ss and the head vector representation h of the object entity os , the calculation formula is: Step 45: The subject's head vector is represented as h ss and the object's head vector representation h os Send it to the relation extraction task table classifier; get the predicted score of the relation type l between the subject and the object through the biaffine classifier model and the activation function sigmoid The calculation formula is: Among them, U r and W r is the training parameter, l is the type of relationship, is a collection of relationship types; Step 46: Calculate the binary cross entropy loss for the relationship extraction task The calculation formula is: Where N represents the number of samples, y ab Indicates whether it belongs to the current category; Step 47: Add relational symmetry loss of learning, The relationship between the subject and the object that indicates symmetry should be symmetrical about the diagonal of the table. The calculation formula is: Among them, N represents the number of samples, l is the relationship type, is the predicted score for the relationship type l between subject b and a in the reverse relationship, The predicted score when the relationship type between subject b and a is l.

5. The hierarchical coding-based joint entity recognition and relationship extraction method according to claim 4, characterized in that: Total loss The formula is: Among them, α, β, and γ represent weight ratios.

Images