An entity relation extraction method based on pre-judgment on word sequence and multi-round classification

Abstract

An entity relation extraction method based on pre-judgment of token sequence and multi-round classification, a model for predicting text sentences at a token sequence, i.e., span level, is used to find all entities and the relations between entities in the text sentences. The model uses a BERT pre-training model and simultaneously includes three modules, i.e., pre-judgment (PEJ), entity multi-round classification (EMR) and relation multi-round classification (RMR). Through the preliminary judgment of the PEJ module and the multi-round entity classification of the EMR module, entity recognition is performed, and then the RMR module is used for multi-round relation classification to determine the relations between entity pairs, thereby realizing relation extraction. The multi-round classification in the Smrc model enables the data set to be used multiple times and sufficiently, the model is better fitted, multiple classifiers directly determine the relations between entities and entity pairs in multiple categories, the problem of imbalance and large difference in discrimination ability of a single multi-output classifier in different categories is avoided, the model structure is more refined, and the entity pre-judgment module (PEJ) is added, so that the Smrc model has more accurate recognition effect.

Description

TECHNICAL FIELD

[0001] The application is applied to the entity recognition and relation extraction tasks in the field of natural language processing, and proposes a method for utilizing pre-judgment and entity relation multi-round classification to predict a model Smrc on a token sequence, i.e. a span level, to find all entities and the relations between the entities in a text sentence. BACKGROUND

[0002] In recent years, with the emergence of pre-training language models represented by ELMo, GPT, BERT, etc., new breakthroughs have been made in many tasks in the field of natural language processing. The BERT pre-training model comes from the bidirectional encoder representation of the Transformers model. BERT is pre-trained on a large corpus of text sequences to obtain the vector representation of each token in the text. The trained BERT model can be transferred to downstream tasks with less training data. In the Smrc model of the application, the BERT representation of each token in a sentence will be input into a decoder specific to the entity recognition and relation extraction task, and the BERT model parameters will be fine-tuned through backpropagation.

[0003] Entity recognition and relation extraction, which is the core task of information extraction, is to extract entities and the relations between entities from unstructured text, and specifically includes two sub-tasks of named entity recognition and relation extraction. For a given sentence: “Rome is in Lazio province and Naples in Campania”, the goal of entity recognition is to find all entities in the sentence: “Rome”, “Lazio”, “Naples”, “Campania”, and the goal of relation extraction is to find all entity-relation triples in the sentence: (“Rome”, Located-in, “Lazio”), (“Naples”, Located-in, “Campania”). Where “Located-in” represents the relationship type.

[0004] Early work often uses a pipeline method to handle the two sub-tasks separately, and the current joint learning method has become the mainstream. Compared with the pipeline method, the joint learning method uses the close interaction information between entities and relations to identify entities and extract relations between entities at the same time, which improves the error accumulation and propagation problems existing in the pipeline method. According to the different modeling levels, joint learning is divided into parameter sharing and sequence labeling. Parameter sharing models the two sub-tasks separately, while sequence labeling models them together.

[0005] The sequence labeling-based method sets only one label for each word element, and cannot identify overlapping entities, for example: "lithium toxicity", "lithium" all belong to entities, but because the sequence labeling method only assigns a label to the word element "lithium", it cannot identify the two entities "lithium toxicity" and "lithium" at the same time.

[0006] Recently, some span-based joint learning methods have been proposed to explore entities and relationships at the span level. "Lithium toxicity" and "lithium" belong to different spans, so they have a natural advantage in identifying overlapping entities. SUMMARY

[0007] The Smrc model proposed in the present application adopts a parameter sharing method, uses BERT as an encoder, and realizes mutual dependence by jointly learning two sub-tasks of named entity recognition and relationship extraction through a shared encoder. Considering that in the past span-based models, a multi-output decoder is often used to predict and classify spans, the classification ability of the classifier on each category is not considered, and the training data is not fully utilized, therefore in the Smrc model proposed in the present application, the entity and relationship prediction of span is realized through multi-round classification, which solves the above problems. In addition, by pre-judging the entity before multi-round classification, the accuracy of entity and relationship identification is further improved.

[0008] The method adopted by the present application to solve the technical problems is as follows:

[0009] An entity relationship extraction method based on pre-judgment and multi-round classification of word element sequences, comprising the following steps:

[0010] Step (1): using a BERT pre-training language model to process the input sentence X to obtain a vector sequence: (CLS, X1, X2, …, X n ), wherein CLS represents the global information of X. For a span s with a width of L, the BERT vector representation is: (X i ,X i+1 ,…,X i+L-1 ).

[0011] The span vector representation is an encoder module: using a BERT pre-training language model to obtain the vector representation of each candidate span in the input sentence.

[0012] Step (2): input the input information representation corresponding to each span into the sigmoid classifier in the entity pre-judgment module (PEJ) to judge whether the span belongs to an entity. For a span s with a width of L, the input information representation of the span s on the sigmoid classifier is composed of three parts. The first part is the maximum pooling of the BERT vector representation (X i ,X i+1 ,…,X i+L-1 ) of the span s, denoted as i1(s):

[0013] Bert(s)=(X i ,X i+1 ,…,X i+L-1 )

[0014] i1(s)=maxpool(Bert(s))

[0015] Considering the effect of span width features on the model, a width embedding matrix is used, denoted as WE. There is a vector representation of the same dimension corresponding to each different span width (1, 2,...) in WE. WE is optimized by back propagation. The embedding vector WE L is the second part of the input information representation, denoted as i2(s): i2(s)=WE L ;

[0016] The global information CLS plays an important role in entity recognition and is an important source of information for entity recognition. Some key word information contained in the CLS may have a strong directional effect on the judgment of specific categories. Therefore, the CLS is used as the third part of the input information representation, denoted as i3(s): i3(s)=CLS;

[0017] The three parts of the input information representation are spliced, denoted as i(s): i(s)=i1(s).i2(s).i3(s);

[0018] where the symbol [.] represents vector splicing combination.

[0019] Entity pre-judgment module (PEJ): judge whether each span meeting the length limit in the input sentence belongs to an entity, and retain the span judged to belong to an entity.

[0020] i(s) is input into the PEJ module for pre-judgment:

[0021] p(s)=σ(W e *i(s)+b e )

[0022] where the symbol * represents matrix multiplication (as below), W e and b erespectively, and σ denotes the sigmoid function. p(s) is the predicted value of s on ec0. A high predicted value indicates that s is likely to belong to the entity. Given a threshold value a, p(s) > a indicates that s belongs to the entity, otherwise it does not belong.

[0023] Step (3): For the span s determined as an entity in the previous step, input its corresponding input information representation (synchronized with step (2)) into the entity multi-round classification module (EMR) to make further entity judgment and specific entity category judgment for each entity category classifier.

[0024] Entity multi-round classification module (EMR): After the PEJ module preliminarily determines whether the span belongs to the entity, further judgment is made on the remaining spans. Specifically, it is determined whether the span belongs to each entity category predefined in the experimental set. If the span does not belong to any entity category, it is considered that the span does not belong to the entity, and the span is excluded if it does not belong. Otherwise, the span and its corresponding one or more entity categories are retained.

[0025] After the PEJ module preliminarily determines whether the span belongs to the entity, further judgment is made on the remaining spans. Specifically, it is determined whether the span belongs to each entity category predefined in the experimental set. If the span does not belong to any entity category, it is considered that the span does not belong to the entity, and the span is excluded if it does not belong. Otherwise, the span and its corresponding one or more entity categories are retained.

[0026] Suppose that the pre-defined data set contains 4 types of entities and 5 types of relationships. On the classifiers ec1, ec2, ec3, and ec4 corresponding to the 4 types of entities, respectively, it is determined whether the span s belongs to the corresponding entity categories e1, e2, e3, and e4. If it does not belong to any category, the span s is excluded. Each classifier makes a prediction on the span s:

[0027]

[0028]

[0029]

[0030]

[0031] wherein, denote the weights and biases on ec1, ec2, ec3, ec4, respectively. p1(s), p2(s), p3(s), p4(s) denote the predicted two-dimensional vector on each category, and the two values represent the probabilities that span s does not belong to, respectively, belongs to each corresponding category. The spans that are judged to not belong to any category are filtered out, and the remaining spans and their corresponding one or more categories are retained. S pre denotes the set of retained spans.

[0032] Step (4):

[0033] Relationship multi-round classification module RMR: The entity set filtered out in the entity multi-round classification module EMR, each pair of entities is in turn judged whether there is each relationship category predefined in the experimental set, if there is no any relationship category between the entity pair, it is considered that there is no relationship between the entity pair, otherwise the entity pair, the entity category of the two entities in the entity pair and one or more relationship categories existing are retained.

[0034] The multi-round relationship classification module RMR performs relationship extraction on the set S pre ×S pre obtained in the previous step, (assuming the number of relationship categories in the dataset is 5) The classifiers rc1, rc2, rc3, rc4, rc5 corresponding to the 5 relationship categories in the RMR module respectively judge whether the entity pair (s1, s2) in S pre ×S pre exists the corresponding relationship r1, r2, …, r5. The input information of each classifier contains three parts of information, the first part is the maximum pooling i1(s1), i1(s2) of the BERT vector representation of the entity s1, s2 (same as the PEJ and EMR module), the second part is the width embedding information i2(s1), i2(s2) of the two entities (same as the PEJ and EMR module), and the third part is the context information of the entity pair (s1, s2) in the sentence. The model uses the maximum pooling of the BERT representation of the span between the entities s1, s2 (between the end position of s1 and the beginning position of s2) as the third part of the input, denoted as c(s1, s2). Finally, the three parts of information are spliced:

[0035] i(s1, s2) = i1(s1)·i1(s2)·i2(s1)·i2(s2)·c(s1, s2)

[0036] Then each relationship classifier predicts the relationship between the entity pair (s1, s2):

[0037]

[0038]

[0039]

[0040]

[0041]

[0042] respectively represent the weight and bias on each classifier. p1(s1, s2), p2(s1, s2), p3(s1, s2), p4(s1, s2), p5(s1, s2) respectively represent the two-dimensional vector output by (s1, s2) on each classifier, the two values respectively represent the probability that the entity pair (s1, s2) does not exist, and the probability that the corresponding category relationship exists, and the relationship set R is predicted pre .

[0043] Through the above steps, the entity set S pre in the text sentence can be recognized and the relationship set R pre is extracted.

[0044] Each sentence in the data set for model training contains a sentence, an entity, and a relationship, where the entity is represented as [start position index, end position index, entity type], and the relationship is represented as [head entity index, tail entity index, relationship type]. The model loss is the sum of the losses of each classifier in the PEJ, EMR, and RMR modules. The classifier ec0 in the PEJ module is a sigmoid classifier, so the binary-cross-entropy-loss is used to calculate the loss. The classifiers in the EMR and RMR modules are all softmax classifiers, so the cross-entropy-loss is used to calculate the loss. The total loss of the Smrc model is represented as L. respectively represent the loss of the model classifiers ec0, ec1, ec2, ec3, ec4, rc1, rc2, rc3, rc4, and rc5 on the training data. L PEJ , L EMR , L RMR respectively represent the loss of the PEJ, EMR, and RMR modules as a whole, so we have:

[0045]

[0046]

[0047]

[0048] L = L PEJ + L EMR + L RMR

[0049] For the selection of training data of each classifier in PEJ and EMR module, assume that the set of entities with labels in sentence X is S t :{(s1, e1), (s2, e2), (s3, e3), (s4, e4)} as training positive examples, and randomly select ce = 100 spans that do not belong to entities among all spans that meet the width limit: (f1, none), (f2, none), …, (f ce , none).

[0050] The selection of relationship training negative examples in RMR module is different from the random selection in PEJ and EMR. From the free combination of entities with entity labels S t × S t in the sentence, assume that the set of entities in the sentence is S t : {“Franca Chlistovsky”, “Brera metereologic institute”, “Milan”, “1950”}, the relationship contained is: [Franca Chlistovsky, Brera metereologic institute, Work for], where Franca Chlistovsky, Brera metereologic institute are two entities contained in the relationship, and Work for represents the entity relationship type. Then select (“Franca Chlistovsky”, “Milan”, none), (“Brera metereologic institute”, “1950”, none), …, (“Milan”, “1950”, none) and other entity pairs that do not have a relationship between each other as relationship extraction training negative examples.

[0051] The application is directed to the entity recognition and relation extraction tasks in the field of natural language processing, and proposes a model for predicting at the span level: Smrc. The model uses the BERT pre-training model, and simultaneously contains three modules: preliminary judgment (PEJ), entity multi-round classification (EMR), and relation multi-round classification (RMR). Through the preliminary judgment of entities by the PEJ module and the multi-round entity classification of the EMR module, entity recognition is performed, and then the RMR module is used for multi-round relation classification to determine the relation between entity pairs, so as to realize relation extraction. The multi-round classification in the Smrc model makes the data set be used multiple times and sufficiently, and the model is better fitted. Multiple classifiers directly determine the entity and the relation between entity pairs in multiple categories, avoiding the problem of unbalanced and large difference in discrimination ability of a single multi-output classifier in different categories, and the model structure is more detailed. In addition, the entity preliminary judgment module (PEJ) makes the recognition effect of the Smrc model more accurate. Compared with the previous work, the Smrc model has obviously improved the effect in two sub-tasks on the CoNLL04, SciERC and ADE three experimental data sets. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A process overview diagram for the Smrc model described in the application to perform entity recognition and relation extraction on an input example sentence.

[0053] Figure 2 An overview diagram of the BERT model used in the Smrc model described in the application.

[0054] Figure 3 A process overview diagram of the entity preliminary judgment module PEJ in the Smrc model described in the application to process an input example sentence.

[0055] Figure 4 A process overview diagram of the entity multi-round judgment module EMR in the Smrc model described in the application to further determine the entity category of each span preliminarily determined as an entity. Wherein [Location-classifier, Organization-classifier, People-classifier, Other-classifier] respectively represent the classifiers corresponding to the four entity categories [Location, Organization, People, Other] in the CoNLL04 data set.

[0056] Figure 5A process overview diagram of the relationship multi-round judgment module RMR in the Smrc model of the present application for performing relationship category judgment on entity pairs. Wherein [Work for-classifier, Kill-classifier, Organization based in-classifier, Live in-classifier, Located in-classifier] respectively represent the classifiers corresponding to the five relationship categories of [Work for, Kill, Organization based in, Live in, Located in]. Embodiments

[0057] The specific embodiments of the present application are described in detail below in combination with the drawings and technical solutions of the present application.

[0058] Figure 1 A process overview of the Smrc model of the present application for performing entity recognition and relationship extraction on an input example sentence is shown, Figure 2 The BERT pre-training model used in the present application is shown. From Figure 1 It can be seen that an input sentence: [However, Franca Chlistovsky, who heads the Brera metereologic institute in Milan] is first subjected to preliminary judgment by the entity pre-judgment module PEJ to identify entities: [Franca Chlistovsky], [Brera metereologic institute], and [Milan]. The PEJ overview diagram is shown in Figure 3 The BERT information of each candidate span is further processed and integrated as input to the sigmoid classifier in the PEJ to determine whether the candidate span belongs to an entity. The specific information processing and integration process is described in step (2) of the technical solutions of the present application.

[0059] The entities obtained through the preliminary judgment of the PEJ are processed by the entity multi-round judgment module EMR to screen the final recognized entities and their specific categories: [Franca Chlistovsky [People]], [Brera metereologic institute [Organization]], and [Milan [Location]]. The EMR overview is shown in Figure 4 The processed and integrated information of each entity (same as the PEJ module) is input to the classifier corresponding to each entity category in the EMR to determine whether it belongs to the corresponding entity category. Through the above process, the entity recognition task is completed.

[0060] Subsequently, the identified entities are freely combined, and input into a relationship multi-round judgment module RMR for judgment of the relationship category between the entities, so as to extract the relationship between the entities existing in the sentence: [Franca Chlistovsky [People], Brera metereologic institute [Organization], Work-for]. Among them, Franca Chlistovsky and Brera metereologic institute are two entities in the entity pair, [People] and [Organization] are the entity categories of the two entities respectively, and Work-for represents the relationship between the two entities. The overview of the relationship multi-round judgment module RMR is shown in Figure 5 The entity pair information is processed and integrated, and input into the classifier corresponding to each relationship category in the RMR module to judge whether the corresponding relationship exists between the entity pairs, wherein the entity pair information processing and integration process refers to step (4) in the technical solution of the present application. Thus, the model completes the entity recognition and relationship extraction in the sentence.

Claims

1. An entity relation extraction method based on pre-judgment of word sequence and multi-round classification, comprising the following steps: Step (1): The input sentence X is processed by using the BERT pre-training language model to obtain a vector sequence: (CLS, X1, X2, …, X n ), wherein, CLS represents global information of X; for a span s of width L, its BERT vector representation is: i , X i+1 , …, X i+L-1 ); span vector representation encoder module: using BERT pre-training language model to obtain vector representation of each candidate span in input sentence; Step (2): The input information representation corresponding to each span is input into a sigmoid classifier in the entity pre-judgment module (PEJ) to determine whether the span belongs to an entity. For a span s with a width of L, the input information representation of the sigmoid classifier is composed of three parts, the first part is the maximum pooling of the BERT vector representation (X i ,X i+1 ,…,X i+L-1 ) of the span s, denoted as i1(s): Bert(s) = (X i ,X i+1 ,…,X i+L-1 ) i1(s) = maxpool(Bert(s)); Considering the role of span width feature to the model, a width embedding matrix is used, denoted as WE; there is a vector of the same dimension in WE corresponding to different span widths, and WE is optimized by back propagation; the embedding vector WE L As the second part of the input information representation, denoted as i2(s): i2(s) = WE L ; The third part of the input information representation is CLS, denoted as i3(s): i3(s) = CLS; The three parts of the spliced input information representation are denoted as i(s): i(s) = i1(s) i2(s) i3(s) ; Where the symbol [.] represents vector splicing combination; Entity pre-judgment module (PEJ): judge whether each span meeting the length limit in the input sentence belongs to an entity, and retain the span judged to belong to an entity; i(s) input PEJ module for pre-judgment: p(s) = σ(W e i(s) + b e ) where the notation represents matrix multiplication, W e and b e represent the weights and bias of ec0, respectively, and σ represents the sigmoid function; p(s) is the predicted value of s on ec0; a high predicted value indicates that s is likely to belong to the entity; given a threshold value a, p(s) > a indicates that s belongs to the entity, otherwise it does not. Step (3): For the span s judged as an entity in the previous step, input the corresponding input information representation into the entity multi-round classification module (EMR), and further judge the entity category and the specific entity category corresponding to each entity category classifier; Entity multi-round classification module (EMR): after the PEJ module preliminarily judges whether the span belongs to an entity, further judgment is made on the retained span. Specifically, it is judged whether the span belongs to each entity category predefined in the experimental set. If the span does not belong to any entity category, it is considered that the span does not belong to an entity, and the span is excluded. Otherwise, the span and one or more entity categories corresponding to the span are retained. After the PEJ module preliminarily judges whether the span belongs to an entity, further judgment is made on the retained span. Specifically, it is judged whether the span belongs to each entity category predefined in the experimental set. If the span does not belong to any entity category, it is considered that the span does not belong to an entity, and the span is excluded. Otherwise, the span and one or more entity categories corresponding to the span are retained. If the pre-defined data set contains 4 types of entities and 5 types of relations, then on the classifiers ec1, ec2, ec3, ec4 corresponding to the 4 types of entities, it is judged whether the span s belongs to the corresponding entity categories e1, e2, e3, e4. If it does not belong to any category, the span is excluded. Each classifier predicts the span s: ； ； ； ； wherein, , , , , , , , respectively represent weights and biases on ec1, ec2, ec3, ec4; , , , represent a two-dimensional vector predicted on each category, two values respectively represent the probability that span s does not belong to, belongs to each corresponding category; the span judged as not belonging to any category is screened out, and the remaining span and its corresponding one or more categories are retained; S pre represents the set of retained spans; Step (4): Relation multi-round classification module RMR: the entity set filtered out in the entity multi-round classification module EMR, judges whether there is each relationship category predefined in the experimental set between each pair of entities in turn. If there is no relationship category between the entity pair, it is considered that there is no relationship between the entity pair. Otherwise, the entity pair, the entity categories of the two entities in the entity pair, and one or more relationship categories existing between the two entities are retained. A multi-round relation classification module RMR performs relation extraction on the entity pairs in the set S obtained in the previous step pre S pre , sets the number of relation categories in the data set to 5, and the classifiers rc1, rc2, rc3, rc4, and rc5 corresponding to the 5 relation categories in the RMR module respectively judge whether the entity pair (s1, s2) in S pre S pre exists the corresponding relation r1, r2, …, r5; the input information of each classifier contains three parts of information, the first part is the maximum pooling of the BERT vector representation of the entities s1 and s2 , the second part is the width embedding information of the two entities , and the third part is the context information of the entity pair (s1, s2) in the sentence, the model uses the maximum pooling of the BERT representation of the span between the end position of the entity s1 and the start position of the entity s2 as the third part of the input, denoted as c(s1, s2); finally, the three parts of information are spliced: ； Then each relation classifier predicts entity pairs ( The relationship between ) ； ； ； ； ； 、 、 、 、 、 、 、 、 、 respectively represent the weights and biases on each classifier; 、 、 、 、 respectively represent two-dimensional vectors output on each classifier, the two values respectively represent the probabilities that the entity pair does not exist, exists the corresponding category relationship, and the relationship set R pre is predicted. Through the above steps, the entity set S in the text sentence can be identified pre , and the relationship set R is extracted pre ; Each sentence in the data set for model training contains a sentence, an entity, and a relationship, wherein the entity is represented as [start position index, end position index, entity type], and the relationship is represented as [head entity index, tail entity index, relationship type]; the model loss is the sum of the losses of the classifiers in the PEJ, EMR, and RMR modules, the classifier ec0 in the PEJ module is a sigmoid classifier, so the binary-cross-entropy-loss is used to calculate the loss, and the classifiers in the EMR and RMR modules are all softmax classifiers, so the cross-entropy-loss is used to calculate the loss, and the total loss of the Smrc model is represented as respectively represent the losses of the training data on the model classifiers ec0, ec1, ec2, ec3, ec4, rc1, rc2, rc3, rc4, and rc5; , , respectively represent the losses of the PEJ, EMR, and RMR modules as a whole, so we have: ； ； ； ； For PEJ and EMR module, the selection of the training data for each classifier, in statement X, the set of labeled entities S t : { (s1, e1), (s2, e2), (s3, e3), (s4, e4)} as training positive examples, while randomly selecting ce=100 spans that do not belong to entities among all spans that meet the width restriction: (f1, none), (f2, none), , (f ce , none); The selection of the relationship training negative examples in the RMR module is different from the random selection manner in the PEJ and the EMR, and is selected from the free combination of the entities with the entity labels in the RMR module.

Images