An entity relation extraction method based on BERT and entity position information

Abstract

The application discloses a kind of entity relation extraction methods based on BERT and entity position information belonging to the field of natural language processing, which comprises the following steps: step 1: obtaining entity relation extraction text dataset containing entity position mark information, divided into training set, validation set and test set;Step 2: considering the influence of entity position on sentence text features, an entity relation extraction composite model combining BERT, BiLSTM and attention mechanism is constructed;Step 3: the training set in the data set determined in step 1 is used to train the model, and the validation set is used to determine the stop round of training, and the result is used for test set evaluation;Step 4: the model is evaluated using the test set, and the rationality of the model is analyzed.The application realizes more accurate and efficient mining effect, provides a novel way to mine the potential relationship between entities, and brings good versatility and practical value.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing relation extraction, specifically relating to an entity relation extraction method based on BERT and entity location information. Background Technology

[0002] Relation extraction (RE) is the process of extracting triples (subject, relation, object) from given text. For example, given the sentence: Marie Curie, female, born in Poland, we extract (Marie Curie, birthplace, Poland). Relation extraction can support downstream tasks such as knowledge graphs, question answering systems, and search engines. From its definition, we can see that relation extraction mainly does two things. First, it identifies the subject and object of the text; this task is often called the Named Entity Recognition (NER) problem. Second, it determines the relation between these two entities; this task is often called the Relation Classification (RC) problem. Depending on the classification criteria, relation extraction can take several different forms. Based on the training method, it can be classified as supervised entity relation extraction and remotely supervised entity relation extraction; based on whether it includes NER subtasks, it can be divided into pipeline relation extraction and joint relation extraction; based on the method, it can be divided into rule-based relation extraction, deep learning-based relation extraction, and hybrid model relation extraction.

[0003] In current research on relation extraction, combining convolutional neural networks, recurrent neural networks, and attention mechanisms has been the mainstream approach. However, the positional information of entities also has a certain influence on the semantic expression of sentences. For example, in the sentence "Xiaoming has been working for Xiaohong since graduating from school," we can infer that Xiaoming and Xiaohong have an employment relationship, and the word "work" appears near "Xiaohong," providing a key clue. This reveals that entities in different positions within the text receive varying degrees of attention from each word, indicating that model performance needs improvement. Summary of the Invention

[0004] To address the limitations of traditional text representation methods that do not consider entity location information, this invention proposes an entity relationship extraction method based on BERT and entity location information.

[0005] A method for entity relation extraction based on BERT and entity location information includes the following steps:

[0006] Step 1: Obtain the entity relationship extraction text dataset containing entity location tagging information, and divide it into training set, validation set and test set;

[0007] Step 2: Consider the impact of entity location on sentence text features, and construct a composite model for entity relation extraction that combines BERT, BiLSTM, and attention mechanisms;

[0008] Step 3: Train the model using the training set in the dataset determined in Step 1, use the validation set to determine the stopping point of training, and use the results for evaluation on the test set.

[0009] Step 4: Evaluate the model using the test set and analyze its rationality.

[0010] Furthermore, step 1 includes the following specific steps:

[0011] Step 1-1: Through online resources, learn about the currently available open-source entity relationship extraction text datasets, and select a dataset that contains entity location tagging information as the experimental object;

[0012] Steps 1-2: Through preprocessing, the data is converted into the form of {number, head entity position, tail entity position, sentence, relation category}, and the dataset is divided into training set, validation set and test set in an 8:1:1 ratio.

[0013] Furthermore, in step 2, the relation extraction composite model includes a BERT layer, a sentence feature extraction layer, an entity feature extraction layer, an entity attention layer for words at different positions in the sentence, a feature combination layer, and a Softmax classification layer.

[0014] Furthermore, the BERT layer utilizes Google's open-source pre-trained BERT network to perform word embedding on the text, thereby obtaining its text sentence feature vector X;

[0015] The sentence feature extraction layer further extracts the contextual feature vector H of the text through a BiLSTM network;

[0016] The entity feature extraction layer obtains entity feature vectors through pre-labeled entity locations and calculates the average feature vector of the entity.

[0017] The attention layer for the entity at different positions in the sentence obtains the weights e of the two entities at different positions in the sentence by constructing an attention mechanism;

[0018] The feature combination layer obtains the weighted vector O based on the weight e and the context feature vector H;

[0019] The Softmax classification layer inputs the weighted vector O into the classifier to perform relation extraction and classification tasks, and trains the optimal solution model through cross-entropy loss.

[0020] Furthermore, step 2 includes the following specific steps:

[0021] Step 2-1: The text in the training set is processed through the BERT layer for word embedding. The BERT pre-trained model uses the BERT-base-uncased version provided by Huggingface. The vector X = [batch_size, src_len, input_dim], where batch_size is the batch size, src_len is the length of the sentence, and input_dim is the dimension of the word vector.

[0022] Step 2-2: Input the BERT-encoded vector into the BiLSTM to obtain the input vector H = [batch_size, src_len, enc_hid_dim * 2], where enc_hid_dim is the hidden layer vector of the LSTM; after being input into the BiLSTM, it undergoes a feature learning process of forward and backward propagation to obtain the forward hidden state and the backward hidden state H. fw =*H1 fw H2 fw ,…,H n fw ,+ and H bw =*H1 bw H2 bw ,…,H n bw The two are then combined to form the final feature vector H = Concat(H) of the BiLSTM network layer. fw H bw );

[0023] Steps 2-3, utilizing the entity's <el>The location of the entity is marked, and the vector information of the corresponding entity is obtained through the location index. The average of the two entity information is then calculated, as shown below:

[0024]

[0025]

[0026] Where i to j are the sentence position indices corresponding to the head entities, m to n are the sentence position indices corresponding to the tail entities, and H... ent1 H represents the average value of the BERT vector corresponding to entity 1. ent2 This represents the average value of the BERT vector corresponding to entity 2;

[0027] Steps 2-4: Calculate the attention e using the average vector of the two entity positions obtained above. The specific formula is as follows:

[0028]

[0029] Where V and W are parameter matrices trained, and when the three vectors are superimposed, H is... ent1 H ent2 Perform broadcasting operations so that the output vector at each time step is the same as H. ent1 H ent2 The attention vectors are stacked to generate an attention vector e = [batch_size, src_len, 1]. In order for e to be able to perform multiplication operations with the context feature vector H at corresponding positions, the last dimension of e is removed, so e = [batch_size, src_len].

[0030] Steps 2-5: Generate the vector O of attention weights that fuse entity information according to the formula, the specific formula being:

[0031] The Softmax function, O = e·H, normalizes O to obtain the output P = ..., P1, P2, P3, ..., P... N -, where P i ∈,0,1-. Further, step 2 uses the cross-entropy function as the loss function of the model, with the specific formula as follows:

[0032]

[0033] in It is the probability distribution predicted by the model, y i is the true distribution of the samples, and is the one-hot representation of the true labels, where i represents the i-th class of relation. During training, a dropout strategy with a value of 0.5 is introduced, and gradient descent is used to train the model parameters.

[0034] Furthermore, step 3 processes the text into the format required by the model and uses it to train the model, continuously adjusting the experimental parameters to obtain the optimal parameters.

[0035] Furthermore, in step 4, precision, recall, and F1 score are selected as evaluation metrics.

[0036] Furthermore, the formula for step 4 is as follows:

[0037]

[0038]

[0039]

[0040] This invention constructs a relation extraction model comprising a BERT layer, a sentence feature extraction layer, an entity feature extraction layer, an attention layer for entities at different positions in the sentence, a feature combination layer, and a Softmax classification layer. It provides an efficient and highly generalizable method for entity relation mining and modeling. Compared to traditional relation mining and extraction methods, this method achieves more accurate and efficient mining results, offering a novel approach to mining potential relationships between entities. It not only brings good versatility and practical value but also has significant reference value. The specific technical effects and innovations are mainly reflected in the following two aspects:

[0041] 1. This invention fully utilizes the advantages of pre-trained models, effectively solving the problem of low accuracy caused by polysemy. Furthermore, in pre-trained models, we only need to fine-tune the downstream tasks to achieve good results, effectively addressing the issue of small sample data.

[0042] 2. This invention fully considers the domain of relation extraction and the degree of influence of entity location on other words in a sentence. Therefore, it proposes an attention calculation method that combines entity location information. This method can effectively reflect the degree of emphasis of entity location information on different words in a sentence. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of an entity relationship extraction network based on BERT and entity location information.

[0044] Figure 2 This is a computational diagram illustrating the attention mechanism. Detailed Implementation

[0045] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] This invention discloses an entity relationship extraction method based on BERT and entity location information, comprising the following steps:

[0047] Step 1: Obtain the entity relationship extraction text dataset containing entity location tagging information, and divide it into training set, validation set and test set;

[0048] Step 1-1: Through online resources, learn about currently available open-source entity relation extraction text datasets. Select a dataset that contains entity location tagging information as the experimental subject.

[0049] In one embodiment, in step 1 above, the selected public dataset is SemEval-2010-Task8, which contains the following categories: Cause-Effect, Component-Whole, Content-Container, Entity-Destination, Entity-Origin, Instrument-Agency, Member-Collection, Message-Topic, Product-Producer, and Other. If the hierarchy of entities is not considered, they can be divided into 10 categories; if the hierarchy is considered, they can be divided into 19 categories.

[0050] In step 1 above, the text format in the public dataset is: "[CLS]The$kitchen$ is the last renovated part of the#house#." The CLS symbol is added at the beginning, and the $ symbol is added before and after the first entity, and the # symbol is added before and after the second entity.

[0051] Steps 1-2: Through preprocessing, the data is converted into the form of {number, head entity position, tail entity position, sentence, relation category}, and the dataset is divided into training set, validation set and test set in an 8:1:1 ratio.

[0052] Step 2: Fully consider the impact of entity location on sentence text features, and construct a composite entity relation extraction model that combines BERT (Bidirectional Encoder Representations from Transformers), BiLSTM, and attention mechanisms;

[0053] In one embodiment, in step 2 above, the relation extraction composite model includes a BERT layer, a sentence feature extraction layer, an entity feature extraction layer, an entity attention layer for words at different positions in the sentence, a feature combination layer, and a Softmax classification layer.

[0054] Specifically, the BERT layer utilizes Google's open-source pre-trained BERT network to embed words into text and obtain its text sentence feature vector X; BERT uses Transformer as the main framework of the algorithm, which can more thoroughly capture the bidirectional relationships in sentences; BERT uses the Mask Language Model (MLM) and Next Sentence Prediction multi-task training objectives; BERT uses more powerful machines to train on larger-scale data, enabling BERT's results to reach a new level.

[0055] The sentence feature extraction layer further extracts the contextual feature vector H of the text through a BiLSTM network. BiLSTM uses memory cells to store past or future information at long intervals, thus capturing the connections between sentence contexts. By setting forget gates, memory gates, and output gates, the model can be described more accurately after multiple iterations.

[0056] Among them, the entity feature extraction layer obtains entity feature vectors by pre-labeling entity locations and calculates the average feature vector of the entity;

[0057] Among them, the attention layer for entities to words at different positions in the sentence obtains the weights e of two entities to words at different positions in the sentence by constructing an attention mechanism;

[0058] Among them, the feature combination layer obtains the weighted vector O based on the weight e and the context feature vector H;

[0059] The Softmax classification layer takes the weighted vector O as input to the classifier to perform relation extraction and classification tasks, and trains the optimal solution model using cross-entropy loss. Figure 1 As shown, it specifically includes:

[0060] Step 2-1: The text in the training set, such as "[CLS]The$kitchen$ is the last renovated part of the#house#.", is processed through a BERT layer for word embedding. The BERT pre-trained model uses the BERT-base-uncased version provided by Huggingface. The vector X = [batch_size, src_len, input_dim], where batch_size is the batch size, src_len is the sentence length, and input_dim is the set dimension of the word vectors.

[0061] Step 2-2: Input the BERT-encoded vector into the BiLSTM to obtain the input vector H = [batch_size, src_len, enc_hid_dim * 2], where enc_hid_dim is the hidden layer vector of the LSTM.

[0062] In this process, after inputting into BiLSTM, the feature learning process involves forward and backward propagation, resulting in the forward hidden state and the backward hidden state H. fw =*H1 fw H2 fw ,…,H n fw ,+ and H bw =*H1 bw H2 bw ,…,H n bw The two are then combined to form the final feature vector H = Concat(H) of the BiLSTM network layer. fw H bw ).

[0063] Steps 2-3, utilizing the entity's <el>The location of the entity is marked, and the vector information of the corresponding entity is obtained through the location index. The average of the two entity information is then calculated, as shown below:

[0064]

[0065]

[0066] Where i to j are the sentence position indices corresponding to the head entities, and m to n are the sentence position indices corresponding to the tail entities. H ent1 H represents the average value of the BERT vector corresponding to entity 1. ent2 This represents the average value of the BERT vector corresponding to entity 2.

[0067] Steps 2-4: Calculate the attention e using the average vector of the two entity positions obtained above. The specific formula is as follows:

[0068]

[0069] Where V and W are parameter matrices used for training. When superimposing the three vectors, H needs to be adjusted. ent1 H ent2 Perform broadcasting operations so that the output vector at each time step is the same as H. ent1 H ent2 The attention vectors are then stacked. The final result is an attention vector e = [batch_size, src_len, 1], as shown below. Figure 2 As shown.

[0070] To enable e to be multiplied at corresponding positions with the context feature vector H, the last dimension of e needs to be removed, thus e = [batch_size, src_len].

[0071] Steps 2-5: Generate the vector O of attention weights that fuse entity information according to the formula, the specific formula being:

[0072] O = e·H

[0073] The Softmax function normalizes O, yielding the output P = ..., P1, P2, P3, ..., P... N -, where P i ∈,0,1-. This invention uses the cross-entropy function as the loss function of the model. Cross-entropy can measure the degree of difference between two different probability distributions in the same random distribution. The specific formula is:

[0074]

[0075] in It is the probability distribution predicted by the model, y i is the true distribution of the samples, and is the one-hot encoding of the true labels. Here, 'i' represents the i-th class of relation. To prevent overfitting, a dropout strategy with a value of 0.5 is introduced during training. Furthermore, gradient descent is used to train the model's parameters.

[0076] Step 3: Train the model using the training set from the dataset determined in Step 1. Use the validation set to determine the stopping point of training, and use the results for evaluation on the test set. Process the text into the format required by the model and participate in the model training, continuously adjusting the experimental parameters to obtain the optimal parameters.

[0077] Step 4: Evaluate the model using the test set and analyze its suitability. The evaluation metrics are precision, recall, and F1 score.

[0078] In one embodiment, step 3 above validates the model using a test set to obtain the final experimental results. Precision, recall, and F1 score are selected as evaluation metrics, as shown in the following formulas:

[0079]

[0080]

[0081]

[0082] For example, if the input sentence is: "[CLS]The$kitchen$is the last renovated part of the#house#.", the result will be: Component-Whole.

[0083] The present invention provides an entity relationship extraction method based on BERT and entity location information. It can obtain the vector representation of the text through the BERT network, further obtain the contextual information of the text through BiLSTM, calculate the attention weights required for different positions according to the entity positions, obtain the weighted feature vector, and finally dynamically classify the relationship category of the text according to the classifier.

[0084] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. It will be apparent to those skilled in the art that various modifications can be made to the above embodiments, and the general principles described herein can be applied to other embodiments without inventive effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.< / el> < / el>

Claims

1. A method for extracting entity relations based on BERT and entity position information, characterized in that, Includes the following steps: Step 1: Obtain the entity relationship extraction text dataset containing entity location tagging information, and divide it into training set, validation set and test set; Step 2: Consider the impact of entity location on sentence text features, and construct a composite model for entity relation extraction that combines BERT, BiLSTM, and attention mechanisms; Step 2-1: The text in the training set is processed through the BERT layer for word embedding. The BERT pre-trained model uses the BERT-base-uncased version provided by Huggingface. The vector X = [batch_size, src_len, input_dim], where batch_size is the batch size, src_len is the sentence length, and input_dim is the set dimension of the word vector. Step 2-2: Input the BERT-encoded vector into the BiLSTM to obtain the input vector H = [batch_size, src_len, enc_hid_dim * 2], where enc_hid_dim is the hidden layer vector of the LSTM. After being input into the BiLSTM, the vector undergoes forward and backward propagation feature learning processes to obtain the forward hidden state and the backward hidden state. and The two are then combined to form the final feature vector of the BiLSTM network layer. ; Steps 2-3, utilizing the entity's <el> The location of the entity is marked, and the vector information of the corresponding entity is obtained through the location index. The average of the two entity information is then calculated, as shown below:< / el> Where i~j are the sentence position indices corresponding to the head entity, and m~n are the sentence position indices corresponding to the tail entity. This represents the average value of the BERT vector corresponding to entity 1. This represents the average value of the BERT vector corresponding to entity 2; Steps 2-4: Calculate the attention e using the average vector of the two entity positions obtained above. The specific formula is as follows: Where V and W are both parameter matrices trained, and when the three vectors are superimposed, [the following is unclear and likely incomplete: "for..."] , Perform a broadcast operation so that the output vector at each time step is equal to... , The attention vectors are stacked to generate an attention vector e=[batch_size, src_len,1]. In order for e to be able to perform multiplication operations with the context feature vector H at corresponding positions, the last dimension of e is removed, so e=[batch_size, src_len]. Steps 2-5: Generate the vector O of attention weights that fuse entity information according to the formula, the specific formula being: The Softmax function normalizes O to obtain the output result. ,in ; Step 3: Train the model using the training set in the dataset determined in Step 1, use the validation set to determine the stopping point of training, and use the results for evaluation on the test set. Step 4: Evaluate the model using the test set and analyze its rationality.

2. The entity relation extraction method based on BERT and entity location information according to claim 1, characterized in that, Step 1 includes the following specific steps: Step 1-1: Through online resources, learn about the currently available open-source entity relationship extraction text datasets, and select a dataset that contains entity location tagging information as the experimental object; Steps 1-2: Through preprocessing, the data is converted into the form of {number, head entity position, tail entity position, sentence, relation category}, and the dataset is divided into training set, validation set and test set in an 8:1:1 ratio.

3. The entity relation extraction method based on BERT and entity location information according to claim 1, characterized in that, In step 2, the relation extraction composite model includes a BERT layer, a sentence feature extraction layer, an entity feature extraction layer, an entity attention layer for words at different positions in the sentence, a feature combination layer, and a Softmax classification layer.

4. The entity relation extraction method based on BERT and entity location information according to claim 3, characterized in that, The BERT layer uses Google's open-source pre-trained BERT network to embed words into the text and obtain its text sentence feature vector X. The sentence feature extraction layer further extracts the contextual feature vector H of the text through a BiLSTM network; The entity feature extraction layer obtains entity feature vectors through pre-labeled entity locations and calculates the average feature vector of the entity. The attention layer for the entity at different positions in the sentence obtains the weights e of the two entities at different positions in the sentence by constructing an attention mechanism; The feature combination layer obtains the weighted vector O based on the weight e and the context feature vector H; The Softmax classification layer inputs the weighted vector O into the classifier to perform relation extraction and classification tasks, and trains the optimal solution model through cross-entropy loss.

5. The entity relation extraction method based on BERT and entity location information according to claim 1, characterized in that, Step 2 uses the cross-entropy function as the loss function of the model, and the specific formula is as follows: in It is the model predicting the probability distribution. is the true distribution of the samples, and is the one-hot representation of the true labels, where i represents the i-th class of relation. During training, a dropout strategy with a value of 0.5 is introduced, and gradient descent is used to train the model parameters.

6. The entity relation extraction method based on BERT and entity location information according to claim 1, characterized in that, Step 3 processes the text into the format required by the model and uses it for model training, continuously adjusting the experimental parameters to obtain the optimal parameters.

7. The entity relation extraction method based on BERT and entity location information according to claim 1, characterized in that, Step 4 selects precision, recall, and F1 score as evaluation metrics.

8. The entity relation extraction method based on BERT and entity location information according to claim 7, characterized in that, The formula for step 4 is as follows: 。

Images