Entity and relation concatenation extraction method for text data

Abstract

The application provides a kind of entity and relation serial extraction method for text data.First, the text is preprocessed, and its word vector is extracted, then the word vector is input into the text entity extraction model BiLSTM-softmax to obtain the text with entity pair, then the text with ambiguity is disambiguated, finally, the text with entity pair is input into the relation discrimination model, and then the relation classification model is used to obtain the relation category of entity.The application can extract entity and relation in series for text data, and has the functions of long text processing, entity disambiguation and rich semantic function of anaphora resolution, and good entity extraction and relation extraction results are obtained.

Description

Technical Field

[0001] This invention belongs to the field of information extraction technology, specifically relating to a method for extracting entities and relationships from text data in a sequential manner. Background Technology

[0002] In information extraction tasks, entity extraction and entity relationship extraction play crucial roles. Entity extraction primarily involves extracting atomic information elements from text, such as names of people, organizations, geographical locations, event names, and times. Entity relationship extraction, on the other hand, involves extracting the categories of relationships between entities in text, such as parent-child relationships, romantic relationships, and superior-subordinate relationships within interpersonal relationships.

[0003] For entity extraction tasks, common methods include rule-based methods and model-based methods. (1) Rule-based methods are for entities with special contexts or texts with many features. The methods are simple and effective. As the amount of corpus increases, the situation becomes more and more complex, and conflicts may occur between rules, making the entire system unmaintainable. Therefore, rule-based methods are more suitable for extraction tasks of semi-structured or relatively standardized texts, and can achieve certain results when combined with business needs. (2) Model-based methods include traditional models such as Hidden Markov Models or Conditional Random Fields, and deep learning models such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), and BiLSTM (Bi-directional Long Short-Term Memory) have also performed well in NLP (Natural Language Processing) tasks. Compared to traditional models, RNNs can consider long-term contextual information and solve the feature selection problem of CRF (Conditional Random Field). Their focus is on network design and parameter tuning. However, RNNs generally require large training datasets, while CRF performs better on smaller datasets. A more effective approach is to combine BiLSTM with CRF, leveraging the strengths of both models.

[0004] For text entity relation extraction tasks, common methods include template-based methods, supervised learning methods, and semi-supervised / unsupervised learning methods. (1) Template-based methods include trigger word / string-based methods and dependency syntax-based methods. Their advantages are high accuracy and customizability, while their disadvantages are difficulty in maintenance and poor portability. (2) Supervised learning methods include machine learning methods and deep learning methods. Machine learning methods include MaxEnt (maximum entropy model). Methods like Bayes (Naive Bayes) and SVM (Support Vector Machine) typically train two classifiers: the first step determines if two entities are related, and the second step determines the relationship category. Deep learning methods are further divided into Pipeline methods (where entity recognition and relationship classification are two independent processes, with relationship recognition depending on the performance of entity recognition) and Joint Model methods (where entity recognition and relationship classification are optimized together). Pipeline methods include CR-CNN (Classifying Relations by Ranking with Convolutional Neural Networks), Att-CNN (Attention-based CNN), and Att-BiLSTM (Attention-Based Bidirectional Long Short-Term Memory Networks). These methods can lead to error propagation, making end-to-end optimization easier. Joint Model methods include LSTM-RNNs (Long Short-Term Memory Recurrent Neural Networks), but in terms of model performance, Joint Model methods do not offer significant improvements. Supervised learning methods have high requirements for datasets, the models are relatively fragile, the generalization ability is limited, it is difficult to extend new relationships, and the cost of obtaining large training sets is also relatively high. (3) Semi-supervised / unsupervised learning methods mainly use a small amount of labeled information for learning, represented by Bootstrap-based methods and distance supervision methods. Bootstrap-based methods primarily utilize a small set of initial seed tuples, then employ pattern learning to iterate and extract instances from unstructured data. New patterns are learned from these newly acquired instances, expanding the pattern set and identifying new potential relation triples. Distant supervision methods automatically construct large amounts of training data by aligning the knowledge base with unstructured text, reducing the model's reliance on manually labeled data and enhancing its cross-domain adaptability. Their advantages include low construction costs and suitability for large-scale construction, but disadvantages include low accuracy for sensitive initial seed results and a lack of confidence calculation for each result. Unsupervised learning methods, because they extract new relations, cannot accurately calculate precision and recall; therefore, they can only estimate them by randomly selecting a relation sample from the result set and then manually verifying it.

[0005] Existing information extraction tasks include extraction from various types of data, such as online medical entity extraction, named entity extraction from electronic medical records, named entity recognition of Chinese tourist attractions, and entity relationship extraction in the military field. Text data, as an indispensable part of the information transmitted daily by the public, contains a large amount of valuable entity and entity relationship information. Extracting entities and relationships from text data is an important component of enriching personal information databases and constructing complete and clear relationship networks.

[0006] The BiLSTM-CRF entity extraction model is a typical existing text entity extraction model belonging to the sequence labeling task. Its input is the text sequence to be processed, and its output is the probability that each character in the sequence belongs to each label class. This model consists of an embedding layer, several BiLSTM (Bidirectional Long Short-Term Memory) layers, and a CRF (Conditional Random Field) layer. The processing is as follows: each character / word in the input text sequence is represented as a vector (including character / word embeddings), and then fed into the BiLSTM layer. This layer outputs the probability that each character / word belongs to a certain label class. This probability is then fed into the CRF layer, where the output probabilities are corrected based on learned label transition relationships, outputting the optimal label sequence, and converting the output numerical symbols into intuitively understandable label categories.

[0007] The BiLSTM-Attention relation extraction model is a typical existing text relation extraction model. Its input is a text sequence with entity pairs labeled with location information, and its output is the relation category between the two entities in the sequence. This model consists of a word / character embedding layer, several BiLSTM layers (usually one layer), and an Attention layer. Its processing is as follows: the input is a sequence containing entity location information (such as "..."). <e1>Xiaoming likes to go shopping at the mall. <e2>pencil< / e2> ." <e1> ,< / e1> , <e2> ,< / e2> It consists of four position indicators that specify the start and end markers of an entity. Each character / word in the sequence is represented as a vector (including character / word embeddings), which is then fed into a BiLSTM layer. This layer outputs the high-order feature information of each character / word. The weight vector is then generated by the Attention layer. The word-level features at each time step are multiplied and merged into a sentence-level feature vector. Finally, the sentence-level feature vector is used for relation classification.

[0008] The two models mentioned above have some problems when directly applied to text data, including: (1) The daily production of text data is in the hundreds of millions. When the BiLSTM-CRF entity extraction model processes massive amounts of text data, the CRF layer takes up a lot of inference time when calculating the optimal label sequence, resulting in extremely low inference efficiency. This makes the model unsuitable for inference tasks involving massive amounts of data. (2) The BiLSTM-Attention model itself cannot generate entity pairs that it can classify. Although the BiLSTM-CRF entity model can generate candidate entities, there may be multiple entities in a text, and the proportion of binary entity pairs with relationships is generally not large. For example, there are 12 combinations of 4 entities, but there is only one entity pair with a real relationship. If the generated entities are directly processed into candidate entity pairs and sent to the relationship extraction model, the uneven distribution of data will have a great impact on the model performance. (3) Since the input sequence of the model only contains the location information of the entities but not the category information, BiLSTM-Attention cannot learn the category information of the entities. Therefore, some entity pairs that do not conform to common sense may be generated, such as classifying "Xiaoming" and "car" as family relationship. However, if the category information of "Xiaoming" entity "person name" and the category information of "car" entity "vehicle" are input into the relationship extraction model together, there should be improvement. Summary of the Invention

[0009] To overcome the shortcomings of existing technologies, this invention provides a concatenative extraction method for entities and relations from text data. First, the text is preprocessed, and its word vectors are extracted. Then, the word vectors are input into a text entity extraction model, BiLSTM-softmax, to obtain text with entity pairs. Next, ambiguous text is disambiguated. Finally, the text with entity pairs is input into a relation discrimination model, and then into a relation classification model to obtain the relation categories of the entities. This invention performs concatenative extraction of entities and relations from text data, and simultaneously possesses the functions of long text processing, entity disambiguation, and semantic dereference resolution, achieving better entity and relation extraction results.

[0010] A method for extracting entities and relationships from text data in a concatenated manner, characterized by the following steps:

[0011] Step 1: Input text. If the text has more than 500 characters, split it into several parts with a length of 500 characters each.

[0012] Step 2: Input the segmented text into the word vector model to obtain the word vectors corresponding to the text; the word vector model refers to the word vector model obtained by encoding the word vectors based on the collected Chinese and English corpora and training it using the word2vec method. Both the Chinese and English word vector models are trained using this method.

[0013] Step 3: Input the text word vectors into the text entity extraction model to extract entities and output text with entity pairs; the text entity extraction model refers to the BiLSTM-softmax entity extraction model trained on a sequence dataset with location information. The BiLSTM-softmax entity extraction model includes a bidirectional LSTM layer and a softmax layer, which is an improved model after replacing the CRF layer of the original BiLSTM-CRF model with a softmax layer.

[0014] Step 4: Determine if the text is ambiguous. If not, proceed to Step 5; otherwise, input the text into the entity disambiguation model for disambiguation processing, and then into the trained entity resolution model for referential resolution processing to obtain the disambiguated text, and proceed to Step 5. The entity disambiguation model refers to the entity disambiguation model trained on the Chinese Short Text dataset, which adopts the Bert-CNN network architecture, including Bert layers, convolutional layers, and Softmax layers. The entity resolution model refers to the SpanBert entity resolution model trained on the open-source dataset ontonotes5.0 labeled by IDC, which includes transformer layers, fully connected layers, and Softmax layers.

[0015] Step 5: Input the unambiguous or disambiguated text obtained in Step 4 into the entity relation discrimination model, and then obtain the relation category of the entity through the entity relation classification model; the entity relation discrimination model refers to the BiLSTM-Attention model trained on a sequence dataset containing entity location information and entity category information, wherein the BiLSTM-Attention model includes BiLSTM layers and Attention layers; the entity relation classification model refers to the BiLSTM-Attention model trained on a sequence dataset containing only entity location information.

[0016] The beneficial effects of this invention are as follows: By replacing the CRF layer in the BiLSTM-CRF model with a softmax layer, the model processing speed and computational efficiency can be greatly improved while achieving higher text entity extraction accuracy; the addition of text entity disambiguation and resolution processes can effectively solve the problem of ambiguity in Chinese entities, and significantly improve the accuracy of text entities in expressing the theme; by adding entity category information to the input of the BiLSTM-Attention model in the entity relationship discrimination model, the model can better take into account the category information between entities when learning the relationships between entities, thereby ensuring the accuracy of entity relationship discrimination; and the use of a serial entity and relationship extraction method standardizes the data connection between different models, enhances the ability to process massive amounts of text data, and allows entities after disambiguation and resolution to better express semantic information, greatly improving the accuracy and rationality of entity extraction and relationship extraction results. Attached Figure Description

[0017] Figure 1 This is a flowchart of the entity and relationship chain extraction method for text data according to the present invention. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments. The present invention includes, but is not limited to, the following embodiments.

[0019] like Figure 1 As shown, this invention provides a method for extracting entities and relationships from text data in a concatenated manner, the specific implementation process of which is as follows:

[0020] Step 1: Input text, determine the length of the text sequence. If the number of characters in the text is greater than 500, split it into several texts with a length of 500 characters, and record the sequence relationship between the texts.

[0021] Step 2: Input the segmented text into the word vector model to obtain the word vectors corresponding to the text; the word vector model refers to the word vector model trained on Chinese and English corpora using the word2vec method, including Chinese word vector model and English word vector model.

[0022] Step 3: Input the text word vectors into the text entity extraction model to extract entities and output text with entity pairs; the text entity extraction model refers to the BiLSTM-softmax entity extraction model trained on a sequence dataset with location information. The BiLSTM-softmax entity extraction model includes a bidirectional LSTM layer and a softmax layer, which is an improved model after replacing the CRF layer of the original BiLSTM-CRF model with a softmax layer.

[0023] Step 4: Determine if the text is ambiguous. If not, proceed to Step 5; otherwise, input the text into the entity disambiguation model for disambiguation processing, and then into the trained entity resolution model for referential resolution processing to obtain the disambiguated text, and proceed to Step 5. The entity disambiguation model refers to the entity disambiguation model trained on the Chinese Short Text dataset provided at CCKS2019 (National Conference on Knowledge Graph and Semantic Computation). The entity disambiguation model adopts the Bert-CNN network architecture, including Bert layers, convolutional layers, and Softmax layers. The entity resolution model refers to the SpanBert entity resolution model trained on the open-source dataset ontonotes5.0 labeled by IDC. The SpanBert entity resolution model includes transformer layers, fully connected layers, and Softmax layers.

[0024] Step 5: Input the unambiguous or disambiguated text obtained in Step 4 into the entity relation discrimination model, and then obtain the relation category of the entity through the entity relation classification model; the entity relation discrimination model refers to the BiLSTM-Attention model trained on a sequence dataset containing entity location information and entity category information. The BiLSTM-Attention model uses a BiLSTM network and adds an Attention mechanism, including BiLSTM layers and Attention layers; the entity relation classification model has the same structure as the entity relation discrimination model, but its training dataset only contains sequences with entity location information, that is, the BiLSTM-Attention model trained on a sequence dataset containing only entity location information.

[0025] To verify the effectiveness of this invention, the BiLSTM-softmax entity extraction model of this invention and the original BiLSTM-CRF model were compared. The BiLSTM-CRF model parameters were set as follows: learning_rate = 0.001, batch_size = 32, embedding_size = 128, hidden_size = 200, num_layers = 2, dropout = 0.2. Here, embedding_size refers to the character word vector dimension, hidden_size refers to the hidden layer dimension, and num_layers refers to the number of LSTM layers. The BiLSTM-SoftMax model parameters were set as follows: learning_rate = 0.001, batch_size = 32, embedding_size = 128, hidden_size = 200, num_layers = 1, dropout = 0.2. When trained and evaluated on a real dataset, the inference speed and F1 score (a metric that balances accuracy and recall; a higher score indicates better model performance) of the two models are shown in Table 1. It can be seen that the inference speed of the BiLSTM-softmax model of this invention is greatly improved.

[0026] Table 1

[0027] F1 Inference speed (items / s) BiLSTM-CRF model 0.8013 25 BiLSTM-SoftMax model 0.7952 700

[0028] Table 2 compares the F1 score and accuracy of the BiLSTM-Attention model with entity category information added in step 5 of the entity relationship discrimination model of this invention with the original BiLSTM-Attention model. It can be seen that the improved model of this invention not only improves the F1 score but also greatly enhances the reasonableness of the reasoning results (i.e., entity category pairs that cannot possibly have a relationship will be filtered out by the model).

[0029] Table 2

[0030] F1 accuracy Original BiLSTM-Attention model 0.8528 87％ This invention model 0.8934 92％

Claims

1. A method for extracting entities and relationships from text data in a concatenated manner, characterized in that... The steps are as follows: Step 1: Input text. If the text has more than 500 characters, split it into several parts with a length of 500 characters each. Step 2: Input the segmented text into the word vector model to obtain the word vectors corresponding to the text; the word vector model refers to the word vector model obtained by encoding the word vectors based on the collected Chinese and English corpora and training it using the word2vec method. Both the Chinese and English word vector models are trained using this method. Step 3: Input the text word vectors into the text entity extraction model to extract entities and output text with entity pairs; the text entity extraction model refers to the BiLSTM-softmax entity extraction model trained on a sequence dataset with location information. The BiLSTM-softmax entity extraction model includes a bidirectional LSTM layer and a softmax layer, which is an improved model after replacing the CRF layer of the original BiLSTM-CRF model with a softmax layer. Step 4: Determine if the text is ambiguous. If not, proceed to Step 5; otherwise, input the text into the entity disambiguation model for disambiguation processing, and then into the trained entity resolution model for referential resolution processing to obtain the disambiguated text, and proceed to Step 5. The entity disambiguation model refers to the entity disambiguation model trained on the Chinese Short Text dataset, which adopts the Bert-CNN network architecture, including Bert layers, convolutional layers, and Softmax layers. The entity resolution model refers to the SpanBert entity resolution model trained on the open-source dataset ontonotes5.0 labeled by IDC, which includes transformer layers, fully connected layers, and Softmax layers. Step 5: Input the unambiguous or disambiguated text obtained in Step 4 into the entity relation discrimination model, and then obtain the relation category of the entity through the entity relation classification model; the entity relation discrimination model refers to the BiLSTM-Attention model trained on a sequence dataset containing entity location information and entity category information, wherein the BiLSTM-Attention model includes BiLSTM layers and Attention layers; the entity relation classification model refers to the BiLSTM-Attention model trained on a sequence dataset containing only entity location information.

Images