End-to-end entity and relation joint extraction method

Abstract

The invention discloses an end-to-end entity and relationship joint extraction method, which comprises the following steps of: performing entity and relationship labeling and relationship category merging on text data, and constructing an entity relationship labeling data set; obtaining a text token by using a pre-training language model; training the entity relationship joint extraction network model by using the marked entity relationship data to obtain a model weight file; and predicting other samples by using the entity relationship joint extraction network model and the trained model weight file to obtain an entity relationship prediction result, and performing relationship splitting according to a preset entity relationship integration rule table to obtain an accurate entity relationship prediction result. The entity and relation joint extraction method is high in entity and relation joint extraction accuracy, high in speed and high in robustness, and has a good entity and relation extraction effect on data with complex entity categories, various relation categories and uneven sample distribution.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an end-to-end entity and relationship joint extraction method. Background Art

[0002] With the rapid development of information technology, data in various fields such as news and media has exploded. Since the text information on the Internet is large in volume, complex in structure, and has diverse sources, the cost of manually processing massive text is very high and inefficient, so information extraction technology has emerged, which uses computers instead of manual work to automatically extract text into structured data with unified structure and easy to store through algorithms. As one of the most critical technologies in the information extraction process, the goal of relationship extraction is to identify multiple entities hidden in the text and the relationships between entities in a given text to form relationship triples (head entity, relationship, tail entity). By extracting entity relationships from massive texts on the Internet, knowledge reasoning capabilities are provided for upper-level applications such as knowledge graphs.

[0003] As an important task in information extraction, entity relationship extraction has high application value and is widely used in downstream tasks such as knowledge graphs, information retrieval, and automatic question answering. Accurately and quickly extracting the required entity relationships can improve the accuracy and efficiency of these tasks.

[0004] With the rapid development of deep learning in recent years, its application fields have become more and more, and have achieved certain results. Combining deep learning with natural language processing has become a trend. Existing entity and relationship extraction generally adopts a step-by-step extraction method, that is, entity extraction is performed first, and then relationship extraction is performed. This method has entity nesting and error transmission problems, which causes entity and relationship extraction errors. Summary of the invention

[0005] In view of this, the present invention provides an end-to-end entity and relationship joint extraction method, which extracts entities in a sentence and the relationships between entities. When extracting entity relationships, the relationships are classified based on the head and tail of the entity respectively, so as to solve the problems of error propagation and nested entities and improve the speed and accuracy of entity and relationship extraction.

[0006] The present invention discloses an end-to-end entity and relationship joint extraction method, which comprises:

[0007] The text data is annotated with entities and relationships, and the entity and relationship categories are merged to construct an entity relationship annotation dataset; a pre-trained language model is used to obtain text tokens; the annotated entity relationship data is used to train the entity relationship joint extraction network model to obtain a model weight file; the entity relationship joint extraction network model and the trained model weight file are used to predict other samples to obtain entity relationship prediction results, and the relationships are split according to the preset entity relationship integration rules to obtain accurate entity and relationship prediction results.

[0008] Furthermore, text data in the required field is collected, the data is segmented, and entity and relationship annotations are performed; relationship categories are merged to determine the entity and relationship categories for joint extraction; and an entity relationship extraction dataset is constructed using head and tail entities, head and tail entity categories, relationships, and token text.

[0009] Furthermore, the entity relationship extraction dataset is constructed using head and tail entities, head and tail entity categories, relations and token texts, including:

[0010] Based on the head and tail entities and text tokens, the head and tail entity position labels are obtained, and then the head and tail entity relationship category labels are constructed based on the entity categories and entity relationships, and the categories without entity relationships are padded with 0.

[0011] Furthermore, the pre-trained language model is combined to obtain text tokens, the long-cut and short-padded operations are performed on the text tokens, and the supplemented length of each text is recorded.

[0012] Furthermore, the input of the entity relationship joint extraction network model is text data. When the input text data exceeds a preset length, the sentences are segmented, and the segmented sentences are converted into vocabulary ids and input into the pre-trained model, which are vectorized to extract sentence features.

[0013] Furthermore, the vectorization is performed to extract sentence features, including:

[0014] The RoBERTa pre-trained language model is used to obtain sentence features, and the head and tail entity features and the entity start and end position relationship features are extracted respectively. When performing entity extraction, the hidden layer of the entity features is split and position encoded, and the head and tail entity matrices are extracted respectively, and the head and tail entities in the sentence are output; when performing relationship extraction, the sentence features are fully connected to obtain a feature matrix, which includes the number of types of entity relationships in the dataset. The feature matrix is ​​split, and the entity start position relationship and entity end position relationship are calculated; when finalizing entities and relationships, the head and tail entity extraction results, the entity start position relationship extraction results, and the entity end position relationship extraction results are merged to form the final entity relationship extraction result.

[0015] Furthermore, the rows of the head and tail entity matrix represent the starting position of the entity, and the columns of the matrix represent the ending position of the entity. - indicates that the calculation result is a negative number and the entity does not exist, and + indicates that the calculation result is a positive number and the entity exists. Since the blank part is symmetrical, the lower triangle does not need to be calculated.

[0016] Furthermore, when performing relationship extraction and determining entities and relationships, first extract the entity start position features and entity end position features in the head and tail entity matrix, then perform relationship classification, take the union of the classification results, and finally compare and merge the extracted head and tail entities to form the entity relationship joint extraction result.

[0017] Furthermore, the entity relationship joint extraction network model is trained using the labeled entity relationship data to obtain a model weight file, including:

[0018] Obtain labeled entity relationship data, and merge entity relationship categories with a small number of samples according to preset entity relationship integration rules; construct an entity relationship training data set; use the data set to train the entity relationship joint extraction network model, and stop training after the loss value of the model loss function converges, and finally obtain the model training weight, that is, the model weight file; wherein the model loss function includes head and tail entity loss, entity starting position relationship prediction loss, and entity ending position relationship prediction loss.

[0019] Furthermore, the text data to be predicted is segmented, and the entity-relationship joint extraction network model and the training weights of the model are used to perform entity and relationship prediction on the segmented sentences to obtain the entity and relationship prediction results of each sentence, and the preset entity relationship integration rule table is combined to obtain accurate entities and relationships; the entities and relationships of all sentences are merged to form the final entity and relationship prediction results.

[0020] Due to the adoption of the above technical solution, the present invention has the following advantages:

[0021] 1. The present invention can effectively solve the error transmission problem generated in the traditional entity relationship model. The traditional entity relationship extraction method processes entity extraction and relationship extraction separately, and the result of relationship extraction often depends on the result of entity extraction. This causes the error generated in the entity extraction process to be transmitted to the relationship extraction process and affects the result of relationship extraction. However, the present invention extracts the relationship between the head and tail entities at the same time through the entity relationship matrix. Its relationship extraction process does not depend on the entity extraction process, so there is no error transmission.

[0022] 2. The present invention can effectively solve the problem of entity nesting. The traditional entity relationship extraction method cannot solve the situation where a position has multiple entity relationships. The present invention extracts multiple start and end positions of the entity through the entity relationship matrix, and the start position and the end position can be arbitrarily combined, which effectively solves the problem of entity nesting.

[0023] 3. The present invention can alleviate the problem of unbalanced relationship category samples. Since there are different relationships between entity categories, when there are many types of relationships, the number of samples of some relationship categories is small, resulting in unbalanced training samples and low accuracy of the model in categories with few samples. The present invention integrates relationship categories with few samples and restores the final relationship through entity relationship integration rules during prediction, so as to achieve better entity and relationship extraction effects even when the relationship category samples are unbalanced.

[0024] 4. The present invention has fast extraction speed and high accuracy. The traditional entity relationship extraction method is divided into two stages: entity extraction and relationship extraction. It requires training two models, which is slow and reduces the extraction accuracy due to error transmission. The present invention directly extracts entities and relationships from sentences through a joint extraction method, and when performing relationship extraction, it also models and classifies the starting and ending positions of the head and tail entities respectively. Only when the head and tail entity relationship categories coincide, the final entity relationship is determined, which has a high accuracy rate.

[0025] 5. The present invention is easy to update and maintain, has strong feasibility, and has a controllable iteration cycle. When encountering an imbalanced relationship category sample, the traditional relationship extraction method generally adopts the method of resampling or undersampling, but this method essentially does not involve new data in training, usually requires more small sample category data to be labeled, and the iteration cycle is unstable; while the present invention, when processing small sample categories, integrates small sample categories into larger categories and adds them to the training data. After repeated several times, better results can be achieved on small sample categories, and maintenance iteration is convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0027] Figure 1 A schematic diagram of the overall process of entity relationship extraction according to an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of a network structure for joint extraction of entity relationships according to an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of the entity relationship extraction model training process according to an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of the entity relationship extraction prediction process according to an embodiment of the present invention;

[0031] Figure 5 A schematic diagram of an entity matrix calculated by an entity relationship joint extraction network according to an embodiment of the present invention;

[0032] Figure 6 Schematic diagram of the entity starting position relationship matrix calculated by the entity relationship joint extraction network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described in conjunction with the accompanying drawings and embodiments, and the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present invention.

[0034] See also Figure 1 The present invention provides an embodiment of an end-to-end entity and relationship joint extraction method, which includes:

[0035] Step 1: Merge entity relationship categories to reduce complexity and alleviate the problem of low model accuracy caused by sample imbalance. Taking the entity relationship extraction task in the economic field as an example, the preset entity relationship integration rules are shown in Table 1. Since there are few samples of the two categories of control and ownership in the real samples, after integration, the entity relationship only retains the two categories of subordination and ownership, which greatly reduces the complexity of the model;

[0036] Table 1 Integration rules for entity relationships in the economic field

[0037]

[0038] Step 2: Based on the entity relationships integrated in step 1, determine the entity relationship categories to be jointly extracted.

[0039] Step 3: Use the head and tail entities, head and tail entity categories, relations and token text to construct the entity relationship extraction data set. The main construction method is: based on the head and tail entities and text tokens, obtain the head and tail entity position labels, and then construct the head and tail entity relationship category labels based on the entity category and entity relationship, and fill the categories without entity relationship with 0 to ensure the unity of the label matrix;

[0040] Step 4: Combine the pre-trained language model to obtain text tokens, perform long-cut and short-fill operations on the text tokens, and record the supplemented length of each text;

[0041] Step 5: Use the entity relationship joint extraction network to train the data set, stop training after the loss value converges, and obtain the model weight file;

[0042] Step 6: Use the entity relationship joint extraction network and the trained model weight file to predict other samples, and split the relationship according to the preset entity relationship integration rules to obtain accurate entity relationship prediction results;

[0043] Step 7: Merge the relationship prediction results to form the document entity relationship extraction results.

[0044] In a specific embodiment, the present invention designs an entity relationship joint extraction network model for the sample imbalance entity relationship extraction task. The network structure is as follows: Figure 2 As shown in the figure, the network input is text data with a maximum length of 256. When the processed text is long text data, the sentences are usually segmented according to the sentence ending punctuation such as the period, and the segmented sentences are converted into the vocabulary id and input into the pre-trained model, and vectorized to extract sentence features: the RoBERTa pre-trained language model is used to obtain sentence features, and the head and tail entity features and the entity start and end position relationship features are extracted respectively. When extracting the head and tail entities, the hidden layer of the entity features is segmented and positionally encoded, and finally the head and tail entity matrices are extracted respectively, and the head and tail entities in the sentence are output. When performing relationship extraction and determining entities and relationships, the entity start position features and entity end position features in the matrix are first extracted, and then the relationship classification is performed, and the classification results are taken as the union, and finally the extracted head and tail entities are compared and merged to form the entity relationship extraction result.

[0045] Figure 1 This is a schematic diagram of the overall process of relationship extraction in the present invention, which is divided into a model training part and a prediction part. The training part uses labeled entity relationship data to train the model and obtain a weight file; during prediction, the text data is input into a network model with weights, and the entity relationship results are output. The relationship is then split in combination with the preset entity relationship integration rules to obtain the final prediction result.

[0046] The training process is as follows Figure 3 As shown in the figure, firstly, the labeled entity relationship data is obtained, and the relationship categories with a small number of samples are merged according to the preset entity relationship integration rules; secondly, an entity relationship training data set is constructed; then the entity relationship joint extraction network is used to train the data set, and the training is stopped after the loss value converges, and finally the training weights of the model are obtained.

[0047] Predict some processes such as Figure 4As shown in the figure, the text data to be predicted is first segmented according to specific punctuation marks, and then the entity relationship joint extraction network and the trained model weights are used to predict the entity relationship of the segmented sentences to obtain the entity relationship prediction results of each sentence, and then the preset entity relationship integration rules are combined to obtain the accurate entity relationship; the relationship between entity pairs of different sentences is merged to form the final entity relationship prediction result.

[0048] In this specific implementation case, the present invention designs an entity relationship joint extraction network for the task of class imbalance entity relationship extraction. The network structure is as follows: Figure 2 As shown, the input of the network is text data, where the length of the text is 1*256. The network uses the RoBERTa pre-trained language model, including a fully connected layer, a position encoding layer, and an attention layer: First, the RoBERTa pre-trained language model is used to extract text features from the input text id, and the text features are fully connected three times. When the head and tail entities are extracted, a feature matrix of [seq_len, 2*128*2] is obtained, where 128 is the hidden layer dimension, one of which is the head and tail entity matrices, and the other 2 is helpful for dividing the calculation of attention. Q and K; then the feature matrix is ​​split into Q and K, with a dimension of [seq_len, 2, 128], and position encoding is performed for Q and K to incorporate position information; finally, the final head and tail entity matrix is ​​calculated, as shown Figure 5 As shown, the rows of the matrix represent the starting position of the entity, the columns of the matrix represent the ending position of the entity, - means that the calculation result is a negative number and the entity does not exist, + means that the calculation result is a positive number and the entity exists. Since the blank part is symmetrical, the lower triangle does not need to be calculated. When performing relationship extraction, a feature matrix of [seq_len, 12*128*2] is obtained by performing a full connection operation on the text features, where 12 represents the number of types of entity relationships in the data set. The feature matrix is ​​also split into Q and K with a dimension of [seq_len, 12, 128]. The starting position relationship of the entity pair is calculated to form an entity pair starting position relationship matrix, as shown in Figure 6 As shown in the figure, it represents the "subordinate" relationship matrix between "position" and "company". - means that the calculation result is a negative number and there is no relationship, and + means that the calculation result is a positive number and there is a relationship. Since the matrix is ​​symmetrical, the lower triangle of the matrix is ​​a blank matrix and does not need to be calculated. The calculation of the entity end position relationship is similar to the calculation of the entity start position relationship, and will not be explained here one by one. Finally, the final entity relationship extraction result is formed based on the head and tail entity extraction results, the entity start position relationship extraction results, and the entity end position relationship extraction results.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An end-to-end entity and relationship joint extraction method, characterized in that: include: Perform entity and relationship annotation on text data and merge relationship categories to construct an entity relationship annotation dataset; Use the pre-trained language model to obtain text tokens; Use the labeled entity relationship data to train the entity relationship joint extraction network model to obtain the model weight file; use the entity relationship joint extraction network model and the trained model weight file to predict other samples to obtain the entity relationship prediction results, and then split the relationship according to the preset entity relationship integration rule table to obtain accurate entity relationship prediction results.

2. The method according to claim 1, characterized in that Collect text data in the required field, segment the data, and annotate entities and relationships; merge relationship categories and determine the entity and relationship categories to be jointly extracted; use head and tail entities, head and tail entity categories, relationships, and token text to construct an entity relationship extraction dataset.

3. The method according to claim 2, characterized in that The entity relationship extraction dataset is constructed using head and tail entities, head and tail entity categories, relations and token texts, including: Based on the head and tail entities and text tokens, the head and tail entity position labels are obtained, and then the head and tail entity relationship category labels are constructed based on the entity categories and entity relationships, and the categories without entity relationships are padded with 0.

4. The method according to claim 1, characterized in that: Combined with the pre-trained language model, the text tokens are obtained, the long-cut and short-fill operations are performed on the text tokens, and the supplemented length of each text is recorded.

5. The method according to any one of claims 1 to 4, characterized in that: The input of the entity relationship joint extraction network model is text data. When the input text data exceeds the preset length, the sentences are segmented, and the segmented sentences are converted into vocabulary IDs and input into the pre-trained model, which are vectorized to extract sentence features.

6. The method according to claim 1, characterized in that The vectorization is performed to extract sentence features, including: The RoBERTa pre-trained language model is used to obtain sentence features, and the head and tail entity features and the entity start and end position relationship features are extracted respectively. When extracting entities, the hidden layer of the entity features is split and positionally encoded, and the head and tail entity matrices are extracted respectively, and the head and tail entities in the sentence are output; when extracting relations, a feature matrix is ​​obtained by performing a full connection operation on the sentence features. The feature matrix includes the number of types of entity relations in the dataset, and the feature matrix is ​​split to calculate the entity start position relationship and the entity end position relationship; when finalizing entities and relations, the head and tail entity extraction results, the entity start position relationship extraction results, and the entity end position relationship extraction results are merged to form the final entity relationship extraction result.

7. The method according to claim 6, characterized in that The rows of the head and tail entity matrix represent the starting position of the entity, and the columns of the matrix represent the ending position of the entity. - indicates that the calculation result is a negative number and the entity does not exist, and + indicates that the calculation result is a positive number and the entity exists. Since the blank part is symmetrical, the lower triangle does not need to be calculated.

8. The method according to claim 1, characterized in that When performing relationship extraction and determining entities and relationships, first extract the entity start position features and entity end position features in the head and tail entity matrices, then perform relationship classification, take the union of the classification results, and finally compare and merge the extracted head and tail entities to form the entity relationship joint extraction result.

9. The method according to claim 1, characterized in that: The entity relationship joint extraction network model is trained using the labeled entity relationship data to obtain a model weight file, including: Obtain labeled entity relationship data, and merge relationship categories with a small number of samples according to preset entity relationship integration rules; construct an entity relationship training data set; use the data set to train the entity relationship joint extraction network model, and stop training after the loss value of the model loss function converges, and finally obtain the model training weight, that is, the model weight file; wherein the model loss function includes head and tail entity loss, entity starting position relationship prediction loss, and entity ending position relationship prediction loss.

10. The method according to claim 1, characterized in that The text data to be predicted is segmented, and the entity-relationship joint extraction network model and the model's training weights are used to predict the entities and relationships of the segmented sentences to obtain the entity and relationship prediction results of each sentence. The preset entity relationship integration rules are combined to obtain accurate entities and relationships; the entity and relationship data of all sentences are merged to form the final entity and relationship prediction results.

Images