Information extraction method and system based on multiple tasks and GlobalPointer model

Abstract

The invention relates to the technical field of natural language processing, and discloses an information extraction method and system based on multiple tasks and a GlobalPointer model, and the method comprises the following steps: constructing an information extraction model comprising an encoder, a GlobalPointer model and a classifier; performing multi-task training on the information extraction model by using a training data set comprising a named entity recognition data set, a relationship extraction data set and a triple extraction data set; inputting a text into the trained information extraction model, and encoding the text into vector representation through an encoder; all spans in the vector representation are extracted through a GlobalPointer model, and the spans are converted into entity vectors; a GlobalPointer model is used for extracting the relation between the entity vectors; classifying the entity vectors through a classifier; and integrating the entity vector, the relationship and the classification result, and outputting a triple data extraction result. The method solves the problems of low reasoning efficiency and high operation consumption in the prior art, and has the characteristics of being capable of improving the label utilization rate and suitable for mass data processing.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and more specifically, to an information extraction method and system based on multi-task and GlobalPointer model. Background Art

[0002] Existing information extraction methods usually handle entity recognition, relation extraction, and triple extraction tasks separately. These methods usually require the design of different models or the introduction of special tags in the model input, which is not only complex and computationally intensive to train, but also requires multiple runs of the model during inference, increasing the consumption of time and computing resources. In addition, many existing methods perform poorly when processing long sequences or massive data, and the accuracy and efficiency of the model cannot be guaranteed.

[0003] In addition, some unified information extraction models have been proposed recently, such as unified information extraction (UIE) based on unified structure generation, universal information extraction framework (USM) based on unified semantic matching, efficient unified information extraction framework (UniEX) based on span extraction, and unified information extraction based on multi-task instruction fine-tuning large model (InstructUIE).

[0004] For example, there is an entity relationship extraction method based on ordered structure encoding pointer network decoding, which includes: using the BERT pre-trained model to train word vectors for Word Embedding in the input layer, and then adding adversarial training to generate negative examples represented by sentence vectors to construct sentence initial vectors; using Bi-LSTM in the encoding layer to capture the global semantic information of the text; using the decoding idea of ​​the pointer network in the decoding layer to perform head entity extraction, tail entity and relationship extraction respectively, using Sigmoid instead of Softmax to predict the input, and completing the entity relationship triple extraction task.

[0005] However, the existing technology has the problems of low reasoning efficiency and high computing consumption. Therefore, how to invent an information extraction method with high reasoning efficiency and low computing consumption is a technical problem that needs to be solved urgently in this technical field. Summary of the invention

[0006] In order to solve the problems of low reasoning efficiency and high computational consumption in the prior art, the present invention provides an information extraction method and system based on multi-tasking and GlobalPointer model, which has the characteristics of being able to improve label utilization and being suitable for massive data processing.

[0007] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows:

[0008] An information extraction method based on multi-task and GlobalPointer model includes the following steps:

[0009] Build an information extraction model including encoder, GlobalPointer model, and classifier;

[0010] Use training datasets including named entity recognition datasets, relationship extraction datasets, and triple extraction datasets to perform multi-task training on the information extraction model;

[0011] Input the text into the trained information extraction model and encode it into a vector representation through the encoder; extract all spans in the vector representation through the GlobalPointer model and convert the spans into entity vectors; use the GlobalPointer model to extract the relationship between entity vectors; classify the entity vectors through the classifier;

[0012] Integrate entity vectors, relations, and classification results to output triple data extraction results.

[0013] Preferably, the encoder adopts a BERT-type model, specifically, any one of BERT, ALBERT, and RoBERTa language model encoders.

[0014] Furthermore, the classifier is specifically a multi-layer perceptron classifier, which supports single-label and multi-label classification.

[0015] Furthermore, a training dataset including a named entity recognition dataset, a relationship extraction dataset, and a triple extraction dataset is used to perform multi-task training on the information extraction model. The specific steps are as follows:

[0016] Input the named entity recognition dataset, the relationship extraction dataset, and the triple extraction dataset into the information extraction model to perform entity recognition, relationship extraction, and triple data extraction tasks, and obtain a mixed batch result;

[0017] The entity recognition loss, relation extraction loss, and triple extraction loss of the mixed batch results are calculated separately; when calculating the loss, the loss mask is used to achieve isolation between tasks.

[0018] Furthermore, when performing entity recognition tasks, the data of the entity recognition dataset is input into the trained information extraction model and encoded into a vector representation through the encoder; all spans in the vector representation are extracted through the GlobalPointer model and the spans are converted into entity vectors; the entity vectors are classified through the classifier to obtain the entity recognition result;

[0019] When performing a relationship extraction task, the data of the relationship extraction dataset is input into the trained information extraction model and encoded into a vector representation through an encoder. The span of the dataset is directly converted into an entity vector, and the relationship between the entity vectors is extracted using the GlobalPointer model to obtain the relationship extraction result.

[0020] When performing triple data extraction tasks, the data of the triple extraction dataset is input into the trained information extraction model and encoded into a vector representation through the encoder; all spans in the vector representation are extracted through the GlobalPointer model, and the spans are converted into entity vectors; the relationship between the entity vectors is extracted using the GlobalPointer model; the entity vectors are classified through the classifier; and the triple data extraction results are obtained by combining the entity vectors, relationships, and classification results.

[0021] Furthermore, loss masks are used to achieve isolation between tasks. Specifically, when calculating the loss of data in the mixed batch results, if the data is an entity recognition task, the loss mask ignores the relationship extraction loss; if the data is a relationship extraction task, the loss mask ignores the entity recognition loss; if the data is a triple data extraction task, the loss mask does not ignore any loss.

[0022] Furthermore, the span is converted into an entity vector, specifically: the head entity span is selected, and the mean vector of the head tag and the tail tag of the span is used as the head entity vector.

[0023] Furthermore, after using the GlobalPointer model to extract the relationship between entity spans, the output of the entity relationship is: head entity, tail entity, and the relationship between the head entity and the tail entity.

[0024] Furthermore, after the entity vector is classified by the classifier, the output of the entity category is: entity, category of the entity.

[0025] An information extraction system based on multi-task and GlobalPointer model, including a model building module, a model training module, an information extraction module, and a result output module;

[0026] The model building module is used to build an information extraction model including an encoder, a GlobalPointer model, and a classifier;

[0027] The model training module is used to perform multi-task training on the information extraction model using a training data set including a named entity recognition data set, a relationship extraction data set, and a triple extraction data set;

[0028] The information extraction module is used to input the text into the trained information extraction model, encode it into a vector representation through an encoder; extract all spans in the vector representation through a GlobalPointer model, and convert the spans into entity vectors; use the GlobalPointer model to extract the relationship between the entity vectors; and classify the entity vectors through a classifier;

[0029] The result output module is used to integrate entity vectors, relations, and classification results, and output triple data extraction results.

[0030] The beneficial effects of the present invention are as follows:

[0031] The present invention proposes an information extraction model including an encoder, a GlobalPointer model, and a classifier, and uses a training data set including a named entity recognition data set, a relationship extraction data set, and a triple extraction data set to perform multi-task training on the information extraction model; compared with the prior art, the information extraction model of the present invention has the advantages of efficient reasoning and simple training. When reasoning different tasks, the information extraction model of the prior art needs to use different prompt templates for multiple reasonings and needs to perform pre-training with a large amount of knowledge injection, while the information extraction model of the present invention only needs one reasoning to output the results of all tasks and does not need to add pre-training steps; compared with USM and UniEX, there is no need to add special tags before inputting text, so the information extraction model used in the present invention has a higher tag utilization rate and is more advantageous in processing big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of an information extraction method based on multi-task and GlobalPointer model of the present invention.

[0033] Figure 2 It is a schematic diagram of a multi-task training process of an information extraction method based on multi-task and GlobalPointer model of the present invention.

[0034] Figure 3 It is a schematic diagram of the reasoning process of an information extraction method based on multi-task and GlobalPointer model in Example 2 of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Example 1

[0037] like Figure 1 As shown, an information extraction method based on multi-task and GlobalPointer model includes the following steps:

[0038] Build an information extraction model GPUIE including encoder, GlobalPointer model, and classifier;

[0039] Use training datasets including named entity recognition datasets, relationship extraction datasets, and triple extraction datasets to perform multi-task training on the information extraction model;

[0040] Input the text into the trained information extraction model and encode it into a vector representation through the encoder; extract all spans in the vector representation through the GlobalPointer model and convert the spans into entity vectors; use the GlobalPointer model to extract the relationship between entity vectors; classify the entity vectors through the classifier;

[0041] Integrate entity vectors, relations, and classification results to output triple data extraction results.

[0042] In this embodiment, when extracting all spans in the vector representation, the GlobalPointer model confirms the head vector and the tail vector, arranges the vector representation obtained by the encoder from the beginning to the end in a horizontal and vertical order to form a score matrix, calculates the score matrix, and obtains all spans based on the scores of the score matrix.

[0043] In this embodiment, when extracting the relationship between entity vectors, the GlobalPointer model identifies the head entity vector and the tail entity vector, arranges the entity vectors horizontally and vertically in order from head to tail to form a score matrix, calculates the score matrix, and obtains the relationship between all entity vectors based on the score matrix.

[0044] The entity extraction method used in the present invention has fewer parameters and is more efficient than the original GlobalPointer model. Only one head entity is needed to extract entities of all categories and then the types of all entities can be output through a classification layer. The number of head entities in the original GlobalPointer model is equal to the number of entity categories. When the number of entity categories reaches hundreds, this will consume huge computing resources. The information extraction model of the present invention innovatively converts the relationship extraction task into a method similar to entity extraction. Compared with GPLinker, which uses GlobalPointer to extract relationships based on label granularity, the information extraction model of the present invention directly applies the GlobalPointer model to extract relationships at the entity granularity. At the same time, because the number of entities is generally much smaller than the number of labels, the amount of computation when extracting relationships is also smaller.

[0045] Example 2

[0046] More specifically, in a specific embodiment, the encoder adopts a BERT-type model, specifically, any one of BERT, ALBERT, and RoBERTa language model encoders.

[0047] In a specific embodiment, the classifier is specifically a multi-layer perceptron classifier, which supports single-label and multi-label classification.

[0048] like Figure 2 As shown, in a specific embodiment, a training data set including a named entity recognition data set, a relationship extraction data set, and a triple extraction data set is used to perform multi-task training on the information extraction model GPUIE, and the specific steps are:

[0049] Input the named entity recognition dataset, the relationship extraction dataset, and the triple extraction dataset into the information extraction model to perform entity recognition, relationship extraction, and triple data extraction tasks, and obtain a mixed batch result;

[0050] The entity recognition loss, relation extraction loss, and triple extraction loss of the mixed batch results are calculated separately; when calculating the loss, the loss mask is used to achieve isolation between tasks.

[0051] In this embodiment, the classification data set is also input into the information extraction model to perform the classification task, and the classification task loss is calculated to train the information extraction model.

[0052] In a specific embodiment, when performing an entity recognition task, the data of the entity recognition data set is input into a trained information extraction model and encoded into a vector representation through an encoder; all spans in the vector representation are extracted through a GlobalPointer model, and the spans are converted into entity vectors; the entity vectors are classified through a classifier to obtain an entity recognition result;

[0053] When performing a relationship extraction task, the data of the relationship extraction dataset is input into the trained information extraction model and encoded into a vector representation through an encoder. The span of the dataset is directly converted into an entity vector, and the relationship between the entity vectors is extracted using the GlobalPointer model to obtain the relationship extraction result.

[0054] When performing triple data extraction tasks, the data of the triple extraction dataset is input into the trained information extraction model and encoded into a vector representation through the encoder; all spans in the vector representation are extracted through the GlobalPointer model, and the spans are converted into entity vectors; the relationship between the entity vectors is extracted using the GlobalPointer model; the entity vectors are classified through the classifier; and the triple data extraction results are obtained by combining the entity vectors, relationships, and classification results.

[0055] In a specific embodiment, a loss mask is used to achieve isolation between tasks, specifically: when calculating the loss of data in a mixed batch result, if the data is an entity recognition task, the loss mask ignores the relationship extraction loss; if the data is a relationship extraction task, the loss mask ignores the entity recognition loss; if the data is a triple data extraction task, the loss mask does not ignore any loss.

[0056] In this embodiment, the present invention adopts a multi-task learning method. During the model training, each task layer is basically independent of each other and does not affect each other. Some similar tasks can share parameters of some layers, but it is not necessary. A mixed batch method is used in training, that is, a batch of data may contain data of one or more tasks. To ensure that the tasks do not affect each other, the present invention uses a loss mask to achieve isolation between tasks.

[0057] In a specific embodiment, before inputting the text into the trained information extraction model, the text is preprocessed by data cleaning, sentence segmentation, and word segmentation.

[0058] In a specific embodiment, the span is converted into an entity vector by selecting a head entity span and using the mean vector of the head tag and the tail tag of the span as the head entity vector.

[0059] In a specific embodiment, after extracting the relationship of entity span using the GlobalPointer model, the output of the entity relationship is: a head entity, a tail entity, and the relationship between the head entity and the tail entity.

[0060] In a specific embodiment, after the entity vector is classified by a classifier, the output of the entity category is: entity, category of the entity.

[0061] In this embodiment, Figure 3 As shown in Figure 1, the reasoning process of using the information extraction model for information extraction is as follows:

[0062] ①Model input, text preprocessing and tokenization;

[0063] ②Text encoding vectorization embedding;

[0064] ③GlobalPointer extracts span and supports nested entities;

[0065] (④ Use the mean vector of the head and tail tokens of the span as the entity vector, such as:

[0066] Where E represents vectorized embedding;

[0067] ⑤GlobalPointer model extracts relations;

[0068] ⑥ Use multi-layer perceptron MLP for entity classification, supporting single-label and multi-label classification;

[0069] ⑦Entity relationship output: the format is (head entity, tail entity, relationship category);

[0070] ⑧Entity category output: the format is (entity, entity category).

[0071] The beneficial effects of the information extraction model adopted by the present invention are:

[0072] Improve efficiency: The model parameters of the information extraction model are not much different from those of the single model. Multiple task labels can be output through one inference, saving computing resources and time.

[0073] Easy to train: Using a multi-task learning approach, there is no need to label datasets for all tasks at the same time, which reduces the difficulty and cost of data preparation.

[0074] Strong versatility: The method of the present invention does not need to introduce special tags in the model input, and the efficiency of processing long sequences and the ability of massive data are improved. In addition to conventional entity recognition, relationship extraction and entity classification, the information extraction model can also integrate other information extraction or extraction tasks such as text classification and event extraction, and has a wide range of applications.

[0075] High accuracy: Despite the unified processing tasks, the overall performance remains superior.

[0076] Example 3

[0077] An information extraction system based on multi-task and GlobalPointer model, including a model building module, a model training module, an information extraction module, and a result output module;

[0078] The model building module is used to build an information extraction model including an encoder, a GlobalPointer model, and a classifier;

[0079] The model training module is used to perform multi-task training on the information extraction model using a training data set including a named entity recognition data set, a relationship extraction data set, and a triple extraction data set;

[0080] The information extraction module is used to input the text into the trained information extraction model, encode it into a vector representation through an encoder; extract all spans in the vector representation through a GlobalPointer model, and convert the spans into entity vectors; use the GlobalPointer model to extract the relationship between the entity vectors; and classify the entity vectors through a classifier;

[0081] The result output module is used to integrate entity vectors, relations, and classification results, and output triple data extraction results.

[0082] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. An information extraction method based on multi-task and GlobalPointer model, characterized by: The following steps are involved: Build an information extraction model including encoder, GlobalPointer model, and classifier; Use training datasets including named entity recognition datasets, relationship extraction datasets, and triple extraction datasets to perform multi-task training on the information extraction model; Input the text into the trained information extraction model and encode it into a vector representation through the encoder; extract all spans in the vector representation through the GlobalPointer model and convert the spans into entity vectors; use the GlobalPointer model to extract the relationship between entity vectors; classify the entity vectors through the classifier; Integrate entity vectors, relations, and classification results to output triple data extraction results.

2. The information extraction method based on multi-task and GlobalPointer model according to claim 1, characterized in that: The encoder adopts a BERT-type model, specifically, any language model encoder of BERT, ALBERT, and RoBERTa is adopted.

3. The information extraction method based on multi-task and GlobalPointer model according to claim 1, characterized in that: The classifier is specifically a multi-layer perceptron classifier, which supports single-label and multi-label classification.

4. The information extraction method based on multi-task and GlobalPointer model according to claim 1, characterized in that: Use training datasets including named entity recognition datasets, relationship extraction datasets, and triple extraction datasets to perform multi-task training on the information extraction model. The specific steps are as follows: Input the named entity recognition dataset, the relationship extraction dataset, and the triple extraction dataset into the information extraction model to perform entity recognition, relationship extraction, and triple data extraction tasks, and obtain a mixed batch result; The entity recognition loss, relation extraction loss, and triple extraction loss of the mixed batch results are calculated separately; when calculating the loss, the loss mask is used to achieve isolation between tasks.

5. The information extraction method based on multi-task and GlobalPointer model according to claim 4, characterized in that: When performing entity recognition tasks, the data of the entity recognition dataset is input into the trained information extraction model and encoded into a vector representation through the encoder; all spans in the vector representation are extracted through the GlobalPointer model and converted into entity vectors; the entity vectors are classified through the classifier to obtain the entity recognition result; When performing a relationship extraction task, the data of the relationship extraction dataset is input into the trained information extraction model and encoded into a vector representation through an encoder. The span of the dataset is directly converted into an entity vector, and the relationship between the entity vectors is extracted using the GlobalPointer model to obtain the relationship extraction result. When performing triple data extraction tasks, the data of the triple extraction dataset is input into the trained information extraction model and encoded into a vector representation through the encoder; all spans in the vector representation are extracted through the GlobalPointer model, and the spans are converted into entity vectors; the relationship between the entity vectors is extracted using the GlobalPointer model; the entity vectors are classified through the classifier; and the triple data extraction results are obtained by combining the entity vectors, relationships, and classification results.

6. The information extraction method based on multi-task and GlobalPointer model according to claim 5, characterized in that: Use loss mask to achieve isolation between tasks. Specifically, when calculating the loss of data in the mixed batch results, if the data is an entity recognition task, the loss mask ignores the relationship extraction loss; if the data is a relationship extraction task, the loss mask ignores the entity recognition loss; if the data is a triple data extraction task, the loss mask does not ignore any loss.

7. The information extraction method based on multi-task and GlobalPointer model according to claim 1, characterized in that: The span is converted into an entity vector, specifically: the head entity span is selected, and the mean vector of the head tag and the tail tag of the span is used as the head entity vector.

8. The information extraction method based on multi-task and GlobalPointer model according to claim 1, characterized in that: After using the GlobalPointer model to extract the relationship between entity spans, the output of the entity relationship is: head entity, tail entity, and the relationship between the head entity and the tail entity.

9. The information extraction method based on multi-task and GlobalPointer model according to claim 1, characterized in that: After the entity vector is classified by the classifier, the output of the entity category is: entity, category of the entity.

10. An information extraction system based on multi-task and GlobalPointer model, characterized by: It includes model building module, model training module, information extraction module and result output module; The model building module is used to build an information extraction model including an encoder, a GlobalPointer model, and a classifier; The model training module is used to perform multi-task training on the information extraction model using a training data set including a named entity recognition data set, a relationship extraction data set, and a triple extraction data set; The information extraction module is used to input the text into the trained information extraction model, encode it into a vector representation through an encoder; extract all spans in the vector representation through a GlobalPointer model, and convert the spans into entity vectors; use the GlobalPointer model to extract the relationship between the entity vectors; and classify the entity vectors through a classifier; The result output module is used to integrate entity vectors, relations, and classification results, and output triple data extraction results.

Images