Information extraction method and system based on multi-task and GlobalPointer model

Abstract

The present invention relates to the field of natural language processing technology, and discloses an information extraction method and system based on a multi-task and GlobalPointer model, comprising the following steps: constructing an information extraction model including an encoder, a GlobalPointer model, and a classifier; performing multi-task training on the information extraction model using a training dataset including a named entity recognition dataset, a relationship extraction dataset, and a triple extraction dataset; inputting text into the trained information extraction model and encoding it into a vector representation through the encoder; extracting all spans in the vector representation through the GlobalPointer model and converting the spans into entity vectors; extracting the relationships between the entity vectors using the GlobalPointer model; classifying the entity vectors through a classifier; and outputting a triple data extraction result by integrating the entity vectors, relationships, and classification results. The present invention solves the problems of low reasoning efficiency and high computational consumption in the prior art, and has the characteristics of being able to improve label utilization and being suitable for massive data processing.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and more specifically, to an information extraction method and system based on multi-task and GlobalPointer models. Background Art

[0002] Existing information extraction methods typically handle entity recognition, relation extraction, and triple extraction tasks separately. These methods often require designing separate models or introducing special tags into the model input. This not only makes training complex and computationally intensive, but also requires running the model multiple times during inference, increasing both time and computing resources. Furthermore, many existing methods perform poorly when processing long sequences or massive amounts of data, and model accuracy and efficiency cannot be guaranteed.

[0003] In addition, some unified information extraction models have been proposed recently, such as unified information extraction (UIE) based on unified structure generation, universal information extraction framework (USM) based on unified semantic matching, efficient unified information extraction framework (UniEX) based on span extraction, and unified information extraction based on multi-task instruction fine-tuning large model (InstructUIE).

[0004] For example, there is an entity relationship extraction method based on ordered structure encoding pointer network decoding. The method includes: using the BERT pre-training model to train word vectors for WordEmbedding in the input layer, and then adding adversarial training to generate negative examples of sentence vector representation to construct the initial sentence vector; using Bi-LSTM in the encoding layer to capture the global semantic information of the text; using the decoding idea of the pointer network in the decoding layer to perform head entity extraction, tail entity and relationship extraction respectively, and using Sigmoid instead of Softmax prediction input to complete the entity relationship triple extraction task.

[0005] However, the existing technology has the problems of low reasoning efficiency and high computational consumption. Therefore, how to invent an information extraction method with high reasoning efficiency and low computational consumption is a technical problem that urgently needs to be solved in this technical field. Summary of the Invention

[0006] In order to solve the problems of low reasoning efficiency and high computational consumption in the prior art, the present invention provides an information extraction method and system based on multi-tasking and GlobalPointer model, which has the characteristics of improving label utilization and being suitable for massive data processing.

[0007] In order to achieve the above-mentioned purpose of the present invention, the technical solutions adopted are as follows:

[0008] The information extraction method based on multi-task and GlobalPointer model includes the following steps:

[0009] Build an information extraction model including encoder, GlobalPointer model, and classifier;

[0010] Multi-task training of the information extraction model using training datasets including named entity recognition datasets, relationship extraction datasets, and triple extraction datasets;

[0011] The text is input into the trained information extraction model and encoded into a vector representation through the encoder. All spans in the vector representation are extracted through the GlobalPointer model and converted into entity vectors. The relationship between the entity vectors is extracted using the GlobalPointer model. The entity vectors are classified using the classifier.

[0012] Integrate entity vectors, relations, and classification results to output triple data extraction results.

[0013] Preferably, the encoder adopts a BERT-type model, specifically, any language model encoder of BERT, ALBERT, and RoBERTa.

[0014] Furthermore, the classifier is specifically a multi-layer perceptron classifier, which supports single-label and multi-label classification.

[0015] Furthermore, a training dataset including a named entity recognition dataset, a relationship extraction dataset, and a triple extraction dataset is used to perform multi-task training on the information extraction model. The specific steps are as follows:

[0016] Input the named entity recognition dataset, relationship extraction dataset, and triple extraction dataset into the information extraction model to perform entity recognition, relationship extraction, and triple data extraction tasks, and obtain mixed batch results;

[0017] The entity recognition loss, relation extraction loss, and triple extraction loss of the mixed batch results are calculated separately; when calculating the loss, the loss mask is used to achieve isolation between tasks.

[0018] Furthermore, when performing entity recognition tasks, the data of the entity recognition dataset is input into the trained information extraction model and encoded into a vector representation through the encoder. All spans in the vector representation are extracted through the GlobalPointer model and converted into entity vectors. The entity vectors are classified through the classifier to obtain the entity recognition results.

[0019] When performing relationship extraction tasks, the data of the relationship extraction dataset is input into the trained information extraction model and encoded into a vector representation through the encoder. The span of the dataset is directly converted into entity vectors, and the relationship between the entity vectors is extracted using the GlobalPointer model to obtain the relationship extraction result.

[0020] When performing a triple data extraction task, the data from the triple extraction dataset is fed into a trained information extraction model and encoded into a vector representation using an encoder. All spans in the vector representation are extracted using the GlobalPointer model and converted into entity vectors. The GlobalPointer model is used to extract relationships between entity vectors. The entity vectors are then classified using a classifier. The entity vectors, relationships, and classification results are combined to obtain the triple data extraction results. Furthermore, a loss mask is used to isolate tasks. Specifically, when calculating the loss of data in a mixed batch result, if the data is for an entity recognition task, the loss mask ignores the relationship extraction loss; if the data is for a relationship extraction task, the loss mask ignores the entity recognition loss; if the data is for a triple data extraction task, the loss mask does not ignore any losses.

[0021] Furthermore, the span is converted into an entity vector, specifically: the head entity span is selected, and the mean vector of the head tag and the tail tag of the span is used as the head entity vector.

[0022] Furthermore, after extracting the relationship between entity spans using the GlobalPointer model, the output of the entity relationship is: a head entity, a tail entity, and the relationship between the head entity and the tail entity.

[0023] Furthermore, after the entity vector is classified by the classifier, the output of the entity category is: entity, category of the entity.

[0024] The information extraction system based on multi-task and GlobalPointer model includes model construction module, model training module, information extraction module, and result output module;

[0025] The model building module is used to build an information extraction model including an encoder, a GlobalPointer model, and a classifier;

[0026] The model training module is used to perform multi-task training on the information extraction model using a training data set including a named entity recognition data set, a relationship extraction data set, and a triple extraction data set;

[0027] The information extraction module is used to input text into a trained information extraction model and encode it into a vector representation through an encoder; extract all spans in the vector representation through a GlobalPointer model and convert the spans into entity vectors; use the GlobalPointer model to extract the relationship between the entity vectors; and classify the entity vectors through a classifier;

[0028] The result output module is used to integrate entity vectors, relations, and classification results, and output triple data extraction results.

[0029] The beneficial effects of the present invention are as follows:

[0030] The present invention proposes an information extraction model including an encoder, a GlobalPointer model, and a classifier, and uses a training dataset including a named entity recognition dataset, a relationship extraction dataset, and a triple extraction dataset to perform multi-task training on the information extraction model. Compared with the existing technology, it has the advantages of efficient reasoning and simple training. The information extraction model of the existing technology requires multiple reasonings using different prompt templates when reasoning different tasks and requires pre-training with a large amount of knowledge injection. However, the information extraction model of the present invention only needs a single reasoning to output the results of all tasks and does not require additional pre-training steps. Compared with USM and UniEX, no special tags are required before the input text. Therefore, the information extraction model used in the present invention has a higher tag utilization rate and is more advantageous when processing big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of the information extraction method based on multi-task and GlobalPointer model of the present invention.

[0032] Figure 2 1 is a schematic diagram of the multi-task training process of the information extraction method based on multi-task and GlobalPointer model of the present invention.

[0033] Figure 3 1 is a flow chart of the reasoning process of the information extraction method based on multi-task and GlobalPointer model in Example 2 of the present invention. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Example 1

[0036] like Figure 1 As shown in FIG, the information extraction method based on the multi-task and GlobalPointer model includes the following steps:

[0037] Build an information extraction model GPUIE including encoder, GlobalPointer model, and classifier;

[0038] Multi-task training of the information extraction model using training datasets including named entity recognition datasets, relationship extraction datasets, and triple extraction datasets;

[0039] The text is input into the trained information extraction model and encoded into a vector representation through the encoder. All spans in the vector representation are extracted through the GlobalPointer model and converted into entity vectors. The relationship between the entity vectors is extracted using the GlobalPointer model. The entity vectors are classified using the classifier.

[0040] Integrate entity vectors, relations, and classification results to output triple data extraction results.

[0041] In this embodiment, when extracting all spans from the vector representation, the GlobalPointer model confirms the head vector and the tail vector, arranges the vector representations obtained by the encoder from the beginning to the end in a horizontal and vertical order to form a score matrix, calculates the score matrix, and obtains all spans based on the scores of the score matrix.

[0042] In this embodiment, when extracting the relationship between entity vectors, the GlobalPointer model identifies the head entity vector and the tail entity vector, arranges the entity vectors horizontally and vertically in order from head to tail to form a score matrix, calculates the score matrix, and obtains the relationship between all entity vectors based on the score matrix.

[0043] The entity extraction method used in the present invention has fewer parameters and is more efficient than the original GlobalPointer model. It only needs one head entity to extract entities of all categories and then output the types of all entities through a classification layer. The number of head entities in the original GlobalPointer model is equal to the number of entity categories. When the number of entity categories reaches hundreds, this will consume huge computing resources. The information extraction model of the present invention innovatively converts the relationship extraction task into a method similar to entity extraction. Compared with GPLinker's use of GlobalPointer to extract relationships based on label granularity, the information extraction model of the present invention directly applies the GlobalPointer model to extract relationships at the entity granularity. At the same time, because the number of entities is generally much smaller than the number of labels, the amount of computation when extracting relationships is also smaller.

[0044] Example 2

[0045] More specifically, in a specific embodiment, the encoder adopts a BERT-type model, specifically, any one of BERT, ALBERT, and RoBERTa language model encoders.

[0046] In a specific embodiment, the classifier is specifically a multi-layer perceptron classifier, which supports single-label and multi-label classification.

[0047] like Figure 2As shown, in a specific embodiment, a training dataset including a named entity recognition dataset, a relationship extraction dataset, and a triple extraction dataset is used to perform multi-task training on the information extraction model GPUIE. The specific steps are:

[0048] Input the named entity recognition dataset, relationship extraction dataset, and triple extraction dataset into the information extraction model to perform entity recognition, relationship extraction, and triple data extraction tasks, and obtain mixed batch results;

[0049] The entity recognition loss, relation extraction loss, and triple extraction loss of the mixed batch results are calculated separately; when calculating the loss, the loss mask is used to achieve isolation between tasks.

[0050] In this embodiment, the classification data set is also input into the information extraction model to perform the classification task, and the classification task loss is calculated to train the information extraction model.

[0051] In a specific embodiment, when performing an entity recognition task, the data of the entity recognition dataset is input into a trained information extraction model and encoded into a vector representation by an encoder; all spans in the vector representation are extracted by a GlobalPointer model and the spans are converted into entity vectors; the entity vectors are classified by a classifier to obtain an entity recognition result;

[0052] When performing relationship extraction tasks, the data of the relationship extraction dataset is input into the trained information extraction model and encoded into a vector representation through the encoder. The span of the dataset is directly converted into entity vectors, and the relationship between the entity vectors is extracted using the GlobalPointer model to obtain the relationship extraction result.

[0053] When performing a triple data extraction task, the data from the triple extraction dataset is input into a trained information extraction model and encoded into a vector representation using an encoder. All spans in the vector representation are extracted using a GlobalPointer model and converted into entity vectors. The GlobalPointer model is used to extract the relationships between the entity vectors. The entity vectors are classified using a classifier. The entity vectors, relationships, and classification results are combined to obtain the triple data extraction result. In one specific embodiment, a loss mask is used to isolate tasks. Specifically, when calculating the loss of data in a mixed batch result, if the data is an entity recognition task, the loss mask ignores the relationship extraction loss; if the data is a relationship extraction task, the loss mask ignores the entity recognition loss; if the data is a triple data extraction task, the loss mask does not ignore any loss.

[0054] In this embodiment, the present invention employs a multi-task learning approach, ensuring that each task layer is essentially independent of each other during model training. Some similar tasks may share parameters at certain layers, but this is not required. Training utilizes a mixed batch approach, meaning that a batch of data may contain data from one or more tasks. To ensure that tasks do not interfere with each other, the present invention employs loss masks to isolate tasks.

[0055] In a specific embodiment, before inputting the text into the trained information extraction model, the text is preprocessed by data cleaning, sentence segmentation, and word segmentation.

[0056] In a specific embodiment, the span is converted into an entity vector by selecting a head entity span and using the mean vector of the head tag and the tail tag of the span as the head entity vector.

[0057] In a specific embodiment, after extracting the relationship of entity spans using the GlobalPointer model, the output of the entity relationship is: a head entity, a tail entity, and the relationship between the head entity and the tail entity.

[0058] In a specific embodiment, after the entity vector is classified by a classifier, the output of the entity category is: entity and category of the entity.

[0059] In this embodiment, Figure 3 As shown in Figure 2, the reasoning process of using the information extraction model for information extraction is as follows:

[0060] Model input, text preprocessing, word segmentation and tokenization;

[0061] Text encoding vectorization embedding;

[0062] GlobalPointer extracts span and supports nested entities;

[0063] Use the mean vector of the span's head token and tail token as the entity vector, such as:

[0064] , where E represents vectorized embedding;

[0065] GlobalPointer model extracts relationships;

[0066] Use multi-layer perceptron MLP for entity classification, supporting single-label and multi-label classification;

[0067] Entity relationship output: The format is (head entity, tail entity, relationship category);

[0068] Entity category output: The format is (entity, entity category).

[0069] The beneficial effects of the information extraction model adopted by the present invention are:

[0070] Improved efficiency: The model parameters of the information extraction model are similar to those of the single model. Multiple task labels can be output through a single inference, saving computing resources and time.

[0071] Easy to train: Using a multi-task learning approach, there is no need to label datasets for all tasks at the same time, which reduces the difficulty and cost of data preparation.

[0072] Strong versatility: The method of the present invention does not require the introduction of special tags in the model input, and the efficiency of processing long sequences and the ability to process massive data are improved. In addition to conventional entity recognition, relationship extraction and entity classification, the information extraction model can also integrate other information extraction or extraction tasks such as text classification and event extraction, and has a wide range of applications.

[0073] High precision: Despite unified processing tasks, the overall performance remains superior.

[0074] Example 3

[0075] The information extraction system based on multi-task and GlobalPointer model includes model construction module, model training module, information extraction module, and result output module;

[0076] The model building module is used to build an information extraction model including an encoder, a GlobalPointer model, and a classifier;

[0077] The model training module is used to perform multi-task training on the information extraction model using a training data set including a named entity recognition data set, a relationship extraction data set, and a triple extraction data set;

[0078] The information extraction module is used to input text into a trained information extraction model and encode it into a vector representation through an encoder; extract all spans in the vector representation through a GlobalPointer model and convert the spans into entity vectors; use the GlobalPointer model to extract the relationship between the entity vectors; and classify the entity vectors through a classifier;

[0079] The result output module is used to integrate entity vectors, relations, and classification results, and output triple data extraction results.

[0080] Obviously, the above embodiments of the present invention are merely examples for the purpose of illustrating the present invention, and are not intended to limit the embodiments of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. An information extraction method based on multi-task and GlobalPointer model, characterized by: The following steps are involved: Build an information extraction model including encoder, GlobalPointer model, and classifier; Use the training datasets including the named entity recognition dataset, the relationship extraction dataset, and the triple extraction dataset to perform multi-task training on the information extraction model. The specific steps are as follows: Input the named entity recognition dataset, relationship extraction dataset, and triple extraction dataset into the information extraction model to perform entity recognition, relationship extraction, and triple data extraction tasks, and obtain mixed batch results; When performing entity recognition tasks, the data of the entity recognition dataset is input into the trained information extraction model and encoded into a vector representation through the encoder. All spans in the vector representation are extracted through the GlobalPointer model and converted into entity vectors. The entity vectors are classified through the classifier to obtain the entity recognition results. When performing relationship extraction tasks, the data of the relationship extraction dataset is input into the trained information extraction model and encoded into a vector representation through the encoder. The span of the dataset is directly converted into entity vectors, and the relationship between the entity vectors is extracted using the GlobalPointer model to obtain the relationship extraction result. When performing triple data extraction tasks, the data of the triple extraction dataset is input into the trained information extraction model and encoded into a vector representation through an encoder. All spans in the vector representation are extracted using the GlobalPointer model and converted into entity vectors. The GlobalPointer model is used to extract the relationship between the entity vectors. The entity vectors are classified using a classifier. The entity vectors, relationships, and classification results are combined to obtain the triple data extraction results. Calculate the entity recognition loss, relation extraction loss, and triple extraction loss of the mixed batch results separately; when calculating the loss, use the loss mask to achieve isolation between tasks; The text is input into a trained information extraction model and encoded into a vector representation using an encoder. All spans in the vector representation are extracted using the GlobalPointer model and converted into entity vectors. The GlobalPointer model is used to extract the relationships between the entity vectors. The entity vectors are classified using a classifier. The output of the entity relationship is: the head entity, the tail entity, and the relationship between the head entity and the tail entity. Integrate entity vectors, relations, and classification results to output triple data extraction results.

2. The information extraction method based on multi-task and GlobalPointer model according to claim 1, characterized in that: The encoder adopts a BERT-type model, specifically, any language model encoder of BERT, ALBERT, and RoBERTa.

3. The information extraction method based on multi-task and GlobalPointer model according to claim 1, characterized in that: The classifier is specifically a multi-layer perceptron classifier, which supports single-label and multi-label classification.

4. The information extraction method based on multi-task and GlobalPointer model according to claim 1, characterized in that: Use loss mask to achieve isolation between tasks. Specifically, when calculating the loss of data in the mixed batch results, if the data is an entity recognition task, the loss mask ignores the relationship extraction loss; if the data is a relationship extraction task, the loss mask ignores the entity recognition loss; if the data is a triple data extraction task, the loss mask does not ignore any loss.

5. The information extraction method based on multi-task and GlobalPointer model according to claim 1, characterized in that: The span is converted into an entity vector. Specifically, the head entity span is selected and the mean vector of the head tag and the tail tag of the span is used as the head entity vector.

6. The information extraction method based on multi-task and GlobalPointer model according to claim 1, characterized in that: After the entity vector is classified by the classifier, the output of the entity category is: entity and the category of the entity.

7. An information extraction system based on multi-task and GlobalPointer model, characterized by: Used to implement the information extraction method according to any one of claims 1 to 6, comprising a model construction module, a model training module, an information extraction module, and a result output module; The model building module is used to build an information extraction model including an encoder, a GlobalPointer model, and a classifier; The model training module is used to perform multi-task training on the information extraction model using a training data set including a named entity recognition data set, a relationship extraction data set, and a triple extraction data set; The information extraction module is used to input text into a trained information extraction model and encode it into a vector representation through an encoder; extract all spans in the vector representation through a GlobalPointer model and convert the spans into entity vectors; use the GlobalPointer model to extract the relationship between the entity vectors; and classify the entity vectors through a classifier; The result output module is used to integrate entity vectors, relations, and classification results, and output triple data extraction results.

Images