An information extraction active learning method and device based on superposition concept discrimination

Abstract

The application provides an information extraction active learning method and device based on superposition concept discrimination, and belongs to the technical field of natural languages.The core idea of the application is to discriminate superposition concepts so that an information extraction model can clearly mark the extraction category of data.The application comprises two main parts: firstly, a superposition concept generation method based on an exclusion method, which generates a series of superposition concept sets through an "A but not B" method;and secondly, a text retrieval model based on superposition concept discrimination, which respectively encodes the semantic representation of a text and a superposition concept set, calculates the confidence score of the text containing the superposition concept set through inner product, and finally retrieves a plurality of texts from unlabeled corpus through the confidence score, requires a labeler to mark, and trains an information extraction model.The application learns the marking information of superposition concept entities, so that the model can learn relevant generalization information, thereby accurately extracting a target category.

Description

Technical Field

[0001] This invention relates to an active learning method and apparatus for information extraction based on superimposed concept discrimination, belonging to the field of natural language processing technology. Background Technology

[0002] Information extraction aims to automatically extract desired information from text, including but not limited to entity information and event information. Taking entity information extraction as an example, given the sentence "The UK reported 36 new cases of monkeypox yesterday," an information extraction system should be able to identify "the UK" as a "country" entity and "monkeypox" as an "infectious disease" entity. Information extraction is a key task in natural language processing.

[0003] The challenge of information extraction lies in training a model to accurately extract information using only a small amount of labeled data. Training an information extraction system capable of accurately extracting target information relies on a large amount of labeled data, but the cost of labeling data is high, making it difficult to obtain large-scale, accurate labeled data. This leads to a major challenge in information extraction: sparsity of labeled data. Sparsity of labeled data raises an inherent problem in information extraction: the challenge of accurate generalization. That is, the information extraction model struggles to accurately understand the extracted content, resulting in overgeneralization or undergeneralization. For example, in the above example, "United Kingdom" can be identified as a "country" entity, a "government" entity, and a "location" entity. If the extraction target is a "government" entity, then identifying the target category as a "country" entity results in undergeneralization, while identifying it as a "location" entity results in overgeneralization. For a specific information extraction requirement, how to accurately enable the model to understand the extraction requirement when only labeled data is available for training is a crucial issue. Furthermore, this problem is inherent to information extraction due to the sparsity of labeled data and cannot be solved without introducing additional information. Therefore, the key to solving this problem is how to introduce enough additional information without incurring excessive costs.

[0004] To address this issue, several terms need to be introduced. A concept refers to a specific generalization of extracted information. For example, the concept of the entity "United Kingdom" includes "country," "government," and "location," all of which could potentially serve as the target extraction category for "United Kingdom." A general concept in the labeled data refers to the common concept shared by labeled entities within the same category. For instance, the general concepts for "United Kingdom" and "London" include "location" and "government." An overlay concept refers to other concepts associated with the general concepts in the labeled data. For example, if the general concepts are "location," "government," and "country," then overlay concepts include "park" and "city," where "park" and "government" are both sub-concepts of "location," and "city" and "country" are both sub-concepts of "government." By determining whether an overlay concept belongs to the target extraction category, the aforementioned challenge of precise generalization can be resolved. For example, if we know that "park" does not belong to the target category, then we know that the target category is not the entity "location"; conversely, if we know that "city" belongs to the target category, then we know that the target category is the entity "government." Therefore, how to effectively find superimposed concepts from labeled data and incorporate the discrimination information of superimposed concepts into the model is an important issue. Summary of the Invention

[0005] To address the challenge of accurate generalization in information extraction, this invention provides an active learning method and apparatus for information extraction based on superimposed concept discrimination. The invention first identifies superimposed concepts in the labeled data, then retrieves text containing entities with superimposed concepts from a large-scale corpus, requiring annotators to annotate them. Finally, additional annotated data is used together with the original labeled data to train the model. By learning the annotation information of superimposed concept entities, the model can learn relevant generalization information, thereby accurately extracting the target category.

[0006] The technical solution adopted in this invention is as follows:

[0007] A method for generating superimposed concepts based on the elimination method includes the following steps:

[0008] Construct a sequence-to-sequence neural network model, which includes an encoder and a decoder;

[0009] This sequence-to-sequence neural network model performs entity concept generation tasks. It inputs labeled entities and their context into an encoder, which captures semantic information to obtain semantic representations. The semantic representations are then input into a decoder, which generates text sequences by predicting words at each time step. The conceptual vocabulary corresponding to the labeled entities is obtained through parsing. Concepts of the same category are aggregated to obtain general concepts for that category.

[0010] A set of superimposed concepts is generated by using a process of elimination to represent a general concept.

[0011] Furthermore, the sequence-to-sequence neural network model performs a concept generation task. It inputs labeled data and prefix prompts into the encoder, which captures the semantic information of the labeled data to obtain semantic feature representations. The decoder processes the semantic feature representations. At each prediction time step, it predicts the probability distribution of each word and selects the word with the highest probability as the predicted word to generate a text sequence. It then parses the concept words generated for each entity mention in the text sequence as the concepts mentioned by the entity. Finally, it merges the concepts mentioned by entities of the same category to obtain a general concept set for the category.

[0012] Furthermore, the method of generating a set of superimposed concepts from general concepts through elimination involves, specifically, for any two general concepts, the corresponding set of superimposed concepts takes the form of retaining one concept and removing the other. This is in the form of "A but not B," where A is the retained concept and B is the excluded concept. For example, "location but not government" includes a series of concepts that are not "government" but are associated with locations, such as "park."

[0013] Furthermore, the encoder of the sequence-to-sequence neural network model is based on a self-attention mechanism.

[0014] Furthermore, the decoder of the sequence-to-sequence neural network model is based on a hybrid attention mechanism.

[0015] Furthermore, the general concepts are matched pairwise (pairwise matching is performed when using the exclusion method) to obtain a set of superimposed concepts.

[0016] An active learning method for information extraction based on superimposed concept discrimination, comprising a text retrieval neural network model based on superimposed concept discrimination, the method including the following steps:

[0017] The above-described method for generating superimposed concepts is used to generate a set of superimposed concepts;

[0018] The text retrieval neural network model takes an overlay set of concepts and unlabeled text as inputs, respectively, and obtains the semantic representation of the overlay set of concepts and the semantic representation of the unlabeled text.

[0019] The confidence score of the superimposed concept set corresponding to the text is obtained by calculating the inner product of the semantic representation of the superimposed concept set and the semantic representation of the unlabeled text.

[0020] Based on the confidence score, a portion of the text is selected from the unlabeled corpus, labeled, and used to train the information extraction model.

[0021] Furthermore, the above-mentioned active learning method for information extraction based on superimposed concept discrimination specifically includes the following steps:

[0022] This text retrieval neural network model performs a text encoding task, inputting arbitrary text into the encoder, which captures the entity information of the text and obtains entity feature representations.

[0023] This text retrieval neural network model performs the task of encoding an overlay concept set. The overlay concept set is input into the encoder, which captures the information of the overlay concept set and obtains the feature representation of the overlay concept.

[0024] This text retrieval neural network model performs the task of matching text with superimposed concepts. The confidence score of the text containing entities with superimposed concepts can be obtained by the inner product of the entity feature representation of the text and the superimposed concept feature representation.

[0025] Calculate the confidence score of all unlabeled texts relative to all overlay concept sets, sort them, and select texts to be labeled from highest to lowest confidence score. Then, request labelers to annotate the texts. This labeled data, along with the original labeled data, will be used to train the information extraction model.

[0026] Furthermore, the text retrieval neural network model is pre-trained. During the pre-training process, the text retrieval neural network model adopts a contrastive learning approach, where positive examples are texts containing entities with corresponding superimposed concepts, and negative examples are texts containing concept entities without corresponding superimposed concepts.

[0027] Furthermore, the encoder of the text retrieval neural network model is based on a self-attention mechanism.

[0028] An active learning device for information extraction based on superposition concept discrimination includes a memory and a processor. The memory stores a computer program configured to be executed by the processor. The computer program includes instructions for performing the above-described method of the present invention.

[0029] The beneficial effects of this invention are:

[0030] 1) By using a method to determine whether superimposed concepts belong to the target category, the challenge of accurate generalization is effectively solved.

[0031] 2) An active learning method based on labeled entities is used to identify overlapping concepts. Annotators label entities containing overlapping concepts, and the labeled data is used to train the model, incorporating the information for identifying overlapping concepts into the model.

[0032] 3) An elimination-based method for generating superimposed concepts is adopted, which directly obtains the set of superimposed concepts from the general concepts of the target category, thus avoiding the direct acquisition of a large number of superimposed concepts.

[0033] 4) Use text retrieval tasks to retrieve texts containing overlapping concepts from an unlabeled corpus. Attached Figure Description

[0034] Figure 1 This is a schematic diagram illustrating the superposition concept and general concept in the embodiments of the present invention.

[0035] Figure 2 The flowchart illustrates a method for generating superposition concepts based on the exclusion method, as provided in an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram of an active learning device for information extraction based on superposition concept discrimination in an embodiment of the present invention. Detailed Implementation

[0037] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings.

[0038] The method proposed in this invention is based on superimposed concept discrimination, and incorporates relevant information from superimposed concept discrimination into the information extraction model through labeled data. The core idea of ​​superimposed concept discrimination is that all entity types can be described by a series of concepts, and these concepts come from a unified concept set. Taking entity recognition tasks as an example, such as... Figure 1 As shown: For two labeled entities, "China" and "United States," their common concepts are "country," "government," and "location." In this case, if the target category is the "government" entity, existing named entity recognition models might mistakenly identify the target category as "location" or "country," leading to overgeneralization and undergeneralization problems. In this situation, concepts like "park" and "city" are called overlapping concepts. Determining whether overlapping concepts belong to the target category helps the model judge the generalization of the target category. By confirming that "park" does not belong to the target category, it can be confirmed that the target category is not "location"; by confirming that "city" belongs to the target category, it can be confirmed that the target category is "government."

[0039] Based on the above ideas, this invention proposes a method for generating superimposed concepts based on the elimination method, which obtains a set of superimposed concepts through the elimination method. For example... Figure 2As shown, the method first uses a concept extraction model to obtain general concepts from the labeled data. Then, it obtains the corresponding set of superimposed concepts in the form of "A but not B". Here, A and B are general concepts for the corresponding target category entities, and "but not" means "but not". "A but not B" includes a series of concepts that are not B but are related to A. For example, when the general concepts are "country", "government", and "location", the set of superimposed concepts includes: "location but not government" (this set includes "park"), "government but not country" (this set includes "city"), "country but not location" (this set is empty), etc. The superimposed concepts and unlabeled text are input into a text retrieval neural network model based on superimposed concept discrimination to calculate confidence scores. High-scoring texts are selected for annotation and used for training the information extraction model. This method can also be adapted to other information extraction tasks: for example, event detection tasks, where the category is the event category and the entity is the event trigger word. Its basic principle is to generate general concepts through the concept extraction module and obtain a set of superimposed concepts, and then retrieve and annotate relevant texts containing superimposed concepts through the retrieval module.

[0040] This invention proposes an active learning method for information extraction based on superimposed concept discrimination, such as... Figure 3 As shown, it mainly includes: 1) overlay concept extraction; 2) overlay instance retrieval. The concept extraction model and sequence-to-sequence neural network model are based on a deep neural network encoder-decoder framework, which performs the concept generation task through a generative model. The text retrieval neural network model is based on a deep neural network encoder model, which takes text or overlay concept set as input, outputs the corresponding feature representation, and calculates the inner product of the text representation and the overlay concept set representation to obtain the confidence score of their matching, thereby performing text retrieval based on the score.

[0041] Taking entity extraction as an example, the technical solution adopted in the embodiments of the present invention is described as follows:

[0042] 1. An elimination-based method for generating superposition concepts

[0043] First, a concept extraction model is included, the purpose of which is to extract relevant concepts from the labeled text, as shown below.

[0044]

[0045] The input consists of two parts: (1) a prefix hint, containing a task symbol [MD] and the entity to be described; and (2) the input text, which is the entity corresponding to the entity to be described. The output is in the form of natural language, i.e., "entity 1 is concept 1, concept 2, ... entity 2 is concept 1, concept 2, ... ". The output is processed to obtain the concept of each entity. Common concepts of the same entity category will be directly merged to obtain the common concept of each entity category.

[0046] The general concept will be constructed by building a set of superimposed concepts in the form of "A but not B", as shown below:

[0047]

[0048] The input is a general concept of the target category. The output is to match all general concepts in pairs to form the form "A but not B". This form represents a set of superimposed concepts, specifically meaning that the concept belongs to A but not to B. For example, "park" belongs to "location but not government" because "park" belongs to "location" but not "government".

[0049] 2. Text Retrieval Neural Network Model Based on Overlay Concept Judgment

[0050] Text retrieval neural network models accept an overlay of concept sets or text as input and contain an encoder.

[0051] For text x1,...,x |x| Where |x| represents the length of the text, the encoder captures semantic information in the text, and obtains the semantic feature representation of the text:

[0052] H = Encoder(x1,...,x) |x| );

[0053] For the superimposed concept set "c i but not c j To speed up the calculation, all identical exclusion concepts are merged. For exclusion concept c... j The corresponding set of superimposed concepts is written as: "c j |c1,c2,…,c n , where "|" indicates exclusion, the expression refers to the exclusion of concept c j When excluded, the concept set {c1,c2,…,c n The encoder captures the semantic information of the corresponding superimposed concept set, obtaining the semantic feature representation of the superimposed concepts:

[0054] C = Encoder(cj |c1...,c n );

[0055] Deep neural network encoder (Encoder) uses "c j |c1,…,c n The input is "" which encodes the superimposed concept set, and the output C is the representation of the superimposed concept set.

[0056] Finally, the semantic feature representation H of the text and the semantic feature representation C of the superimposed concepts are used to calculate the inner product to obtain a confidence score. Text is then selected for annotation based on the confidence score.

[0057] 3. Training using large-scale network data

[0058] In order to enable the text retrieval neural network model based on superimposed concept discrimination to perform superimposed concept instance retrieval, this embodiment of the invention uses large-scale network data to train the model. The following is the pre-training process.

[0059] (1) Data Collection: Pre-training corpora and entity annotation data were collected from Wikipedia and Wikidata, resulting in more than 30,000 entity categories and 50 million text data. The entity categories in Wikipedia have a wide range and fine granularity, and can be regarded as concepts to describe new categories. Therefore, the above 30,000 entity categories are regarded as concepts.

[0060] (2) Training data construction: First, randomly select a concept from each data point as the target superimposed concept. This data point is the positive example. Then, select exclusion concepts from the sibling categories of the target superimposed concept. Finally, sample several concepts from the parent and child categories of the exclusion concepts to form the superimposed concept set together with the exclusion concepts. At the same time, randomly sample two types of negative examples from all data: text containing the exclusion concepts and text that does not meet the conditions for superimposed concepts.

[0061] (3. Model Training: The model is trained using a contrastive learning approach. The input includes a superimposed concept set, positive examples, and several negative examples. First, the confidence scores of the superimposed concept set and the positive and negative examples are calculated. Finally, the parameters are updated using the contrastive learning loss function.)

[0062] The information extraction model trained by the method of this invention can be used to automatically extract the required information from text, including but not limited to entity information, event information, etc.

[0063] Another embodiment of the present invention provides an active learning device for information extraction based on superposition concept discrimination. It is an electronic device (computer, server, smartphone, etc.) including a memory and a processor. The memory stores a computer program configured to be executed by the processor. The computer program includes instructions for performing the above-described method of the present invention.

[0064] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0065] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A method for generating superimposed concepts based on the elimination method, characterized in that, Includes the following steps: Construct a sequence-to-sequence neural network model, which includes an encoder and a decoder; This sequence-to-sequence neural network model performs entity concept generation tasks. It inputs labeled entities and their context into the encoder, which captures semantic information to obtain semantic representations. The semantic representations are then input into the decoder, which predicts words at each time step to generate text sequences. The conceptual vocabulary corresponding to the labeled entities is obtained through parsing. Concepts of the same category are aggregated to obtain general concepts for that category. The general concept is used to generate a set of superimposed concepts through the process of elimination; the process of generating a set of superimposed concepts through the process of elimination is to generate the set of superimposed concepts in the manner of "A but not B".

2. The method as described in claim 1, characterized in that, The encoder of the sequence-to-sequence neural network model is based on a self-attention mechanism.

3. The method as described in claim 1, characterized in that, The decoder of the sequence-to-sequence neural network model is based on a hybrid attention mechanism.

4. The method as described in claim 1, characterized in that, By matching general concepts in pairs, we can obtain a set of superimposed concepts.

5. An active learning method for information extraction based on superimposed concept discrimination, characterized in that, Includes the following steps: The method for generating superimposed concepts according to any one of claims 1 to 4 is used to generate a set of superimposed concepts; The text retrieval neural network model takes an overlay set of concepts and unlabeled text as inputs, respectively, and obtains the semantic representation of the overlay set of concepts and the semantic representation of the unlabeled text. The confidence score of the superimposed concept set corresponding to the text is obtained by calculating the inner product of the semantic representation of the superimposed concept set and the semantic representation of the unlabeled text. Based on the confidence score, a portion of the text is selected from the unlabeled corpus, labeled, and used to train the information extraction model.

6. The method as described in claim 5, characterized in that, The training process of the text retrieval neural network model adopts a contrastive learning approach, where positive examples are texts containing entities with corresponding superimposed concepts, and negative examples are texts that do not contain entities with corresponding superimposed concepts.

7. The method as described in claim 5, characterized in that, The encoder of the text retrieval neural network model is based on a self-attention mechanism.

8. An active learning device for information extraction based on superimposed concept discrimination, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.

Images