General named entity recognition method based on continuous learning and prompt template
Through the SANER framework and continuous learning method, the problems of poor robustness and resource waste in the NER model when incrementing cross-domain data sets are solved. The prompt template and elastic weight consolidation regularization are adopted to realize cross-domain entity recognition and resource optimization. Easing catastrophic forgetting and improving the robustness and efficiency of the model.
Patent Information
- Application Number
- CN202310581034.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2024-11-22
AI Technical Summary
Existing NER models are poorly robust when incrementing across domain datasets, fail to effectively utilize knowledge of pre-trained language models, and require frequent expansion or retraining of the model to support new entity types, resulting in waste of resources and catastrophic forgetting question.
A general NER framework, SANER, is proposed, which performs task transformation through prompt templates, combines continuous learning and elastic weight consolidation regularization method, uses PyTorch framework and BART pre-trained model, and adopts an adaptive two-stage inference method to avoid model expansion and retraining, alleviating the catastrophic forgetting problem.
The ability to identify new entity types under cross-domain settings is realized, reducing resource waste, alleviating model forgetting problems, and improving model robustness and efficiency.
Smart Images

Figure CN119005187A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer technology, and in particular to the application of deep learning methods in the field of computer technology. The purpose is to make full use of the advantages of continuous learning and prompt template methods, make full use of the knowledge obtained by pre-training language models, and solve the problem of poor robustness of NER models when cross-domain data sets are incremented. Background Art
[0002] Named Entity Recognition (NER) is a task to locate named entities in unstructured text and classify them into predefined types (such as names, locations, organizations, etc.). Previous work has been studying class-incremental learning for NER to support the growing demand for identifying new entity types in the real world. However, according to research, existing methods still have at least three problems: (1) Most methods only consider class increments in single-domain datasets without cross-domain, which makes the model less robust in practical scenarios; (2) Existing methods treat the NER task as a sequence labeling problem, where the entity type label is only used as a class index (expressed as one-hot encoding), lacking prior semantic knowledge of the entity type label, and unable to fully utilize the knowledge obtained by the pre-trained language model; (3) Under sequence labeling, existing methods add or expand the original output layer to enable the model to support new entity types, which requires training a new model when a new entity type appears, resulting in a waste of resources. Summary of the invention
[0003] The main problem solved by the present invention is that the deployed NER system does not support the recognition of cross-domain entities that appear continuously.
[0004] The technical solution of the present invention is a universal NER framework SANER, which can continuously identify new entity types regardless of whether they are from the same domain dataset. It includes the following steps: 1. NER task conversion through prompt templates; 2. Adding categories in domain and cross-domain settings through continuous learning; 3. The model determines whether the text span belongs to a predefined entity type; 4. If the model belongs, it determines which predefined entity it corresponds to.
[0005] The characteristics of the present invention are: 1. The continuous NER of Monaikul et al. needs to rely on a large amount of annotated text of the previous entity at each step, while we only need a small number of annotated samples of past entities; 2. We do not need to expand the neural network layer, nor do we need to copy a new encoder layer every time to build a new model. Our method is always trained on the same model, so we can avoid the waste of storage space and do not need to dynamically adjust and retrain the output layer of the model every time; 3. Since the catastrophic forgetting problem causes the performance of the NER model to drop rapidly in continuous learning, we innovatively combine typical strategies in continuous learning to alleviate this problem.
[0006] The beneficial effects of the present invention are as follows: a general NER model-like incremental learning framework is proposed, which can support the recognition of new entity types regardless of intra-domain or cross-domain settings. The problem of inconsistent label sets between training and test sets is solved. Then a sample replay method based on representative difficulty and an EWC-based regularization method are proposed to alleviate the catastrophic forgetting problem encountered by the model during continuous learning. Finally, an adaptive two-stage reasoning method of pre-training generation task is used to predict the entity text span and entity type in the input text. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 The training phase of the SNAER model in the present invention
[0008] Figure 2 The inference stage of the SNAER model in the present invention DETAILED DESCRIPTION
[0009] Several key technologies involved in this invention are: Pytorch framework, Bart pre-training model, Prompt template method, and Continue learning method. The following are some explanations:
[0010] 1. Pytorch framework
[0011] PyTorch is an open source Python machine learning library based on Torch for applications such as natural language processing. PyTorch can be seen as numpy with GPU support, but it can also be seen as a powerful deep neural network with automatic differentiation capabilities.
[0012] 2. Bart pre-trained model
[0013] BART absorbs the characteristics of BERT's bidirectional encoder and GPT's left-to-right decoder. It is built on the basis of the standard seq2seq Transformer model, which makes it more suitable for text generation scenarios than BERT. In addition, it also has more bidirectional context information than GPT. While making progress in generation tasks, it can also achieve SOTA in some text understanding tasks.
[0014] 3. Prompt template method
[0015] Prompt Templates is a reproducible way to generate prompts. It contains a text string that can accept a set of parameters from the end user and generate a prompt. Prompt Templates can contain instructions, a small number of examples, and a question to the language model. We can use Prompt Templates technology to guide the language model to generate higher quality text, so as to better complete our tasks.
[0016] 4. Continue learning method
[0017] We humans have the ability to apply the knowledge of one task to another task, and we will not forget how to do the previous task when learning the next task. This ability is called continuous learning (continual learning / life-long learning). This ability can be summed up in two main issues: how to use the experience of the previous task to learn the current task faster and better; and how to not forget the previously learned tasks when learning the current task.
[0018] Usage Examples
[0019] 1. We use the prompt template to convert the sequence labeling task into a text generation task and use two tasks to fine-tune the Bart model. One is a supervised text generation task and the other is a self-supervised masked language model task.
[0020] 2. At the same time, considering the sampling of negative samples, we use the template "<candidate entity> is not a predefined entity", so that we can train negative samples at the same time. We feed the input X into the encoder of BART to obtain the hidden representation of the sentence.
[0021] 3. After converting the NER task into the task required by the pre-trained language model, SANER can continuously complete the update based on the data in the new task to recognize new entity types. However, at the same time, our model will suffer from the problem of catastrophic forgetting. Therefore, we propose a new continuous learning method to alleviate this problem, which mainly includes two parts:
[0022] 3.1. One part is the replay method based on representative difficulty. The core is to select a small proportion of old data from past tasks to retain the learned knowledge. When new data arrives, it will be trained together with the new data to avoid losing the knowledge of old tasks as much as possible.
[0023] 3.2. The other part is a regularization method based on elastic weight consolidation (EWC). The main idea behind it is to restrict parameter updates so as to minimize the changes of parameters that have a significant impact on past tasks.
[0024] 4. After the model training is completed, we generate all possible text spans and place them in the corresponding positions in the predefined templates. Next, we use the fine-tuned pre-trained language model to calculate the scores of each template (including negative sample templates and positive sample templates).
[0025] 5. We compare the score of the negative sample with the maximum score of the positive sample. If the score of the negative sample is larger, the text span to be evaluated is determined to be a non-predefined entity; if the score of the positive sample is larger, the mask prediction ability of the pre-trained language model is used to calculate the score of each predefined entity type.
Claims
1. A universal named entity recognition method based on continuous learning and prompt templates. The technical solution of the present invention is a universal named entity recognition method framework SANER, which is characterized by It can continuously identify new entity types regardless of whether they are from the same domain dataset. It includes the following steps:
1. NER task conversion through prompt templates 2. Increase the category of entity types in the domain and cross-domain settings through continuous learning 3. The model determines whether the text span belongs to a predefined entity type 4. If it does, determine which predefined entity it corresponds to.
2. A general named entity recognition method based on continuous learning and prompt templates according to claim 1, characterized in that The sequence labeling task is converted into a text generation task using a prompt template, and two tasks are used to fine-tune the Bart pre-trained model. One is a supervised text generation task, and the other is a self-supervised masked language model task. At the same time, considering the sampling of negative samples, the template "<candidate entity> is not a predefined entity" is used so that negative samples can be trained at the same time.
3. A method for universal named entity recognition based on continuous learning and prompt templates according to claim 1, in order to solve the catastrophic forgetting problem suffered by SANER in the incremental learning process of NER. The method is characterized in that a new continuous learning method is proposed to alleviate the catastrophic forgetting problem, including: One is a sample replay method based on representative difficulty, and the other is a regularization method based on elastic weight consolidation (EWC).
4. A method for universal named entity recognition based on continuous learning and prompt templates according to claim 1, characterized in that: To evaluate an entity, the model generates all possible text spans and places them in the corresponding positions in the predefined templates. Next, the fine-tuned pre-trained language model is used to calculate the score of each template (including negative and positive templates). The score of the negative sample is compared with the maximum score of the positive sample. If the score of the negative sample is larger, the entity to be evaluated is determined to be an unplanned entity; if the score of the positive sample is larger, the mask prediction ability of the pre-trained language model is used to calculate the score of each predefined entity type.
Citation Information
Cited By
Text data labeling method and system based on weak supervised learning
CN119669477A
