NER model training method combined with large-model single-category labeling
By combining the method of single-category labeling of large models, the unmarked text in the field of traditional Chinese medicine is automatically marked, and the automatic labeling results are used to conduct two rounds of training of the BERT-based NER model, which solves the problem of limited improvement in the recognition effect of target fields in the existing technology and achieves a more efficient naming entity recognition effect.
Patent Information
- Application Number
- CN202411838546.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-04-04
AI Technical Summary
The existing named entity recognition technology has limited improvement in the target field, especially when the data in the non-target field is very different from the target field, and the large language model is not fully understood when identifying multiple categories of entities, resulting in unfavorable output results.
The NER model training method combined with single-category annotation of large models is adopted, and the label-free text in the field of traditional Chinese medicine is automatically marked through a large language model. The automatic labeling results are used to conduct two rounds of training of the BERT-based NER model, and the category classification head and the main classification head are used for identification respectively.
The effect of naming entity recognition models in the target field is improved, especially when there are fewer manual annotation samples, the use of automatic annotation results of large models increases the training data volume and improves the automatic annotation quality, and improves the recognition ability of the model.
Smart Images
Figure CN119761513A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of named entity recognition, and in particular relates to a NER model training method combined with large model single category annotation. Background Art
[0002] Related research mainly involves named entity recognition based on few-shot settings and named entity recognition based on large language models. Currently, there are mainly methods based on meta-learning, contrastive learning, prototype networks, etc. for named entity recognition based on few-shot settings. The main implementation method is to learn from existing named entity recognition data in non-target fields to obtain a model that can be transferred to the target field.
[0003] Although these methods have certain advantages over directly training on a small amount of labeled data for the target task, they also generally have some shortcomings. For example, the data in the non-target domain may be very different from the target domain, resulting in limited improvement in effect.
[0004] In addition, some entity categories in the target domain may not need to be recognized in the non-target domain. In this case, using non-target domain data may result in the final model having poor recognition effect on these entity categories.
[0005] There have also been some research works on using large language models to name entities. They mainly focus on the design of prompt words and the verification of results. These methods usually allow the large model to recognize entities of all target categories at the same time, which will lead to insufficient understanding of each entity category by the large model and have an adverse impact on the output results. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a NER model training method combined with large-model single-category labeling in response to the shortcomings of the background technology; the results of automatic labeling of the large model can be used to improve the effect of the named entity recognition model trained in this case.
[0007] The present invention adopts the following technical solutions to solve the above technical problems: A NER model training method combined with large model single category annotation specifically comprises the following steps; Step 1: Use a large language model to automatically annotate a large amount of unannotated text in the field of traditional Chinese medicine; Step 2: Train the BERT-based NER model based on the annotation results of the large language model; use manually annotated samples to further train the BERT-based NER model; Step 3: After two rounds of training of the BERT-based NER model, the target text to be recognized is input into the model and the entities are predicted using the label sequence output by the main classification head after the model completes the calculation.
[0008] As a further preferred solution of the NER model training method combined with large model single category annotation of the present invention, in step 1, for each target entity category, the large model is called to identify the entity of the category in the unlabeled text, so as to achieve the effect of automatic annotation; specifically, it includes the following: There is a text , for each entity type , all call the big model to identify the List of entities .
[0009] As a further preferred solution of the NER model training method combined with large model single category annotation of the present invention, in step 2, a BERT-based NER model is adopted, and a BiLSTM layer can be added on the basis of BERT. There are k entity categories to be identified. The BERT-based NER model needs to be trained in two rounds, that is, the first round of training is performed using the data automatically annotated by the large model and the second round of training is performed using the manually annotated data; Among them, the category 1, 2, …, k classification heads are used in the first round of training, and the main classification head is used in the second round of training and in actual prediction; The BERT-based NER model recognizes entity locations and categories by labeling BIO sequences. Each classification head predicts a BIO label for each token in the input text. K different label sequences can be obtained through k category classification heads, and a label sequence can also be obtained through the main classification head.
[0010] As a further preferred solution of the NER model training method combined with large model single category labeling of the present invention, in step 2, since each of the k category classification heads predicts only one category, the labels that need to be predicted for each token are only B-{target category}, I-{target category}, and O; suppose for the first tokens, the output vector of BiLSTM is , then the category The label probabilities predicted by the classification head are: ; in, is the parameter matrix corresponding to the classification head; For a containing For a sample of tokens, the loss function of the first round of training is: ; in, For this sample In the label sequence of entity categories The one-hot encoding vector of the actual label of the token, CE means calculating the cross entropy; The second round of training uses manually annotated samples, so there will be no overlapping of entity positions. The common BIO sequence annotation method is used, that is, only one label sequence is predicted to identify all categories of entities in the text; the model training also uses the common cross entropy loss function; for a For a sample of tokens, the loss function is: ; in, For this sample The one-hot encoding vector of the true label of the token, The label probability distribution predicted by the main classification head for this token; The calculation formula is ; in, is the trainable parameter matrix corresponding to the main classification head; since each category corresponds to a B-{category}, I-{category} label, plus the O label, the main classification head needs to predict for each token Different labels.
[0011] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects: 1. The present invention uses the named entity recognition data automatically annotated by the large language model to pre-train the BERT-based model, which can effectively utilize the large language model and the originally unannotated data, so that the BERT-based model has preliminary named entity recognition capabilities. When training named entity recognition models for certain specific fields (such as traditional Chinese medicine, biomedicine, network security, etc.), the trained model may not be very effective due to the small number of manually annotated samples. The present invention can use the results of automatic annotation of the large model to improve the effect of the trained named entity recognition model in this case; 2. The single-category automatic labeling results of large models may have entity overlap, which traditional model construction and training methods cannot cope with. The present invention makes full use of such labeling results by specifically designing the model structure and training method. It can be applied in fields such as traditional Chinese medicine, biomedicine, network security, etc. where there are no large number of manually labeled samples. The automatic labeling of large models increases the amount of available training data, and the quality of automatic labeling is improved through single-category prediction. Ultimately, the effect of the trained named entity recognition model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0013] Figure 1 It is a schematic diagram of the model structure of the BERT-based NER model of the present invention. DETAILED DESCRIPTION
[0014] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings: The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The present invention is described in detail below based on the drawings and preferred embodiments, and the purpose and effect of the present invention will become clearer. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.
[0015] This method combines the generation results of a large language model to improve the performance of a named entity recognition (NER) model based on BERT (Note: it can also be other Transformer Encoder-based models). The task is set as follows: there is a pre-designed list of entity categories to be recognized, sample data with manually labeled entity locations and categories, and unlabeled text data. The goal is to train the model so that when there is new and unseen text, the model can predict the location and category of the entities that appear in it.
[0016] The overall process of the method is as follows. Step 1: Use a large language model to automatically annotate a large amount of unlabeled text in the target field (such as traditional Chinese medicine); Step 2: Use the annotation results of the large model to train the BERT-based NER model; Step 3: Use the model trained in the second step to further train it using manually annotated samples. The automatic annotation in the first step uses a large model to identify each entity category separately, which may cause the positions of entities of different categories to overlap, so the corresponding model and training method are designed for this situation. The following will first introduce how to use a large language model for automatic annotation in the first step, and then introduce the structure and training details of the NER model in the second and third steps.
[0017] In general, this method uses the named entity recognition data automatically annotated by the large language model to pre-train the BERT-based model, which can effectively utilize the large language model and the originally unannotated data, so that the BERT-based model has preliminary named entity recognition capabilities. Based on this, it uses manually annotated data for further fine-tuning, which can achieve better recognition results than ordinary direct fine-tuning methods. Since the automatically annotated data also belongs to the target domain, and the entity category is the same as the target task, there will be no adverse effects due to data inconsistency. In terms of automatic annotation of large language models, the method of annotating by entity category is adopted, which can enable it to have a more comprehensive understanding of each category, thereby outputting better annotation results. Since the automatic annotation method of classification may cause the positions of entities of different categories to overlap, a multi-classification head model that can cope with overlapping entity annotations is designed, so that the data can be fully and effectively utilized even in the case of annotation conflicts.
[0018] When the present invention trains named entity recognition models for certain specific fields (such as traditional Chinese medicine, biomedicine, network security, etc.), the trained model may not be very effective due to the small number of manually labeled samples. This method can use the results of automatic labeling of large models to improve the effect of the named entity recognition model trained in this case. 2) The automatic labeling results of a single category of a large model may have entity overlaps, which traditional model construction and training methods cannot cope with. This solution makes full use of this labeling result by designing the model structure and training method in a targeted manner.
[0019] In this step, for each target entity category, the large model is called to identify the entity of that category in the unlabeled text, thereby achieving the effect of automatic labeling. Specifically, there is a text , for each entity type , all call the big model to identify the List of entities For example, if the text is "Studies have found that the active ingredients of danshen, salvianolic acid and tanshinone, have good pharmacological activity", if the entity category is "traditional Chinese medicine", then the entity list expected to be recognized by the large model is ["danshen"], and if the entity category is "ingredients", then the entity list expected to be recognized is ["danshen", "tanshinone"].
[0020] The prompt template used to call the large model for recognition is: Please identify entities of type "{entity_type}" in the input text. The output format is: ["entity"], if not, output [].
[0021] Here are some examples.
[0022] {demo} The following are the target inputs.
[0023] {target} According to the example, the target input has "{entity_type}" type entities: {entity_type} is replaced with the entity category to be recognized at runtime. {demo} is replaced with a sample of the entity category. For example, the following is a sample of the "Chinese medicine" category: Input: Contains 2-4 parts of dandelion, 2-4 parts of mugwort, and 15-17 parts of perlite.
[0024] The entities of the "Chinese medicine" type include: ["Dandelion", "Artemisia argyi", "Perlite"] Multiple examples can be provided. {target} in the template is replaced with the target text to be recognized.
[0025] The BERT-based NER model is used, and a BiLSTM layer can be added on the basis of BERT. If there are k types of entity categories to be identified, the model structure is as follows: Figure 1 As shown in the figure. The model needs two rounds of training, i.e. the first round of training uses data automatically annotated by the large model and the second round of training uses manually annotated data. The category 1, 2, …, k classification heads are used in the first round of training, and the main classification head is used in the second round of training and in actual prediction. The model recognizes the location and category of entities by BIO sequence annotation. Each classification head predicts a BIO label for each token in the input text. Finally, k different label sequences can be obtained through k category classification heads, and a label sequence can also be obtained through the main classification head.
[0026] During the first round of training, since the large model recognizes entities of each category separately, the positions of entities of different categories may overlap. For example, for the text "which contains 2-4 parts of dandelion, 2-4 parts of mugwort, and 15-17 parts of perlite", "dandelion", "mugwort", and "perlite" may be marked as both "traditional Chinese medicine" type entities and "ingredient" type entities during automatic labeling. In this case, it is impossible to use only one classification head for sequence labeling prediction, so k classification heads are used, each of which corresponds to an entity category and predicts a label sequence. For the above example, the label sequences predicted by the classification head corresponding to the category "traditional Chinese medicine" and the classification head corresponding to the category "ingredient" should mark "dandelion", "mugwort", and "perlite" as entities.
[0027] Since each of the k-category classification heads predicts only one category, the labels they need to predict for each token are only B-{target category}, I-{target category}, and O. tokens, the output vector of BiLSTM is , then the category The label probabilities predicted by the classification head are: ; in, is the parameter matrix corresponding to the classification head.
[0028] For a containing For a sample of tokens, the loss function of the first round of training is: ; in, For this sample In the label sequence of entity categories The one-hot encoding vector of the actual label of the token, CE means calculating the cross entropy.
[0029] The second round of training uses manually annotated samples, so there will be no overlap of entity positions. Therefore, the common BIO sequence annotation method is used, that is, only one label sequence is predicted to identify all categories of entities in the text. The model training also uses the common cross entropy loss function. For a sample of tokens, the loss function is: ; in, For this sample The one-hot encoding vector of the true label of the token, The label probability distribution predicted by the main classification head for this token. The calculation formula is ; in, is the trainable parameter matrix corresponding to the main classification head. Since each category corresponds to a B-{category}, I-{category} label, plus the O label, the main classification head needs to predict for each token Different labels.
[0030] After two rounds of model training, for the target text that needs to be recognized, after the model completes the calculation, the label sequence output by the main classification head can be used to predict the entities in it.
[0031] The experiment done by the method of the present invention is as follows: The experimental goal is to identify five types of entities: traditional Chinese medicine, traditional Chinese medicine waste, ingredients, activity, and use. The data used is text extracted from patents related to traditional Chinese medicines such as Danshen, Panax notoginseng, and ginseng. 1,683 paragraphs were manually annotated and divided into training, development, and test sets, containing 100, 100, and 1,483 paragraphs respectively, and each paragraph contains multiple sentences. In addition, there are about 99,000 paragraphs of text with or without annotations.
[0032] Using the designed method, the unlabeled text is first automatically annotated using the Tongyi Qianwen 2 large model Qwen2-7B-Instruct, and then the NER model based on BERT-BiLSTM is trained in the first round using the samples obtained by automatic annotation. The model is then trained in the second round using the manually annotated training set, and finally the obtained model is used to predict the samples in the test set and the results are evaluated. In addition, the model trained using only the manually annotated training set is used as a reference. The experimental results of named entity recognition are shown in Table 1.
[0033] Table 1 method Precision Recall F1 BERT-BiLSTM (Traditional Method) 0.7186 0.7298 0.7242 This method 0.7450 0.7612 0.7530 The named entity recognition model trained by this method can have better recognition effect than the model trained directly with manually annotated data, especially when the amount of manually annotated data is small. After experimental testing, for five types of entities, including traditional Chinese medicine, traditional Chinese medicine waste, ingredients, activity, and use, 100 randomly selected manually annotated training sample paragraphs and 1484 randomly selected test sample paragraphs were used. The F1 of the recognition result of fine-tuning the BERT-based model using manually annotated data was 0.724, and the F1 of the model obtained by this method was 0.753.
[0034] The present invention can be applied in fields such as traditional Chinese medicine, biomedicine, network security, etc. where there are no large number of manually labeled samples. The amount of available training data is increased through automatic labeling of large models, and the quality of automatic labeling is improved through single-category prediction. Ultimately, the effect of the trained named entity recognition model can be improved.
[0035] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention should be included in the protection scope of the invention. All technical features in this embodiment can be freely combined according to actual needs.
[0036] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A NER model training method combined with large model single category annotation, characterized by: The specific steps include: Step 1: Use a large language model to automatically annotate a large amount of unannotated text in the field of traditional Chinese medicine; Step 2: Train the BERT-based NER model based on the annotation results of the large language model; use manually annotated samples to further train the BERT-based NER model; Step 3: After two rounds of training of the BERT-based NER model, the target text to be recognized is input into the model and the entities are predicted using the label sequence output by the main classification head after the model completes the calculation.
2. According to claim 1, a NER model training method combined with large model single category labeling is characterized by: In step 1, for each target entity category, the large model is called to identify the entity of that category in the unlabeled text, so as to achieve the effect of automatic labeling; specifically, it includes the following: There is a text , for each entity type , all call the big model to identify the List of entities .
3. The NER model training method combined with large model single category labeling according to claim 1, characterized in that: In step 2, a BERT-based NER model is used, and a BiLSTM layer can be added on the basis of BERT. There are k entity categories to be identified. The BERT-based NER model needs to be trained in two rounds, i.e., the first round of training is performed using data automatically annotated by the large model and the second round of training is performed using data manually annotated; Among them, the category 1, 2, …, k classification heads are used in the first round of training, and the main classification head is used in the second round of training and in actual prediction; The BERT-based NER model recognizes entity locations and categories by labeling BIO sequences. Each classification head predicts a BIO label for each token in the input text. K different label sequences can be obtained through k category classification heads, and a label sequence can also be obtained through the main classification head.
4. The NER model training method combined with large model single category labeling according to claim 3, characterized in that: In step 2, since each of the k-category classification heads predicts only one category, the labels that need to be predicted for each token are only B-{target category}, I-{target category}, and O. tokens, the output vector of BiLSTM is , then the category The label probabilities predicted by the classification head are: ； in, is the parameter matrix corresponding to the classification head; For a containing For a sample of tokens, the loss function of the first round of training is: ； in, For this sample The first The one-hot encoding vector of the actual label of the token, CE means calculating the cross entropy; The second round of training uses manually annotated samples, so there will be no overlapping of entity positions. The common BIO sequence annotation method is used, that is, only one label sequence is predicted to identify all categories of entities in the text; the model training also uses the common cross entropy loss function; for a For a sample of tokens, the loss function is: ； in, For this sample The one-hot encoding vector of the true label of the token, The label probability distribution predicted by the main classification head for this token; The calculation formula is ； in, is the trainable parameter matrix corresponding to the main classification head; since each category corresponds to a B-{category}, I-{category} label, plus the O label, the main classification head needs to predict for each token Different labels.