Templated named entity recognition method based on prompt learning and antagonism training

By converting NER tasks to seq2seq tasks and combining adversarial training, the problems of accuracy and robustness of traditional methods in a small sample environment are solved, and efficient naming entity recognition is achieved to adapt to the needs of low resource environments.

CN119005193APending Publication Date: 2024-11-22HUAIYIN INSTITUTE OF TECHNOLOGY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411257297.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The traditional named entity recognition (NER) method performs poorly with few samples or no labeled data, making it difficult to achieve high recognition accuracy and robustness, and the inference speed is slow, making it difficult to meet the needs of real-time applications.

Method used

Convert NER tasks into sequence-to-sequence (seq2seq) tasks, use pre-trained language model T5 for prompt learning and adversarial training, and use the templated prompt input set and adversarial samples to perform supervised comparison learning and mixed adversarial training to improve the model robustness and generalization ability.

Benefits of technology

It significantly improves the accuracy and robustness of named entity recognition, adapts to low resource environments, improves the generalization ability of the model in different data distribution and field transfer scenarios, and realizes fast and efficient named entity recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005193A_ABST
    Figure CN119005193A_ABST
Patent Text Reader

Abstract

The invention discloses a template named entity recognition method based on prompt learning and antagonism training. The method comprises the following steps: firstly, generating a synthetic sentence using a template, converting an NER task into a seq2seq task, defining prompt templates for different types of entities, converting a target entity into a sentence in a specific format, and combining the sentence with an input sentence to generate prompt input; secondly, a prompt input set is input into the T5, and fine adjustment is conducted on the model through prompt learning; a projection gradient descent method is used to generate an adversarial sample, and supervised contrast learning and hybrid adversarial training are carried out. And combining the generated adversarial sample with the original sample to form a new training set, inputting the new training set to T5 for joint training, further improving the performance through model fusion, and comprehensively evaluating the model. And finally, inputting a new sentence, generating prompt input through the prompt template, reasoning by using the T5 model after fine tuning and antagonism training, and identifying and marking named entities in the sentence.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of natural language processing (NLP), and relates to a method for named entity recognition (NER), and in particular to a templated named entity recognition method based on prompt learning and adversarial training. Background Art

[0002] Named Entity Recognition (NER) is an important task in the field of natural language processing. It is traditionally implemented through sequence labeling methods, that is, classifying each token in the input sentence to determine whether it belongs to a specific named entity category. Common NER methods include conditional random fields (CRFs), hidden Markov models (HMMs), and neural network-based methods such as long short-term memory networks (LSTMs) and convolutional neural networks (CNNs). These methods rely on training on a large amount of labeled data and can achieve good results when there is abundant labeled data. However, in the case of few samples or no labeled data, the performance of these methods is often unsatisfactory, and it is difficult to achieve high recognition accuracy and robustness.

[0003] In recent years, pre-trained language models (such as BERT, T5) have demonstrated strong performance in natural language processing tasks. The model learns rich language representations by pre-training on a large amount of text data, and then fine-tunes it in downstream tasks to achieve excellent performance. However, traditional NER methods are difficult to apply directly to these generative models, because NER tasks require separate classification of each token, while generative models are usually used to generate complete sentences or texts. In addition, existing methods are slow in the reasoning process and cannot meet the needs of real-time applications. At the same time, they perform poorly in a few-sample environment and lack effective use of negative samples. Summary of the invention

[0004] Purpose of the invention: To solve the above problems, the present invention proposes a templated NER method based on prompt learning and adversarial training, which significantly improves the accuracy, robustness and generalization ability of NER tasks, bringing new breakthroughs in the fields of natural language processing and so on.

[0005] Technical solution: The present invention discloses a templated named entity recognition method based on prompt learning and adversarial training, comprising the following steps:

[0006] Step 1: Convert the NER task to a seq2seq task, define prompt templates for different types of entities, convert the target entity into a sentence in a specific format, and combine it with the input sentence to generate a prompt input set;

[0007] Step 2: Input the prompt input set into the pre-trained language model T5 and fine-tune it through prompt learning so that the model can generate sentences containing entity annotation information based on the prompts;

[0008] Step 3: Use the projected gradient descent method PGD to impose small perturbations on the prompt input set to generate adversarial samples, and perform supervised contrastive learning and hybrid adversarial training;

[0009] Step 4: Combine the generated adversarial samples with the original samples to form a new training set, which is input into the pre-trained language model T5 for joint training. The performance is further improved through model fusion, and the model is comprehensively evaluated.

[0010] Step 5: Input a new sentence, generate prompt input through the prompt template, use the fine-tuned and adversarially trained T5 model for reasoning, and identify and annotate the named entities in the sentence.

[0011] Furthermore, the specific method of step 1 is:

[0012] Step 1.1: Convert the NER task from a traditional sequence labeling task to a sequence-to-sequence (seq2seq) task, converting the input sentence into an output sentence containing the target entity labeling information;

[0013] Step 1.2: Determine the named entity type that needs to be identified, and define a corresponding prompt template for each entity type. The template format is as follows:

[0014] Positive sample template: "In the sentence, <sentence>,the <entity-type>entities are=><target_entity> ”;

[0015] Negative sample template: "In the sentence,' <sentence>',the <entity-type>entities are=>None";

[0016] Using the hint template, the target entity in the input sentence is embedded into a sentence with a specific format;

[0017] Step 1.3: Generate positive and negative samples

[0018] For each entity type in each training instance, generate a positive sample and replace the target entity in the input sentence with the positive sample template;

[0019] For non-existent entity types, negative samples are generated by combining the input sentence with the negative sample template to indicate that there is no entity of that type;

[0020] Step 1.4: Combine the generated special format sentence with the original input sentence to generate a prompt input;

[0021] Step 1.5: Build the prompt input set and divide it into training set, validation set and test set.

[0022] Furthermore, the specific method of step 2 is:

[0023] Step 2.1: Use the TokenizerFast object of the pre-trained language model T5 to encode the training set and generate a tensor X that the model can process;

[0024] Step 2.2: Forward propagate the encoding through the multi-layer Transformer of the pre-trained language model T5, and output the logits score vector for each position;

[0025] Step 2.2.1: Set the mask position mask_idx, which is the index of the entity position that the model needs to predict;

[0026] Step 2.2.2: Pass the input tensor X through the weight matrix W 1 and the bias vector b 1 Perform a linear transformation and then calculate the hidden layer output H through the activation function f:

[0027] H=f(W 1 X+b 1 )

[0028] Step 2.2.3: Pass the output H of the hidden layer through the weight matrix W 2 and the bias vector b 2 Perform a linear transformation and then use the activation function g to get the final output O:

[0029] O=g(W 2 X+b 2 )

[0030] Step 2.2.4: According to the mask position, convert the logits into a probability distribution p through the softmax function ij :

[0031]

[0032] Among them, p ij 、z ij are the predicted probability of sample i in category j and the logit value of sample i in category j, respectively, and C is the number of categories;

[0033] Step 2.2.5: Compare the model prediction results with the true label y and calculate the cross entropy loss:

[0034]

[0035] Among them, L is the loss function value, N is the number of samples, and y ij One-hot encoding of the true label;

[0036] Step 2.3: Use the back-propagation algorithm to calculate the gradient of the loss function and use AdamW to update the T5 parameters according to the gradient.

[0037] Furthermore, the specific process in step 2.3 is:

[0038] Step 2.3.1: Calculate the gradient of the loss function:

[0039]

[0040] Among them, θ is the model parameter;

[0041] Step 2.3.2: The back-propagation algorithm calculates the gradient layer by layer through the chain rule, propagating from the output layer to the input layer;

[0042] Output layer gradient calculation:

[0043]

[0044] Hidden layer gradient calculation:

[0045]

[0046] Step 2.3.3: Use AdamW optimizer to update model parameters according to the gradient;

[0047] Initialize the first-order moment estimate m 0 = 0 and the second moment estimate v 0 = 0, at each time step t, calculate the gradient of the objective function L(θ) with respect to the parameter θ:

[0048] Update first-order moment estimate: m t =β 1 m t-1 +(1-β 1 ) t , where β 1 is the decay rate of the first-order moment estimate, m t is the first-order moment estimate at the current time t, m t-1 is the first-order moment estimate at the previous moment t-1;

[0049] Update the second moment estimate: where v t is the second-order moment estimate at the current time t, v t-1 is the second-order moment estimate of the previous moment t-1;

[0050] Compute bias-corrected first- and second-order moment estimates:

[0051] Use the corrected moment estimate to update the parameters and add a weight decay term:

[0052]

[0053] Among them, θ t is the model parameter of the tth iteration, θ is the weight decay coefficient, η is the learning rate, and a is a small positive constant used to ensure numerical stability.

[0054] Furthermore, the specific method of step 3 is:

[0055] Step 3.1: Starting from the original training set, the initial control sample X' 0 , copy each input example X as the initial adversarial sample X' 0 :

[0056] X' 0 =X

[0057] Step 3.2: Set the allowed perturbation range ∈, that is, the maximum amplitude of the perturbation applied to the input data;

[0058] Step 3.3: Calculate the gradient of the loss function;

[0059] Step 3.3.1: Perform forward propagation on the current adversarial sample X′ and calculate the model's predicted output y′:

[0060]

[0061] Among them, θ is the model parameter, is the forward propagation function of the model;

[0062] Step 3.3.2: Calculate the loss function L(y′,y):

[0063]

[0064] Among them, y is the true label, y′ is the prediction result of the model, and N batch is the total number of samples in the current adversarial training batch, C is the total number of categories, and y ij is the true label of sample i in category j, p ij is the predicted probability of sample i in category j;

[0065] Step 3.3.3: Calculate the gradient of the loss function with respect to the adversarial example

[0066]

[0067] Step 3.4: Update the adversarial example based on the calculated gradient:

[0068]

[0069] Among them, X′ t , X′ t+1 are the current adversarial sample and the updated adversarial sample respectively, and α is the perturbation step size applied in each iteration;

[0070] Step 3.5: Ensure that the updated adversarial example is within the allowed perturbation range:

[0071] X t+1 =clip(X t+1 , X-∈, X+∈)

[0072] The clip(·) function is used to limit the adversarial sample to the range of the original sample X. Repeat the above steps until the predetermined number of iterations K is reached.

[0073] Step 3.6: Construct contrastive learning pairs and calculate contrastive learning loss:

[0074] Construct a contrastive learning pair for each adversarial sample X′ and the corresponding original sample X, denoted as (X, X′). Generate positive sample pairs (X, X) and negative sample pairs (X, X″), where X″ is a perturbed version of the other sample; calculate the contrastive learning loss to make the distance between positive sample pairs closer and the distance between negative sample pairs farther:

[0075]

[0076] Among them, h i 、h j is a sample X i , X j The representation vector obtained by the encoder, sim(u,v)=u T v / (||u||||v||) is the cosine similarity of vectors u and v, τ is the temperature hyperparameter that controls the scaling of the similarity, ∑ k exp(sim(h i ,h k ) / τ) is the sum of all negative sample pairs in the current mini-batch;

[0077] Step 3.7: Generate joint adversarial samples and calculate joint adversarial loss:

[0078] Use a variety of adversarial attack methods to generate different types of adversarial sample sets X adv1 , X adv2 , …, X advM , combine them with the original sample X to form a joint adversarial sample set X joint =X∪X adv1 ∪X adv2 ∪…∪X advM ; For each joint adversarial example x * ∈X joint , calculate its corresponding adversarial loss function:

[0079] L adv (x * )=L(x * ,y adv ;θ)

[0080] Among them, L adv is the adversarial loss function, y adv is x * The corresponding true label, θ is the model parameter; the final joint adversarial loss is the mean of all sample losses:

[0081]

[0082] Among them, N adv is the total number of joint adversarial sample sets;

[0083] By using the contrastive learning loss L contrast and the joint adversarial loss L joint Combined, the model is able to simultaneously learn adversarially robust representations and enhance defense capabilities against different types of adversarial examples.

[0084] Furthermore, the specific method of step 4 is:

[0085] Step 4.1: Combine the joint adversarial sample set with the original sample X in the training set to form a new training set;

[0086] Step 4.2: Use the Tokenizer of the T5 model to jointly train the new training set, and use different training parameters or different initialization methods to train i independent T5 models;

[0087] Step 4.3: Model Fusion

[0088] Step 4.3.1: Train i T5 models independently using different validation sets; get the prediction result prediction of each model i and the corresponding weight w i ;

[0089] Step 4.3.2: Based on the prediction results of multiple models and the weights shown on the validation set, calculate the weighted value as the final prediction:

[0090] Step 4.3.3: Select dynamic quantization, static quantization, or hybrid quantization, quantize the trained T5 model, and test the performance of the quantized model.

[0091] Furthermore, the specific method of step 5 is:

[0092] Step 5.1: Input a new unlabeled sentence as the input of the model, combine the new sentence with the predefined prompt template to generate prompt input;

[0093] Step 5.2: Use the T5 model's Tokenizer to encode the generated prompt input into a tensor format that the model can process, input the encoded prompt into the fine-tuned and controlled trained T5 model for reasoning, and output a sentence containing entity annotation information;

[0094] Step 5.3: Parse the model output, extract named entities, and apply the identified and annotated named entities to the original sentence based on the model parsing results to generate the final named entity recognition results.

[0095] The present invention adopts the above technical solution and has the following beneficial effects:

[0096] Convert the NER task into a seq2seq task to avoid generating all possible N-grams. Use a templated approach to generate prompt inputs and design standardized prompts to reduce the language space. Use prompt learning to train the T5 model to generate labeled outputs based on a small amount of labeled data, leveraging T5's advantages in seq2seq tasks and enabling pre-trained language models to better adapt to small sample environments.

[0097] The present invention combines prompt learning and adversarial training to significantly improve the robustness and recognition accuracy of the model in a few-sample environment; adversarial samples are introduced during the training process, and the robustness of the model in the face of noise and data disturbances is enhanced through supervised contrastive learning and hybrid adversarial training. The adversarial training method helps the model learn a more robust feature representation, and model fusion further improves the generalization ability of the model in different data distributions and domain transfer scenarios. In the reasoning stage, it can quickly and efficiently identify and annotate named entities in sentences, significantly improving the accuracy and stability of named entity recognition, adapting to low-resource environments, and promoting the development of NER technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 It is a step diagram of the present invention;

[0099] Figure 2 It is the overall flow chart of the present invention;

[0100] Figure 3 A schematic diagram of the method of the template of the present invention;

[0101] Figure 4 This is the projection gradient descent flow chart of the present invention. DETAILED DESCRIPTION

[0102] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0103] The present invention discloses a templated named entity recognition method based on prompt learning and adversarial training, comprising the following steps:

[0104] Step 1: Convert the NER task to a seq2seq task, define prompt templates for different types of entities, convert the target entity into a sentence in a specific format, and combine it with the input sentence to generate the prompt input.

[0105] Step 1.1: Convert the NER task from a traditional sequence labeling task to a sequence-to-sequence (seq2seq) task. Convert the input sentence to an output sentence containing the target entity labeling information;

[0106] Step 1.2: Determine the named entity type that needs to be identified, and define a corresponding prompt template for each entity type. The template format is as follows:

[0107] Positive sample template: "In the sentence, <sentence>,the <entity-type>entities are=><target_entity> ”

[0108] Negative sample template: "In the sentence,' <sentence>',the <entity-type>entities are=>None"

[0109] The target entity in the input sentence is embedded into a sentence with a specific format using a hint template.

[0110] Step 1.3: Generate positive and negative samples

[0111] For each entity type in each training instance, generate a positive sample and replace the target entity in the input sentence with the positive sample template. For example, the input sentence is "John lives in the UK", and the generated positive sample is:

[0112] "In the sentence,'John lives in the UK',the location entities are=>UK"

[0113] "In the sentence,'John lives in the UK',the person entities are=>John".

[0114] For non-existent entity types, negative samples are generated by combining the input sentence with the negative sample template to indicate that there is no entity of this type. For example, if the input sentence is "John lives in the UK", the generated negative sample is:

[0115] "In the sentence,'John lives in the UK',the miscellaneous entities are=>None".

[0116] Step 1.4: Combine the generated specially formatted sentence with the original input sentence to generate a prompt input.

[0117] Step 1.5: Build the prompt input set and divide it into training set, validation set and test set.

[0118] Step 2: Input the prompt input set into T5 and fine-tune it through prompt learning so that the model can generate sentences containing entity annotation information based on the prompts.

[0119] Step 2.1: Use the TokenizerFast object of the pre-trained language model T5 to encode the training set and generate a tensor X that the model can process.

[0120] Step 2.2: Forward propagate the encoding through the multi-layer Transformer of the T5 model and output the logits score vector for each position.

[0121] Step 2.2.1: Set the mask position (mask_idx), which is the index of the entity position that the model needs to predict.

[0122] Step 2.2.2: Pass the input tensor X through the weight matrix W 1 and the bias vector b 1 Perform a linear transformation and then calculate the hidden layer output H through the activation function f:

[0123] H=f(W 1 X+b 1 )

[0124] Step 2.2.3: Pass the output H of the hidden layer through the weight matrix W 2 and the bias vector b 2 Perform a linear transformation, and then use the activation function g (such as softmax) to get the final output O:

[0125] O=g(W 2 X+b 2 )

[0126] Step 2.2.4: According to the mask position, convert the logits into a probability distribution p through the softmax function ij :

[0127]

[0128] Among them, p ij 、z ij are the predicted probability of sample i in category j and the logit value of sample i in category j, respectively, and C is the number of categories.

[0129] Step 2.2.5: Compare the model prediction results with the true label y and calculate the cross entropy loss:

[0130]

[0131] Among them, L is the loss function value, N is the number of samples, and y ij is the one-hot encoding of the true label.

[0132] Step 2.3: Use the back-propagation algorithm to calculate the gradient of the loss function and use AdamW to update the T5 model parameters according to the gradient.

[0133] Step 2.3.1: Calculate the gradient of the loss function:

[0134]

[0135] Among them, θ is the model parameter;

[0136] Step 2.3.2: The back-propagation algorithm calculates the gradient layer by layer through the chain rule and propagates from the output layer to the input layer.

[0137] Output layer gradient calculation:

[0138]

[0139] Hidden layer gradient calculation:

[0140]

[0141] Step 2.3.3: Update the model parameters based on the gradients using the AdamW optimizer.

[0142] Initialize the first-order moment estimate m 0 = 0 and the second moment estimate v 0 = 0, at each time step t, calculate the gradient of the objective function L(θ) with respect to the parameter θ:

[0143] Update first-order moment estimate: m t =β 1 m t-1 +(1-β 1 ) t , where β 1 is the decay rate of the first-order moment estimate, m t is the first-order moment estimate at the current time t, m t-1 is the first-order moment estimate at the previous time t-1.

[0144] Update the second moment estimate: where v t is the second-order moment estimate at the current time t, v t-1 is the second-order moment estimate of the previous time t-1.

[0145] Compute bias-corrected first- and second-order moment estimates:

[0146] Use the corrected moment estimate to update the parameters and add a weight decay term:

[0147]

[0148] Among them, θ t is the model parameter of the tth iteration, λ is the weight decay coefficient, η is the learning rate, and a is a small positive constant used to ensure numerical stability.

[0149] Step 3: Generate adversarial samples by applying small perturbations on the prompt input set using the projected gradient descent (PGD) method, generate adversarial samples, and perform supervised contrastive learning and hybrid adversarial training.

[0150] Step 3.1: Starting from the original training set, the initial control sample X′ 0 , copy each input example X as the initial adversarial sample X′ 0 :

[0151] X′ 0 =X

[0152] Step 3.2: Set the allowed perturbation range ∈, that is, the maximum amplitude of the perturbation applied to the input data.

[0153] Step 3.3: Calculate the gradient of the loss function

[0154] Step 3.3.1: Perform forward propagation on the current adversarial sample X′ and calculate the model's predicted output y′:

[0155]

[0156] Where θ is the model parameter, is the forward propagation function of the model.

[0157] Step 3.3.2: Calculate the loss function L(y′, y),

[0158]

[0159] Among them, y is the true label, y′ is the prediction result of the model, and N batch is the total number of samples in the current adversarial training batch, C is the total number of categories, and y ij is the true label of sample i in category j, p ij is the predicted probability of sample i in category j;

[0160] Step 3.3.3: Calculate the gradient of the loss function with respect to the adversarial example

[0161]

[0162] Step 3.4: Update the adversarial example based on the calculated gradient:

[0163]

[0164] Among them, X′ t , X′ t+1 are the current adversarial sample and the updated adversarial sample respectively, and α is the perturbation step size applied in each iteration.

[0165] Step 3.5: Ensure that the updated adversarial examples are within the allowed perturbation range

[0166] X′ t+1 =clip(X′ t+1 , X-∈, X+∈)

[0167] The clip(·) function is used to limit the adversarial sample to the range of the original sample X. Repeat the above steps until the predetermined number of iterations K is reached.

[0168] Step 3.6: Construct contrastive learning pairs and calculate contrastive learning loss.

[0169] Construct a contrastive learning pair for each adversarial sample (X′) and the corresponding original sample (X), denoted as ((X, X′)). Generate positive sample pairs ((X, X)) and negative sample pairs ((X, X″)), where (X″) is a perturbed version of the other sample. Calculate the contrastive learning loss to make similar sample pairs (positive sample pairs) closer and dissimilar sample pairs (negative sample pairs) farther apart:

[0170]

[0171] Among them, h i 、h j is a sample X i , X j The representation vector obtained by the encoder, sim(u,v)=u T v / (||u||||v||) is the cosine similarity of vectors u and v, τ is the temperature hyperparameter that controls the scaling of the similarity, ∑ k exp(sim(h i ,h k ) / τ) is the sum of all negative sample pairs in the current mini-batch.

[0172] Step 3.7: Generate joint adversarial samples and calculate the joint adversarial loss.

[0173] Use a variety of adversarial attack methods to generate different types of adversarial sample sets X adv1 , X adv2 , …, X advM , combine them with the original sample X to form a joint adversarial sample set X joint =X∪X adv1 ∪X adv2 ∪…∪X advM For each joint adversarial example x * ∈X joint , calculate its corresponding loss function:

[0174] L adv (x * )=L(x * , y; θ)

[0175] Where L is the task cross entropy loss function, y is x * The corresponding true label, θ is the model parameter; the final joint adversarial loss is the mean of all sample losses:

[0176]

[0177] Where N is the total number of joint adversarial sample sets;

[0178] By using the contrastive learning loss L contrast and the joint adversarial loss L joint Combined, the model is able to simultaneously learn adversarially robust representations and enhance defense capabilities against different types of adversarial examples.

[0179] Step 4: Combine the generated adversarial samples with the original samples to form a new training set, which is input into the pre-trained language model T5 for joint training. The performance is further improved through model fusion, and the model is comprehensively evaluated.

[0180] Step 4.1: Combine the joint adversarial sample set with the original sample X in the training set to form a new training set.

[0181] Step 4.2: Use the Tokenizer of the T5 model to jointly train the new training set, and train i independent T5 models using different training parameters or different initialization methods.

[0182] Step 4.3: Model Fusion

[0183] Step 4.3.1: Use different validation sets to independently train the above i T5 models; obtain the prediction results of each model i and the corresponding weight w i ;

[0184] Step 4.3.2: Based on the prediction results of multiple models and the weights shown on the validation set, calculate the weighted value as the final prediction:

[0185] Step 4.3.3: Select an appropriate quantization method, such as dynamic quantization, static quantization, or hybrid quantization, to quantize the trained T5 model to reduce the size and computational overhead of the model, and test the performance of the quantized model to ensure that the inference speed is improved while maintaining accuracy.

[0186] Step 5: Input a new sentence, generate prompt input through the prompt template, use the fine-tuned and adversarially trained T5 model for reasoning, and identify and label the named entities in the sentence.

[0187] Step 5.1: Input a new unlabeled sentence as the input of the model. Combine the new sentence with the predefined prompt template to generate the prompt input.

[0188] Step 5.2: Use the T5 model's Tokenizer to encode the generated prompt input into a tensor format that the model can process. Input the encoded prompt into our fine-tuned and controlled trained T5 model for reasoning, and output a sentence containing entity annotation information.

[0189] Step 5.3: Parse the model output and extract named entities. Based on the model parsing results, apply the identified and annotated named entities to the original sentence to generate the final named entity recognition results.

[0190] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention. < / sentence> < / sentence> < / sentence> < / sentence>

Claims

1. A templated named entity recognition method based on prompt learning and adversarial training, characterized in that: The steps include: Step 1: Convert the NER task to a seq2seq task, define prompt templates for different types of entities, convert the target entity into a sentence in a specific format, and combine it with the input sentence to generate a prompt input set; Step 2: Input the prompt input set into the pre-trained language model T5 and fine-tune it through prompt learning so that the model can generate sentences containing entity annotation information based on the prompts; Step 3: Use the projected gradient descent method PGD to impose small perturbations on the prompt input set to generate adversarial samples, and perform supervised contrastive learning and hybrid adversarial training; Step 4: Combine the generated adversarial samples with the original samples to form a new training set, which is input into the pre-trained language model T5 for joint training. The performance is further improved through model fusion, and the model is comprehensively evaluated. Step 5: Input a new sentence, generate prompt input through the prompt template, use the fine-tuned and adversarially trained T5 model for reasoning, and identify and annotate the named entities in the sentence.

2. According to claim 1, a templated named entity recognition method based on prompt learning and adversarial training is characterized in that: The specific method of step 1 is: Step 1.1: Convert the NER task from a traditional sequence labeling task to a sequence-to-sequence (seq2seq) task, converting the input sentence into an output sentence containing the target entity labeling information; Step 1.2: Determine the named entity type that needs to be identified, and define a corresponding prompt template for each entity type. The template format is as follows: Positive sample template: "In the sentence, <sentence>,the <entity-type> entities are=><target_entity> ”; < / sentence> Negative sample template: "In the sentence,' <sentence>',the <entity-type> entities are=>None"; < / sentence> Using the hint template, the target entity in the input sentence is embedded into a sentence with a specific format; Step 1.3: Generate positive and negative samples For each entity type in each training instance, generate a positive sample and replace the target entity in the input sentence with the positive sample template; For non-existent entity types, negative samples are generated by combining the input sentence with the negative sample template to indicate that there is no entity of that type; Step 1.4: Combine the generated special format sentence with the original input sentence to generate a prompt input; Step 1.5: Build the prompt input set and divide it into training set, validation set and test set.

3. According to claim 1, a templated named entity recognition method based on prompt learning and adversarial training is characterized in that: The specific method of step 2 is: Step 2.1: Use the TokenizerFast object of the pre-trained language model T5 to encode the training set and generate a tensor X that the model can process; Step 2.2: Forward propagate the encoding through the multi-layer Transformer of the pre-trained language model T5 and output the logits score vector for each position; Step 2.2.1: Set the mask position mask_idx, which is the index of the entity position that the model needs to predict; Step 2.2.2: The input tensor X is linearly transformed by the weight matrix W1 and the bias vector b1, and then the hidden layer output H is calculated by the activation function f: H=f(W1X+b1) Step 2.2.3: The output H of the hidden layer is linearly transformed through the weight matrix W2 and the bias vector b2, and then the final output O is obtained through the activation function g: O=g(W2X+b2) Step 2.2.4: According to the mask position, convert the logits into a probability distribution p through the softmax function ij : Among them, p ij 、z ij are the predicted probability of sample i in category j and the logit value of sample i in category j, respectively, and C is the number of categories; Step 2.2.5: Compare the model prediction results with the true label y and calculate the cross entropy loss: Among them, L is the loss function value, N is the number of samples, and y ij One-hot encoding of the true label; Step 2.3: Use the back-propagation algorithm to calculate the gradient of the loss function and use AdamW to update the T5 parameters according to the gradient.

4. According to claim 3, a templated named entity recognition method based on prompt learning and adversarial training is characterized in that: The specific process in step 2.3 is: Step 2.3.1: Calculate the gradient of the loss function: Among them, θ is the model parameter; Step 2.3.2: The back-propagation algorithm calculates the gradient layer by layer through the chain rule, propagating from the output layer to the input layer; Output layer gradient calculation: Hidden layer gradient calculation: Step 2.3.3: Use the AdamW optimizer to update the model parameters according to the gradient; Initialize the first-order moment estimate m0=0 and the second-order moment estimate v0=0, and at each time step t, calculate the gradient of the objective function L(θ) with respect to the parameter θ: Update first-order moment estimate: m t =β1m t-1 +(1-β1)g t , where β1 is the decay rate of the first-order moment estimate, m t is the first-order moment estimate at the current time t, m t-1 is the first-order moment estimate at the previous moment t-1; Update the second moment estimate: where v t is the second-order moment estimate at the current time t, v t-1 is the second-order moment estimate of the previous moment t-1; Compute bias-corrected first- and second-order moment estimates: Use the corrected moment estimate to update the parameters and add a weight decay term: Among them, θ t is the model parameter of the tth iteration, θ is the weight decay coefficient, η is the learning rate, and a is a small positive constant used to ensure numerical stability.

5. The templated named entity recognition method based on prompt learning and adversarial training according to claim 1, characterized in that: The specific method of step 3 is: Step 3.1: Starting from the original training set, the initial control sample X'0, copy each input example X as the initial adversarial sample X'0: X'0=X Step 3.2: Set the allowed perturbation range ∈, that is, the maximum amplitude of the perturbation applied to the input data; Step 3.3: Calculate the gradient of the loss function; Step 3.3.1: Perform forward propagation on the current adversarial sample X′ and calculate the model's predicted output y′: Among them, θ is the model parameter, is the forward propagation function of the model; Step 3.3.2: Calculate the loss function L(y′,y): Among them, y is the true label, y′ is the prediction result of the model, and N batch is the total number of samples in the current adversarial training batch, C is the total number of categories, and y ij is the true label of sample i in category j, p ij is the predicted probability of sample i in category j; Step 3.3.3: Calculate the gradient of the loss function with respect to the adversarial example Step 3.4: Update the adversarial example based on the calculated gradient: Among them, X' t , X' t+1 are the current adversarial sample and the updated adversarial sample respectively, and α is the perturbation step size applied in each iteration; Step 3.5: Ensure that the updated adversarial example is within the allowed perturbation range: X′ t+1 =clip(X′ t+1 ,X-∈,X+∈) The clip(·) function is used to limit the adversarial sample to the range of the original sample X. Repeat the above steps until the predetermined number of iterations K is reached. Step 3.6: Construct contrastive learning pairs and calculate contrastive learning loss: For each adversarial sample X' and the corresponding original sample X, construct a contrastive learning pair, denoted as (X,X'). Generate positive sample pairs (X,X) and negative sample pairs (X,X"), where X" is a perturbed version of the other sample; calculate the contrastive learning loss to make the distance between positive sample pairs closer and the distance between negative sample pairs farther: Where hi, hj are the representation vectors of samples Xi, Xj obtained by the encoder, sim(u,v)=uTv / (||u||||v||) is the cosine similarity of vectors u and v, τ is the temperature hyperparameter that controls the scaling of the similarity, ∑ k exp(sim(h i ,h k ) / τ) is the sum of all negative sample pairs in the current mini-batch; Step 3.7: Generate joint adversarial samples and calculate joint adversarial loss: Use a variety of adversarial attack methods to generate different types of adversarial sample sets X adv1 , X adv2 , …, X advM , combine them with the original sample X to form a joint adversarial sample set X joint =X∪X adv1 ∪X adv2 ∪…∪X advM ; For each joint adversarial example х * ∈X joint , calculate its corresponding adversarial loss function: L adv (X * )=L(x * ,y adv (θ) Among them, L adv is the adversarial loss function, y adv Yes * The corresponding true label, θ is the model parameter; the final joint adversarial loss is the mean of all sample losses: Among them, N adv is the total number of joint adversarial sample sets; By using the contrastive learning loss L contrast and the joint adversarial loss L joint Combined, the model is able to simultaneously learn adversarially robust representations and enhance defense capabilities against different types of adversarial examples.

6. A templated named entity recognition method based on prompt learning and adversarial training according to claim 1 or 5, characterized in that: The specific method of step 4 is: Step 4.1: Combine the joint adversarial sample set with the original sample X in the training set to form a new training set; Step 4.2: Use the Tokenizer of the T5 model to jointly train the new training set, and use different training parameters or different initialization methods to train i independent T5 models; Step 4.3: Model Fusion Step 4.3.1: Train i T5 models independently using different validation sets; get the prediction result prediction of each model i and the corresponding weight w i ; Step 4.3.2: Based on the prediction results of multiple models and the weights shown on the validation set, calculate the weighted value as the final prediction: Step 4.3.3: Select dynamic quantization, static quantization, or hybrid quantization, quantize the trained T5 model, and test the performance of the quantized model.

7. The templated named entity recognition method based on prompt learning and adversarial training according to claim 1, characterized in that: The specific method of step 5 is: Step 5.1: Input a new unlabeled sentence as the input of the model, combine the new sentence with the predefined prompt template to generate prompt input; Step 5.2: Use the T5 model's Tokenizer to encode the generated prompt input into a tensor format that the model can process, input the encoded prompt into the fine-tuned and controlled trained T5 model for reasoning, and output a sentence containing entity annotation information; Step 5.3: Parse the model output, extract named entities, and apply the identified and annotated named entities to the original sentence based on the model parsing results to generate the final named entity recognition results.