A small sample named entity recognition model training method and recognition method

Abstract

The application provides a small sample named entity recognition model training method, comprising the following steps: S1, obtaining a training set, a training set type description set, a support set and a support set type description set; S2, mining clue words in each sample on the training set and the support set respectively and performing clue word labeling to obtain the training set and the support set containing named entity labels and clue word labels respectively; S3, performing multi-round iterative training on a basic named entity recognition model until convergence by using the training set and the training set type description set processed in step S2; and S4, performing migration training on the basic named entity recognition model trained in step S3 until convergence by using the support set and the support set type description set processed in step S2, to obtain a small sample named entity recognition model composed of an encoder and a nearest neighbor classifier.

Description

Technical Field

[0001] This invention relates to the field of Internet information technology processing, specifically to the field of knowledge graphs and information extraction in Internet information technology processing, and more specifically to a training method for a few-shot named entity recognition model in knowledge graphs and information extraction, and a few-shot named entity recognition method based thereon. Background Technology

[0002] Named Entity Recognition (NER), a fundamental task belonging to the field of lexical analysis, aims to detect named entities from unstructured text and classify them into predefined entity types, such as person names, place names, and organization names. To address the challenges posed by data sparsity and the scarcity of labeled data, researchers have proposed few-shot named entity recognition as a research task and direction. Currently, few-shot named entity recognition tasks generally sample benchmark datasets according to entity types and construct N-way K-shot tasks in the form of few-shot classification tasks. An N-way K-shot task refers to a few-shot classification task with N categories and K samples for each category (K is kept at a very low value to simulate a few-shot scenario). To reduce the volatility caused by insufficient labeled samples, multiple sets of N-way K-shot tasks are constructed through sampling during the training and testing phases, and the average test metrics across all tasks are used as the final result.

[0003] Under current technology, facing the challenge of scarce labeled data, the general approach to solving the few-shot named entity recognition problem is to train a high-performance model on a large amount of source domain data, and design some transfer learning methods to enable the model to effectively utilize learned knowledge and quickly generalize to the target domain. Based on the transfer learning methods employed, current work on few-shot named entity recognition can be broadly categorized into three types: meta-learning-based methods, multi-task learning-based methods, and template learning-based methods.

[0004] Regardless of the transfer learning approach used, adaptability to new entity types is crucial for few-shot named entity recognition tasks. Building named entity recognition models in new domains presents challenges beyond just a lack of domain-specific labeled data; new, manually defined entity types constantly emerge due to business needs. Existing methods treat this task merely as a cross-domain semi-supervised learning problem, where the target domain is single and the set of entity types is known. While they improve performance by using label-specific decoding frameworks, they fail to generalize to new entity types and target domains. Adaptation to changes in the target domain type set requires retraining, which is extremely costly.

[0005] Furthermore, existing methods are prone to negative transfer. Specifically, this usually manifests as an impact caused by inconsistencies in entity classification standards between the source and target domains. This difference can be further subdivided into two categories: first, inconsistent negative classification standards, meaning that non-entities in the source domain may be classified as a certain entity type in the target domain, and vice versa; second, finer-grained entity type classification in the target domain, meaning that the target domain labels fine-grained entities while the source domain only labels coarse-grained entities. During multi-source domain joint training, due to the significant differences between the source and target domains, sometimes increasing the number of source domains can negatively impact the final performance on the target domain.

[0006] Furthermore, existing methods suffer from insufficient cross-domain representation capabilities. Existing methods enhance the model's ability to capture domain-independent features by incorporating additional knowledge and tasks such as part-of-speech tagging, prediction results from named entity recognition tools, knowledge graph node relationship prediction, and masked language models, thereby ultimately improving prediction accuracy in the target domain. However, while these additional knowledge and tasks can improve the cross-domain robustness of the generated text representations, they require additional annotation resources or are not closely integrated with the named entity recognition task, resulting in insufficient gains.

[0007] In summary, the existing technologies mainly suffer from insufficient adaptability to new entity types, susceptibility to negative transfer phenomena, and insufficient cross-domain representation capabilities. Summary of the Invention

[0008] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a training method for a few-shot named entity recognition model that is more adaptable to new entity types, less prone to negative transfer, and has strong cross-domain representation capabilities, as well as a few-shot named entity recognition method based thereon.

[0009] According to a first aspect of the present invention, a method for training a few-shot named entity recognition model is provided. The method includes: S1, obtaining a training set, a training set type description set, a support set, and a support set type description set, wherein the training set contains multiple named entity types, each named entity type includes multiple sentence samples, and each sample contains a corresponding named entity type label; the training set type description set contains all named entity type descriptions in the training set; the support set includes new named entity types that are different from the named entity types in the training set, each new named entity type includes sentence samples that meet the few-shot task requirements, and each sample contains a corresponding new named entity type label; the support set type description set contains all named entity type descriptions in the support set. S2. Mine clue words in each sample on the training set and support set respectively, and annotate the clue words to obtain the training set and support set containing named entity labels and clue word labels respectively; S3. Use the training set and training set type description set processed in step S2 to perform multiple rounds of iterative training on the basic named entity recognition model until convergence, wherein the basic named entity recognition model includes an encoder, an induction module, a masked language model classifier, a nearest neighbor classifier, and a clue word annotator; S4. Use the support set and support set type description set processed in step S2 to perform transfer training on the basic named entity recognition model trained in step S3 until convergence, to obtain a few-sample named entity recognition model composed of an encoder and a nearest neighbor classifier.

[0010] Preferably, in step S2, the cue words in the training set and support set are mined by a pre-trained named entity recognition model consisting of a pre-trained language model and a CRF annotator, wherein the cue words in each sample are determined by the named entity prediction probability of each sample in the training set or support set output by the converged CRF annotator.

[0011] In some embodiments of the present invention, the pre-trained language model is a BERT model.

[0012] Preferably, in step S2, clue words in each sample are mined as follows: S21, name entity words in the original sample are masked to obtain a baseline sample; S22, words in other positions in the baseline sample are masked to obtain multiple comparison samples; S23, the converged name entity recognition model is used to perform name entity recognition on the baseline sample and all comparison samples respectively, and the accuracy of name entity recognition based on the entity prediction probability output by CRF is calculated based on the baseline sample and each comparison sample. The difference between the name entity recognition accuracy of each comparison sample and the baseline sample is calculated as the influence score of the masked words in the comparison sample on the name entity recognition. The masked words in the comparison samples whose influence scores exceed a preset threshold are the clue words of the name entities in the original sample.

[0013] Preferably, in step S3, each round of iterative training includes: S31, using the encoder from the previous training to encode each sample in the training set and each type description in the training set type description set to obtain the training set encoding representation set and the training set type description encoding representation set respectively; wherein, the encoding representation of each sample or each type description is the concatenation of the word vector representation of each word therein, and the encoder encodes the word vector of each word through a dictionary; S32, using a masked language model classifier to perform masked type description word prediction for each type description based on the training set type description encoding representation set, and calculating the type description prediction loss based on the predicted type description and the actual type description in the training set type description; S33, using an induction module to perform masked type description word prediction for each type description in the training set type description set. S34. The set of all word vectors of the same named entity type is summarized to obtain the type prototype representation of the corresponding named entity type; S35. The nearest neighbor classifier is used to perform named entity recognition based on the training set encoding representation set and the type prototype representations corresponding to all named entity types in the training set to obtain the predicted named entity type corresponding to each sample, and the named entity prediction loss is calculated based on the predicted named entity type and named entity type label of all samples; S36. The cue word labeler is used to predict the cue words in each sample based on the training set encoding representation set, and the cue word prediction loss is calculated based on the predicted cue words and the actual labeled cue words of all samples; S37. The parameters of the basic named entity recognition model are updated using the type description prediction loss, named entity prediction loss and cue word prediction loss.

[0014] Preferably, an inductive module is used to obtain the type prototype representation of the named entity type by summing all word vectors of the same named entity type and then averaging them.

[0015] Preferably, in step S35, the parameters of the encoder are updated using type description prediction loss and named entity prediction loss, and the parameters of the cue word labeler are updated using cue word prediction loss.

[0016] In some embodiments of the present invention, the type description prediction loss is:

[0017]

[0018] in, The i-th type description in the type description set t, and the type word vector is words in the encoding representation When masked, the encoded representation is the normalized probability predicted as the i-th type description, where the i-th type description is represented by the encoded word. When covered, m i =1, otherwise, m i=0; P(c t |x t ) represents the word x in the type description set t. t It is predicted to be the correct category c. t The probability, where N represents the number of all words in the type description set t;

[0019] The named entity prediction loss is:

[0020] L contras =L token-token +L token-proto

[0021] Among them, L token-token The contrastive loss is between named entity word vectors of samples in the training set, and:

[0022]

[0023] Let represent the named entity word vectors of the i-th sample and the j-th sample in the training set s, respectively. express and The similarity between them, N′ represents the number of samples in the training set, K represents the number of samples of each class in each training set, and τ1 represents the temperature coefficient. L token-proto Let represent the contrastive learning loss between the encoded representations of samples in the training set and the type prototype representations, and:

[0024]

[0025] Let c represent the named entity word vector of the i-th sample in the training set s. j This represents the prototype representation of the j-th type. express With c j The similarity is given by N′, where N′ represents the number of samples in the training set, K represents the number of samples in each category in the training set, and τ2 represents the temperature coefficient.

[0026] The loss for predicting the cue words is:

[0027]

[0028] Where, |x s | indicates the number of cue words in the training set. The table shows the cue words in the i-th sample and the support words. This represents the named entity word vector of the i-th sample in the training set s. express and The similarity.

[0029] Preferably, the encoder is a pre-trained language model. In each iteration of training, the type prototype representation obtained by the induction module is fed back to the encoder to update the dictionary in the encoder used to encode type descriptions.

[0030] Preferably, the basic named entity recognition model further includes a template network, which is used to generate independent template representation vectors for the encoder to be inserted into each of its hidden layers. During iterative training, the parameters of the encoder are fixed and the template vector parameters are updated.

[0031] According to a second aspect of the present invention, a few-shot named entity recognition method is provided, the method comprising: T1, acquiring a target sample; T2, performing named entity recognition on the target sample using a few-shot named entity recognition model trained based on the method described in the first aspect of the present invention.

[0032] Compared with existing technologies, the advantages of this invention are as follows: This invention not only introduces a masked language model classifier during training to integrate type-related semantic information into the training process, but also introduces a cue word annotation task through multi-task learning to learn and capture the semantic relationships between entities and cue words, thereby increasing entity recognition accuracy. Furthermore, this invention introduces a template network into the encoder, which reduces storage costs by fine-tuning the pre-trained language model parameters while maintaining their fixed parameters. By introducing additional tasks strongly related to named entity recognition, the recognition performance and accuracy of the named entity recognition model are significantly improved. Attached Figure Description

[0033] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0034] Figure 1 This is a schematic diagram of the training method for a few-sample named entity recognition model according to an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram illustrating an example of clue word mining according to an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram illustrating the basic named entity recognition model framework and training principle according to an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of a template network structure according to an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram illustrating the principle of the encoder self-attention mechanism for the inserted template representation vector according to an embodiment of the present invention;

[0039] Figure 6This is a schematic diagram illustrating the training principle of the basic named entity recognition model according to an embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0041] As described in the background section, small-sample named entity recognition models obtained using existing transfer learning methods suffer from insufficient adaptability to new entity types, susceptibility to negative transfer, and inadequate cross-domain representation. While some have proposed adding extra knowledge and tasks to enhance model accuracy, these additional knowledge and tasks require additional annotation resources or are not closely related to the named entity recognition task, resulting in limited enhancement. The inventors, through research, discovered that every unstructured text contains cue words closely related to named entities. For example, in "Tom is a professor at the UCAS," "Tom" is a person's named entity, "professor" indicates "Tom's" title, and "UCAS" indicates his workplace. These two words are closely related to the named entity "Tom" and can serve as cue words, effectively guiding the model's recognition of "Tom." By automatically mining cue words related to named entities, cue word annotations can be obtained, thus acquiring additional annotation resources. These resources can then be used to introduce a cue word annotation training task to enhance the model's ability to capture domain-independent features. Furthermore, the inventors introduced type description text corresponding to named entities to introduce an additional type description training task, which, together with the cue word annotation training task, enhances the model's prediction accuracy. It should be noted that in this invention, the dataset with a large number of labeled samples is called the training set, the dataset with only a few labeled samples is called the support set, and the unlabeled dataset used for testing, which has the same named entity types as those in the support set, is called the query set. Since cue words and type descriptions are closely related to named entity types, when used as additional tasks to assist training, compared to additional tasks in existing technologies that are not closely related to named entities, they can significantly improve the model's performance, enhance adaptability to new entity types, reduce negative transfer phenomena, and enhance cross-domain representation capabilities. Therefore, this invention proposes a small-sample named entity recognition model training method based on cue word annotation and type description prediction tasks. In summary, as follows... Figure 1As shown, the training method for the few-shot named entity recognition model proposed in this invention includes: S1, obtaining a training set, a training set type description set, a support set, and a support set type description set, wherein the training set contains multiple named entity types, each named entity type includes multiple sentence samples, and each sample contains a corresponding named entity type label; the training set type description set contains all named entity type descriptions in the training set; the support set includes new named entity types that are different from the named entity types in the training set, each new named entity type includes sentence samples that meet the few-shot task requirements, and each sample contains a corresponding new named entity type label; the support set includes... S2. The support set type description set contains all named entity type descriptions in the support set; S3. Catch words are mined from each sample in the training set and support set respectively, and the catch words are labeled to obtain the training set and support set containing named entity labels and catch word labels respectively; S4. The basic named entity recognition model is trained through multiple rounds of iteration using the training set and training set type description set processed in step S2 until convergence; S5. The basic named entity recognition model trained in step S3 is transferred to the support set and support set type description set processed in step S2 until convergence to obtain a few-sample named entity recognition model composed of an encoder and a nearest neighbor classifier.

[0042] It should be noted that type descriptions are text used to describe the specific type of named entities. Type descriptions can come from various sources, including type names, officially published annotation rules, knowledge base-aligned type descriptions, manually designed descriptions, and automatically generated descriptions from pre-trained models. For ease of understanding, this invention uses type names as type descriptions as examples in the embodiments. For instance, in "Tom is a professor at the UCAS," which is a person's name named entity, the type description for a person's name using type names would be "The name is a word or set of words by which a person is known, or referred to."

[0043] To better understand this invention, it will be described below from several aspects: clue word mining, training on the training set (source domain), and transfer learning on the support set (target domain). It should be noted that, due to the needs of variable representation, the enumerated variables may all use i and j in different formula descriptions, but this does not mean that i and j have the same meaning. They are used only for the convenience of describing the enumerated variables; the specific meaning is subject to the specific description.

[0044] I. Discovery of Clue Words

[0045] In this invention, the purpose of cue word mining is to accurately obtain words closely related to named entities, which can then be used as annotation resources to assist in training the model. According to one embodiment of the invention, a named entity recognition model composed of a pre-trained language model (e.g., BERT model) and a CRF annotator is trained to converge using a training set consisting of a large number of samples with named entity labels, and the network parameters are updated using classification loss. Then, the named entity model is used to perform named entity recognition on the training set, and the cue words in each sample are determined using the named entity prediction probabilities corresponding to the samples in the training set output by the converged CRF. Since training the named entity model using a training set is a technique known to those skilled in the art, this invention will not elaborate on this training process, but will focus on explaining the cue word mining process.

[0046] When mining cue words for each sample, firstly, the named entity words in the original sample are masked to obtain a baseline sample. Then, words in other positions in the baseline sample are masked to obtain multiple comparison samples. Finally, a trained named entity recognition model is used to perform named entity recognition on the baseline and comparison samples. Based on the named entity prediction probability output by the CRF, the accuracy of named entity recognition based on the baseline sample and each comparison sample is calculated. The difference between the named entity recognition accuracy of each comparison sample and the baseline sample is calculated as the influence score of the masked words in the comparison samples on named entity recognition. The masked words in the comparison samples whose influence scores exceed a preset threshold are the cue words of the named entities in the original sample. According to an example of the present invention, such as... Figure 2 As shown, suppose we have a sample that says "Tom is a professor at the UCAS". This is the original sample, and the named entity in it is "Tom". In order to extract clue words related to "Tom" from this sample, we cover "Tom" to obtain the baseline sample. That is, “[Mask] is a professor at the UCAS”, and then by masking the words in other positions and the entity reference, a set of contrasting samples is generated. That is, {"[Mask][Mask]a professor at the UCAS", "[Mask]is[Mask]professor at the UCAS", "[Mask]is a[Mask]at the UCAS", "[Mask]is a professor[Mask]the UCAS", "[Mask]is a professor at[Mask]UCAS", "[Mask]is a professor at the[Mask]"}. These two types of samples are fed into the named entity recognition model, and then the forward-backward algorithm of CRF is used to calculate the correct prediction probability for each entity reference. and Next, the difference {δ} between the correct prediction probabilities of the comparison sample and the baseline sample will be calculated. i1 ,δ i2 ,...,δ im} The influence score of the masked words on named entity prediction is used as the criterion. Finally, words with influence scores exceeding a preset threshold τ are selected, where τ is a manually set hyperparameter, i.e., δ is chosen. im The masked words in the comparison samples greater than or equal to τ are used as cue words. As explained in the background, in this example sentence, "professor" and "the UCAS" are cue words for "Tom", which will not be repeated here.

[0047] Through the above process, we can extract and label the cue words in each sample of the training set and support set, thereby obtaining a training set and support set that contain both named entity labels and cue word labels.

[0048] II. Training in the source domain

[0049] As mentioned earlier, since cue words and type descriptions are closely related to named entity types, using them as additional tasks to assist training can significantly improve the model's performance compared to additional tasks that are not closely related to named entities in existing technologies. This can enhance the model's adaptability to new entity types, reduce negative transfer phenomena, and strengthen cross-domain representation capabilities.

[0050] This invention trains a basic named entity recognition model using a training set that includes both named entity tags and cue word tags. During training, cue word mining and type description prediction tasks are introduced. According to one embodiment of the invention, as... Figure 3As shown, the basic named entity recognition model includes an encoder, an induction module, a masked language model classifier, a nearest neighbor classifier, and a cue word labeler. The encoder uses a pre-trained language model to encode the input text to obtain its encoded representation. In this embodiment, the encoder is described using the BERT model as an example. The implementation of other pre-trained language models as encoders is similar to BERT, and will not be elaborated further in this invention.

[0051] To better understand, let's first briefly introduce the basic structure of the BERT pre-trained language model. The BERT model mainly consists of embedding layers and multiple sequentially connected hidden units. The embedding layers include positional embedding layers, word type embedding layers, and word embedding layers. The positional embedding layer obtains the positional feature embeddings of words in the input text; the word type embedding layer obtains the word type embeddings of words in the input text; and the word embedding layer obtains the word vector embeddings of words in the input text based on dictionary knowledge. The hidden units encode the embeddings of the input text and finally output the encoded representation of the input text formed by concatenating word vectors. Each hidden unit is a Transformer unit. It should be noted that, to reduce the amount of parameter updates during training, such as... Figure 3 As shown, this invention introduces a template network (Prompting-tuning) into the encoder. The template network generates a continuous embedding representation, called a template representation vector sequence, for each hidden unit of the encoder. During training, the encoder parameters are fixed, and only the parameters of the template representation vector sequence are updated. This significantly reduces the memory consumption of the encoder parameters without affecting the encoder's training performance. According to one embodiment of the invention, as... Figure 4 As shown, the template network consists of an embedding layer and two fully connected layers connected sequentially. The embedding layer has a dictionary size of L and is used to generate embedding vectors at the corresponding positions for a template of length L. Then, after passing through two fully connected layers, the final template representation vector sequence is output.

[0052] p = {p i} i＝1,2,…,M

[0053] in, These are template representation vector pairs inserted into the i-th hidden layer of BERT, and they participate in the computation of the self-attention mechanism of the Transformer unit. It is the part that participates in the key calculation. This is the part that participates in the value calculation. When encoding text w with the participation of the template representation vector sequence p inserted into BERT:

[0054] h = BERT(w, p)

[0055] In the j-th hidden layer, the original self-attention mechanism is calculated as follows:

[0056]

[0057]

[0058]

[0059]

[0060] Among them, X (j) It is the input of the j-th hidden layer. These are the generation of the key vector K(X) (j) ), value vector V(X) (j) ) and query vector Q(X) (j) The linear layer weights, Att(X) (j) ) is the final calculated attention score, d k This represents the number of BERT self-attention heads. In the insertion template representation p j Afterwards, as Figure 5 As shown, the computational process of the self-attention mechanism is as follows:

[0061]

[0062]

[0063] Among them, X (j)′ It is the input of the j-th hidden layer, and respectively with K(X) (j) ) and V(X (j) The key vector and value vector are concatenated together to form a new key vector and value vector. Through calculations between the key vector, value vector, and query vector, a new attention score Att(X) is obtained. (j)′ ).

[0064] Since the principles of template networks and self-attention mechanisms are known to those skilled in the art, only the above brief introduction is given in the embodiments of the present invention, and no further details are provided.

[0065] The following is based on Figure 3 Taking the basic Named Entity Network in the example, this article will explain in detail the process of training it using a training set. For example, ... Figure 6As shown, during training, the encoder encodes each sample in the training set and each type description in the training set type description set to obtain the training set encoding representation set and the training set type description encoding representation set, respectively. The masked language model classifier predicts the type description words based on the training set type description encoding representation set for each type description. The induction module inductively summarizes all word vectors of the same named entity type based on the training set encoding representation set to obtain the type prototype representation of the corresponding named entity type. The nearest neighbor classifier performs named entity recognition based on the training set encoding representation set and the type prototype representations corresponding to all named entity types in the training set to obtain the predicted named entity type for each sample. The cue word labeler predicts the cue words in each sample based on the training set encoding representation set.

[0066] In this invention, training in the source domain refers to iteratively training the basic named entity recognition model with the training set. The specific process of this training is illustrated below using one iteration as an example.

[0067] According to one embodiment of the present invention, each iteration of training includes steps S31-S36, each of which is described in detail below.

[0068] In step S31, the encoder after the previous training is used to encode each sample in the training set and each type description in the training set type description set to obtain the training set encoding representation set and the training set type description encoding representation set respectively; wherein, the encoding representation of each sample or each type description is the concatenation of the word vector representation of each word therein, and the encoder encodes the word vector of each word through a dictionary.

[0069] In step S32, a masked language model classifier is used to predict the type description words for each type description in the training set type description encoding representation set based on the mask, and the type description prediction loss is calculated based on the predicted type description and the actual type description in the training set type description. According to an embodiment of the present invention, the type description prediction loss is:

[0070]

[0071] in, This represents the i-th type description in the type description set t, and its type word vector is... words in the encoding representation When masked, the encoded representation is the normalized probability predicted as the i-th type description, where the i-th type description is represented by the encoded word. When covered, m i =1, otherwise, m i =0; P(c t |xt ) represents the word x in the type description set t. t It is predicted to be the correct category c. t The probability is given by N, where N represents the number of all words in the type description set t.

[0072] In step S33, an induction module is used to inductively represent all word vectors of the same named entity type based on the training set encoding representation set to obtain the type prototype representation of the corresponding named entity type. According to one embodiment of the present invention, the induction module obtains the type prototype representation of the named entity type by summing all word vectors of the same named entity type and then averaging them, and feeds the type prototype representation obtained by the induction module back to the encoder to update the dictionary used for encoding type descriptions in the encoder. According to one embodiment of the present invention, the induction module employs a capsule network with a dynamic routing mechanism, which uses the training set encoding representation set... As input, Let the representation vector of the j-th word representing the i-th entity type in the training set be used. Then, for each word representation sample, a capsule representation vector is predicted. for:

[0073]

[0074]

[0075] By iteratively running a dynamic routing mechanism, the type prototype representation is ensured to automatically encapsulate capsule representations of such samples. In each iteration, this process dynamically modifies the connection weights d. ij And by using Softmax, we ensure that the prototype representation of candidate types in each class is guaranteed. and the capsule representation vector of this type of training set samples The sum of the connection weights between them is equal to 1:

[0076] d i =softmax(b i )

[0077] Where, d i It is the prototype representation of all samples of type i in the training set and the candidate type. The coupling coefficients between them are initialized to zero vectors in the first iteration. Given a capsule representation vector of type i... Type candidate vector prototype representation It is the prediction vector of all samples in type i. The capsule represents a weighted sum of vectors, then:

[0078]

[0079]

[0080] Among them, c i This is the type prototype representation of type i, and is the output of the inductive module after multiple iterations. The last step of each iteration is to adjust the coupling coefficient b of the training set samples belonging to type i using the "routing by agreement" method. ij . d ij The update method is as follows:

[0081]

[0082] Since dynamic routing mechanisms are known to those skilled in the art, they will not be described in detail here.

[0083] During the continuous iterative training of the basic named entity recognition model, the model is constantly updated, and the encoding representation set obtained from each training set becomes more and more accurate. As a result, the type prototype representation obtained by the induction module also becomes more and more accurate. Using this representation to update the dictionary of encoded type descriptions in the encoder can yield a more accurate type description encoding representation.

[0084] In step S34, a nearest neighbor classifier is used to perform named entity recognition based on the training set's encoded representation set and the type prototype representations corresponding to all named entity types in the training set to obtain the predicted named entity type for each sample. The named entity prediction loss is then calculated based on the predicted named entity types and named entity type labels for all samples. According to an embodiment of the present invention, the named entity prediction loss is:

[0085] L contras =L token-token +L token-proto

[0086] Among them, L token-token The contrastive loss is between named entity word vectors of samples in the training set, and:

[0087]

[0088] Let represent the named entity word vectors of the i-th sample and the j-th sample in the training set s, respectively. express and The similarity between them, N′ represents the number of samples in the training set, K represents the number of samples in each category of the training set, and τ1 represents the temperature coefficient.

[0089] L token-proto Let represent the contrastive learning loss between the encoded representations of samples in the training set and the type prototype representations, and:

[0090]

[0091] Let c represent the named entity word vector of the i-th sample in the training set s. j This represents the prototype representation of the j-th type. express With c j The similarity is given by N′, where N′ represents the number of samples in the training set, K represents the number of samples in each category in the training set, and τ2 represents the temperature coefficient.

[0092] In step S35, a cue word labeler is used to predict the cue words in each sample based on the encoded representation set of the training set, and the cue word prediction loss is calculated based on the predicted cue words and the actually labeled cue words for all samples. According to an embodiment of the present invention, the cue word prediction loss is:

[0093]

[0094] Where, |x s | indicates the number of cue words in the training set. The table contains the cue words in the i-th sample of the training set. This represents the named entity word vector of the i-th sample in the training set s. express and The similarity.

[0095] In step S36, the parameters of the basic named entity recognition model are updated using type description prediction loss, named entity prediction loss, and cue word prediction loss. According to one embodiment of the present invention, the parameters of the encoder are updated using type description prediction loss and named entity prediction loss, and the parameters of the cue word annotator are updated using cue word prediction loss.

[0096] III. Transfer Learning

[0097] The purpose of transfer learning is to transfer a model trained on a source domain to a target domain. Specifically, it refers to using a support set to perform transfer training on a model trained on the training set, ultimately obtaining a few-shot named entity recognition model composed of an encoder and a nearest neighbor classifier. Since transfer learning is a common technique known to those skilled in the art, it will not be described in detail here.

[0098] As demonstrated by the above embodiments, this invention not only introduces a masked language model classifier during training to integrate type-related semantic information into the training process, but also introduces a cue word annotation task through multi-task learning to learn and capture the semantic relationships between entities and cue words, thereby increasing entity recognition accuracy. Furthermore, this invention introduces a template network into the encoder; under the condition that the pre-trained language model parameters are fixed, fine-tuning through the template network reduces storage costs. By introducing additional tasks strongly related to named entity recognition, the recognition performance and accuracy of the named entity recognition model are significantly improved.

[0099] To further verify the effectiveness of this invention, the inventors conducted experiments on publicly available datasets such as SNIPS and Few-NERD, and validated the model's recognition performance and accuracy using query sets. Experimental results show that the method of this patent improves the average F1 score of the best-performing existing methods by 6.3% and 9.7% on the SNIPS dataset in single-sample and 5-sample settings, respectively; and improves the average F1 score by 4.1% and 20.1% on the corresponding settings of the Few-NERD dataset, respectively. It should be noted that although the steps are described in a specific order above, this does not mean that the steps must be executed in the specified order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required functionality is achieved.

[0100] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0101] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0102] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a few-shot named entity recognition model, characterized in that, The method includes: S1. Obtain the training set, the training set type description set, the support set, and the support set type description set. The training set contains multiple named entity types, each named entity type includes multiple sentence samples, and each sample contains a corresponding named entity type label. The training set type description set contains all named entity type descriptions in the training set. The support set includes new named entity types that are different from the training set named entity types. Each new named entity type includes sentence samples that meet the minimum number of sentences required for the small sample task, and each sample contains a corresponding new named entity type label. The support set type description set contains all named entity type descriptions in the support set. S2. Mine clue words in each sample on the training set and support set respectively, and annotate the clue words to obtain the training set and support set containing named entity labels and clue word labels respectively; where, the clue words are words related to named entities; S3. The basic named entity recognition model is trained iteratively multiple times until convergence using the training set and training set type description set processed in step S2. The basic named entity recognition model includes an encoder, an induction module, a masked language model classifier, a nearest neighbor classifier, and a cue word labeler. The encoder encodes each sample in the training set and each type description in the training set type description set to obtain the training set encoded representation set and the training set type description encoded representation set, respectively. The induction module inductively summarizes all word vectors of the same named entity type based on the training set encoded representation set to obtain the corresponding type prototype representation of the named entity type. The masked language model classifier predicts the type description words based on the masked type description encoded representation set for each type description. The nearest neighbor classifier identifies the predicted named entity type for each sample based on the training set encoded representation set and the type prototype representations corresponding to all named entity types in the training set. The cue word labeler predicts the cue words in each sample based on the training set encoded representation set. S4. Using the support set and support set type description set processed in step S2, perform transfer training on the basic named entity recognition model trained in step S3 until convergence, to obtain a few-sample named entity recognition model composed of an encoder and a nearest neighbor classifier.

2. The method according to claim 1, characterized in that, In step S2, the named entity recognition model, which consists of a pre-trained language model and a CRF annotator, mines clue words in the training set and the support set. The clue words in each sample are determined by the named entity prediction probability of each sample in the training set or the support set output by the converged CRF annotator.

3. The method according to claim 2, characterized in that, The pre-trained language model is the BERT model.

4. The method according to claim 2, characterized in that, In step S2, clue words in each sample are mined in the following way: S21. Obtain the baseline sample by covering the named entity words in the original sample; S22. Cover up words in other positions in the baseline sample to obtain multiple comparison samples; S23. Using the converged named entity recognition model, named entity recognition is performed on the benchmark sample and all comparison samples respectively. The accuracy of named entity recognition based on the entity prediction probability output by CRF is calculated based on the benchmark sample and each comparison sample. The difference between the named entity recognition accuracy of each comparison sample and the benchmark sample is calculated as the influence score of the masked words in the comparison sample on named entity recognition. The masked words in the comparison samples whose influence scores exceed the preset threshold are the clue words of named entities in the original samples.

5. The method according to claim 1, characterized in that, In step S3, each round of iterative training includes: S31. Using the encoder from the previous training, each sample in the training set and each type description in the training set type description set are encoded to obtain the training set encoding representation set and the training set type description encoding representation set respectively; wherein, the encoding representation of each sample or each type description is the concatenation of the word vector representation of each word therein, and the encoder encodes the word vector of each word through a dictionary; S32. A masked language model classifier is used to predict the type description words for each type description in the training set type description encoding representation set based on the masked type description, and the type description prediction loss is calculated based on the predicted type description and the actual type description in the training set type description. S33. The induction module is used to inductively represent all word vectors of the same named entity type based on the training set encoding representation set to obtain the type prototype representation of the corresponding named entity type. S34. The nearest neighbor classifier is used to identify named entities based on the encoding representation set of the training set and the type prototype representation corresponding to all named entity types in the training set to obtain the predicted named entity type corresponding to each sample, and the named entity prediction loss is calculated based on the predicted named entity type and the named entity type label corresponding to all samples. S35. The cue word labeler is used to predict the cue words in each sample based on the encoding representation set of the training set, and the cue word prediction loss is calculated based on the predicted cue words of all samples and the actual labeled cue words. S36. Update the parameters of the basic named entity recognition model using type description prediction loss, named entity prediction loss, and clue word prediction loss.

6. The method according to claim 5, characterized in that, In step S33, the inductive module is used to obtain the type prototype representation of the named entity type by adding all word vectors of the same named entity type and then averaging them.

7. The method according to claim 5, characterized in that, In step S35, the parameters of the encoder are updated using type description prediction loss and named entity prediction loss, and the parameters of the cue word labeler are updated using cue word prediction loss.

8. The method according to claim 5, characterized in that, The type description predicts the loss as follows: in, The type description set t contains the first type. There are type descriptions and type word vectors. words in the encoding representation When masked, the encoded representation is predicted as the first... The normalized probability described by the nth type, the i-th The terms in the type description encoding representation When covered, ,on the contrary, ; This represents the terms in the type description set t. It is predicted to be the correct category. The probability, This represents the number of all words in the type description set t; The named entity prediction loss is: in, The contrastive loss is between named entity word vectors of samples in the training set, and: , They represent the training set, respectively. The Middle The sample and the first Named entity word vectors in each sample, express and Similarity between them Indicates the first Predicted named entity word vectors for each sample. express and Similarity between them This represents the number of samples in the training set, where K represents the number of samples in each class in the training set. First temperature coefficient; Let represent the contrastive learning loss between the encoded representations of samples in the training set and the type prototype representations, and: Indicates training set The Middle Named entity word vectors for each sample, Indicates the first Each type prototype is represented. express and similarity, Indicates the first Predictive representation of each type of prototype, express and similarity, This represents the number of samples in the training set, where K represents the number of samples in each class in the training set. Temperature coefficient; The loss for predicting the cue words is: in, This indicates the number of cue words in the training set. Indicates support for the centralized first Clue words in a sample Indicates training set The Middle Named entity word vectors for each sample, express and The similarity.

9. The method according to claim 5, characterized in that, The encoder is a pre-trained language model. In each iteration of training, the type prototype representation obtained by the induction module is fed back to the encoder to update the dictionary used to encode type descriptions in the encoder.

10. The method according to claim 8, characterized in that, The basic named entity recognition model also includes a template network, which is used to generate independent template representation vectors for the encoder to be inserted into each of its hidden layers. During iterative training, the parameters of the encoder are fixed and the template vector parameters are updated.

11. A method for few-sample named entity recognition, characterized in that, The method includes: T1. Obtain the target sample; T2. The target sample is named entity recognized using a few-sample named entity recognition model trained based on any one of the methods described in claims 1-10.

12. A computer-readable storage medium, characterized in that, It contains a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 10.

13. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the steps of the method as described in any one of claims 1 to 10.

Images