A few-shot learning named entity recognition method that integrates entity label encoding

Abstract

This invention discloses a few-shot learning named entity recognition method that integrates entity label encoding, comprising: step (1) obtaining character features and their sequences; step (2) obtaining word pair feature matrix, distance feature matrix, and region feature matrix, and concatenating them to obtain word pair relation feature matrix; step (3) obtaining the feature representation of each entity label; step (4) calculating the dot product similarity between each word pair relation feature and entity label feature to obtain the transition probability matrix; step (5) selecting the optimal model as the final pre-trained model; and step (6) loading the pre-trained model for few-shot learning, inputting sentences into the model, and outputting the word pair relation matrix. This method encodes word pairs and entity labels separately, and then matches the representations of word pairs and entity labels to obtain the entity relationships between words. It can effectively identify continuous entities, overlapping entities, and discontinuous entities under low resource conditions.

Description

Technical Field

[0001] This invention relates to the field of information extraction, specifically a few-sample named entity recognition method that integrates tag encoding, which can effectively extract named entities from text. Background Technology

[0002] Named entity recognition (NER) is a classic text information extraction task, aiming to extract meaningful entity information from unstructured text. The types of named entities to be identified vary depending on the context. When we have a large number of labeled samples in one domain but only a small number in the target domain, one way to improve model performance is to use transfer learning. This involves pre-training on a dataset with a large number of samples in the source domain and then fine-tuning it on a small number of samples in the target domain. This allows us to quickly obtain a model that performs well in the target domain; this is known as few-shot learning.

[0003] In the named entity recognition (NER) problem, the types of entities to be predicted differ across different scenarios, making direct transfer learning impossible. To address this task under few-shot learning, several methods have been proposed. Similarity-based methods categorize all sentences in the known domain according to entity type, using the average embedding of all sentences in each class to represent that class. For each sentence in a new domain during few-shot learning, the similarity between its embedding representation and the average embedding representation of each class is calculated, thus leveraging prior knowledge for transfer learning. However, this method does not fully utilize the semantic information of entity labels. Later, entity label encoding methods were proposed, using two BERT pre-trained models to encode the input sentence and entity labels respectively, and predicting a label for each word by maximizing similarity. This method can distinguish semantically similar labels, providing a simple and efficient way to perform few-shot learning. However, this method is based on sequence labeling and therefore cannot handle overlapping and discontinuous entities.

[0004] For overlapping and discontinuous entities, W has been proposed. 2 Named Entity Recognition (NER) is a unified modeling method for word-pair relationships, which can be used to identify continuous, nested, and non-continuous entities. This method defines two types of word-pair relationships: adjacency relationships, indicating that two words can be adjacent, and entity relationships, indicating that two words are the head and tail of an entity, respectively. The method identifies named entities by predicting these relationships between pairs of words. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by incorporating a few-shot learning method that integrates entity label encoding into the W2NER modeling method. This method encodes word pairs and entity labels separately, and then matches the representations of word pairs with the representations of entity labels to obtain the entity relationships between words. It can effectively identify continuous, overlapping, and discontinuous entities with low resources. Furthermore, this method is robust to changes in entity label names, can distinguish semantically similar labels, and enables few-shot transfer learning.

[0006] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0007] Step (1) Use BERT to perform character embedding transformation on each character in the input sentence to obtain character-level feature representations.

[0008] Step (2) uses CLN (Condition Layer Normalization) to represent the character feature sequence from step (1) as a word pair feature matrix V, and simultaneously calculates the distance between each character to represent it as a distance feature matrix E. d And a feature matrix E of upper and lower triangular regions. t The features of the three feature matrices are concatenated to obtain the word pair relation feature matrix C.

[0009] Step (3) convert each entity label into natural language, such as converting "ORG" into "Organization" and "PER" into "Person", and then use Bert to encode these labels to obtain the feature representation L of each entity label.

[0010] Step (4) For each word pair relation feature representation, calculate the dot product similarity with the feature representation of each entity label, and obtain the transition probability matrix of each word pair and each entity label through Softmax operation.

[0011] Step (5) Train the model with a large number of labeled samples and select the best model as the final pre-trained model.

[0012] Step (6) Load the pre-trained model, perform few-sample learning with a small number of labeled samples from other domains, input the sentence into the trained model, output the word pair relation matrix Y, decode the word pair relation matrix, and obtain the final entity words and their types.

[0013] The beneficial effects of this invention are as follows:

[0014] This invention proposes a few-shot learning named entity recognition method that integrates entity label encoding. It encodes entity labels to distinguish semantically similar labels while introducing the W2NER modeling method, enabling effective identification of continuous, overlapping, and discontinuous entities with low resources. Results show that this method has better robustness and adaptability. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall structure of the present invention;

[0016] Figure 2 This is a schematic diagram of word pair relationships in this invention; Specific implementation methods

[0017] The present invention will now be further described with reference to the accompanying drawings.

[0018] Reference Figure 1 This is a schematic diagram of the overall structure of the present invention. A few-sample learning named entity recognition method that integrates entity label encoding includes the following steps:

[0019] Step (1) Use BERT to perform character embedding transformation on each character in the input sentence to obtain character-level feature representations.

[0020] Step (2) uses CLN (Condition Layer Normalization) to represent the character feature sequence from step (1) as a word pair feature matrix V, and simultaneously calculates the distance between each character to represent it as a distance feature matrix E. d And a feature matrix E of upper and lower triangular regions. t The features of the three feature matrices are concatenated to obtain the word pair relation feature matrix C.

[0021] Step (3) convert each entity label into natural language, such as converting "ORG" into "Organization" and "PER" into "Person", and then use Bert to encode these labels to obtain the feature representation L of each entity label.

[0022] Step (4) For each word pair relation feature representation, calculate the dot product similarity with the feature representation of each entity label to obtain the transition probability matrix of each word pair and each entity label.

[0023] Step (5) Train the model with a large number of labeled samples and select the best model as the final pre-trained model.

[0024] Step (6) Load the pre-trained model, perform few-sample learning with a small number of labeled samples from other domains, input the sentence into the trained model, output the word pair relation matrix Y, decode the word pair relation matrix, and obtain the final entity words and their types.

[0025] Furthermore, the specific implementation process of step (1) is as follows:

[0026] BERT is used to perform character embedding transformation on each character in the input sentence to obtain a character-level embedding representation; for an input sentence of length N, X = {x1, x2, ..., x...} N}∈R N Use Bert to process each character x i Encode to obtain character feature sequence d h The dimension of character features.

[0027] Furthermore, the specific implementation process of step (2) is as follows:

[0028] 2-1 Using CLN (Condition Layer Normalization) to represent the character feature sequence of step (1) as a word pair feature matrix Where V ij The calculation formula is:

[0029] The character feature sequence is represented as a word pair feature matrix using CLN. Where V ij The calculation formula is:

[0030]

[0031] V ij for h i Regarding h j The feature representation, where γ ij =W α h i +b α , λ ij =W β h i +b β W is obtained after training through fully connected layers. α and W β It is a learnable weight matrix, b α and b β These are learnable bias vectors, where μ and σ are h. i The formulas for the mean and standard deviation are shown below:

[0032]

[0033] Where d h h represents the dimension of the character features. jk Let be the k-th element of the j-th character feature in the character feature sequence H.

[0034] 2-2 Simultaneously calculate the distance between each character and represent it as a distance feature matrix. For the input sentence X, two words (x i ,x j The distance between the two words is represented as the absolute distance |ij|, which is then passed through an embedding layer, which is a regular linear neural network, to obtain a distributed representation of the distance.

[0035] The regional feature matrix described in 2-3 By manually generating an N×N matrix, all values ​​in the upper triangular region of this matrix are set to 1, while those in the lower triangular region are set to 2. After passing through an embedding layer, different feature representations for the upper and lower triangular regions are obtained.

[0036] 2-4 Concatenate the features of the three feature matrices to obtain the word pair relation feature matrix. in

[0037] Furthermore, the specific implementation process of step (3) is as follows:

[0038] Each entity label undergoes natural language transformation, such as converting "ORG" to "Organization" and "PER" to "Person". Since W2NER defines two word relationships: adjacency (meaning two words can be adjacent) and entity relationship (meaning two words are the head and tail of an entity), the number of labels is the number of entity types + 2 ("adjacency" labels and "no relationship" labels). These labels are then encoded using BERT to obtain the feature representation of each entity label.

[0039] Furthermore, the specific implementation process of step (4) is as follows:

[0040] For each word pair, the relation feature representation is c. i ,i∈{1,2,…,N×N}, and the feature representation l of each entity label j ,j∈{1,2,…,N L Calculate the dot product similarity, and then perform a softmax operation to obtain the transition probability matrix between each word pair and each entity label. The calculation formula is as follows:

[0041] y ij =softmax(c i ·l j )

[0042] Furthermore, the specific implementation process of step (5) is as follows:

[0043] The English corpus from Ontonotes v5.0 was selected as the pre-training dataset. This dataset contains over 1.74 million sentences, with entities labeled as 18 types, including PERSON, ORGANIZATION, and LOCATION. This corpus was input into the model mentioned above for training. The BERT pre-training model was set to "bert-base-uncased", with 5 epochs, a learning rate of 0.00001, and a batch size of 4. The Adam optimizer was used to progressively adjust the learning rate, and the model was trained using standard cross-entropy loss, as shown in the following formula:

[0044]

[0045] in, Indicates whether it is a correct label, y ij This represents the predicted transition probability.

[0046] After multiple rounds of training, the optimal model is selected as the pre-trained model.

[0047] Furthermore, the specific implementation process of step (6) is as follows:

[0048] 6-1 Load the optimal model described in step (5). In this embodiment, CoNLL 2003 is selected as the dataset for few-shot learning. This dataset includes 1393 English news articles labeled with four entity types: PER, LOC, ORG, and MISC. Sampling is performed in a k-shot manner. For n-shot, n texts are randomly sampled from each entity type as input for further training. The BERT pre-trained model is selected as "bert-base-uncased", with epochs set to 100, a learning rate of 0.00001, and a batch size of 4. The Adam optimizer is used to progressively adjust the learning rate, and the model is trained using standard cross-entropy loss.

[0049] 6-2 After small-sample training, the sentence is input into the trained model, and the output word pair relation matrix Y∈R is generated. N ×N As attached Figure 2As shown, the word pair relation matrix is ​​decoded to obtain the final entity words and their types. The goal of decoding is to find all entity word index sequences and their corresponding categories. We construct a graph where nodes are words and edges are NNW relations. Then, we use a depth-first search algorithm to find all paths from the head word to the tail word, i.e., the word index sequence of the corresponding entity. Then, we take the head index i and tail index j of the entity, and find the THW-* relation at position [j,i] in the word pair relation matrix, which is the category of the entity. The table below shows some prediction results:

[0050]

Claims

1. A few-shot learning named entity recognition method integrating entity label encoding, characterized in that, Includes the following steps: Step (1) Construct a dataset containing several samples and corresponding entity labels for each sample; Step (2) Use BERT to perform character embedding transformation on each character in the input sentence to obtain the character feature sequence; Step (3) uses CLN to represent the character feature sequence from step (2) as a word pair feature matrix V, and simultaneously calculates the distance between each character to represent it as a distance feature matrix E. d And a feature matrix E of upper and lower triangular regions. t The features of the three feature matrices are concatenated to obtain the word pair relation feature matrix C; Step (4) convert each entity label into natural language and then encode these labels using BERT to obtain the feature representation L of each entity label; Step (5) For each word pair relation feature representation in the word pair relation feature matrix C, calculate the dot product similarity with the feature representation of each entity label to obtain the transition probability matrix between each word pair and each entity label; Step (6) Use labeled samples to train the model composed of steps (2)-(5), and select the optimal model as the final pre-trained model; Step (7) Load the pre-trained model, label the samples for few-sample learning, input the sentence into the trained model, output the word pair relation matrix Y, decode the word pair relation matrix, and obtain the final entity words and their types.

2. The few-sample learning named entity recognition method fused with entity label encoding according to claim 1, characterized in that, The specific implementation process of step (2) is as follows: BERT is used to perform character embedding transformation on each character in the input sentence to obtain a character-level embedding representation; for an input sentence of length N, X = {x1, x2, ..., x...} N }∈R N Use Bert to process each character x i Encode to obtain character feature sequence d h R is the dimension of the character feature, and R is the dimension representing the feature.

3. The few-sample learning named entity recognition method fused with entity label encoding according to claim 2, characterized in that, In step (3), the method for obtaining the word pair feature matrix V is as follows: The character feature sequence is represented as a word pair feature matrix using CLN. Where V ij The calculation formula is: V ij for h i Regarding h j The feature representation, where γ ij =W α h i +b α , λ ij =W β h i +b β W is obtained after training through fully connected layers. α and W β It is a learnable weight matrix, b α and b β These are learnable bias vectors, where μ and σ are h. i The formulas for the mean and standard deviation are shown below: Where d h h represents the dimension of the character features. jk Let be the k-th element of the j-th character feature in the character feature sequence H.

4. The few-sample learning named entity recognition method integrating entity label encoding according to claim 3, characterized in that, In step (3), the distance feature matrix E d How to obtain: The distance feature matrix is ​​obtained by calculating the distance between each character feature. For the input sentence X, two words (x i ,x j The distance between two words is represented as their absolute distance |ij|. After passing through an embedding layer, a distributed distance feature matrix is ​​obtained.

5. The few-sample learning named entity recognition method fused with entity label encoding according to claim 4, characterized in that, In step (3), the regional network matrix By manually generating an N×N matrix, all values ​​in the upper triangular region are set to 1, while those in the lower triangular region are set to 2. This is then passed through an embedding layer to obtain the feature matrices for the upper and lower triangular regions.

6. A few-sample learning named entity recognition method integrating entity label encoding according to claims 3-5, characterized in that, In step (3), the word pair feature matrix V and the distance feature matrix E are... d and the feature matrix E of the upper and lower triangular regions t The features of these three feature matrices are concatenated to obtain the word pair relation feature matrix. in 7. The few-sample learning named entity recognition method fused with entity label encoding according to claim 6, characterized in that, The specific implementation process of step (4) is as follows: Each entity label undergoes natural language processing. W2NER defines two word relationships: adjacency relationships, indicating that two words can be adjacent, and entity relationships, indicating that two words are the head and tail of an entity, respectively. Therefore, the number of labels is the number of entity types + 2. These labels are adjacency relationship labels and no-relationship labels. BERT is then used to encode these labels to obtain the feature representation of each entity label.

8. The few-sample learning named entity recognition method fused with entity label encoding according to claim 7, characterized in that, The specific implementation process of step (5) is as follows: For each word pair, the relation feature representation is c. i ,i∈{1,2,…,N×N}, and the feature representation l of each entity label j ,j∈{1,2,…,N L Calculate the dot product similarity, and then perform a softmax operation to obtain the transition probability matrix between each word pair and each entity label. The calculation formula is as follows: y ij =softmax(c i ·l j )。 9. The few-sample learning named entity recognition method fused with entity label encoding according to claim 8, characterized in that, In step (6), the BERT pre-trained model is selected as "bert-base-uncased", the epoch is set to 5, the learning rate is 0.00001, the batch size is 4, the Adam optimizer is used to progressively adjust the learning rate, and the model is trained using standard cross-entropy loss, as shown in the following formula: in, Indicates whether it is a correct label, y ij The predicted transition probability. After multiple rounds of training, the optimal model is selected as the pre-trained model.

10. The few-sample learning named entity recognition method fused with entity label encoding according to claim 9, characterized in that, In step (7), the optimal model after training is loaded, and the dataset is sampled in a k-shot manner. For n-shot, n corpora are randomly sampled from each entity type as input for continued training. The Bert pre-trained model is selected as "bert-base-uncased", the epoch is set to 100, the learning rate is 0.00001, the batch size is 4, the Adam optimizer is used to progressively adjust the learning rate, and the model is trained using standard cross-entropy loss. After training with a few samples, the sentences are input into the trained model, and the output word pair relation matrix Y∈R is generated. N×N The word pair relation matrix is ​​decoded to obtain the final entity words and their types.

Images