Multi-attention mechanism span-level few-shot named entity recognition method

Abstract

The present application relates to a multi-attention mechanism span-level small sample named entity recognition method, and belongs to the field of natural language processing. The present application firstly performs span enumeration on a sentence and generates span coherence representation through a BERT encoder; secondly, proposes a span-in attention mechanism to focus on important character information, and introduces a span-between attention mechanism to focus on other spans with high relevance to the target span; then, utilizes a span boundary attention mechanism to focus on the common features of the support set entity span boundary, and utilizes an instance-level span attention mechanism to focus on the support set sample span with high relevance to the query set sample span, to generate an entity category prototype; finally, calculates the distance between the support set and the query set to perform entity category prediction. The present application can improve the semantic truncation problem in the token-level metric-based meta-learning method, and effectively improve the accuracy of the small sample named entity recognition task.

Description

Technical Field

[0001] This invention relates to a multi-attention mechanism span-level few-sample named entity recognition method, belonging to the field of natural language processing. Background Technology

[0002] Named entity recognition (NER) is a fundamental task in natural language processing, aiming to identify specific types of entities in text. It serves as the basis for downstream tasks such as information extraction and question answering systems, and has a wide range of applications. Current deep learning-based NER methods have achieved considerable success, but their effectiveness relies on large-scale labeled data. However, data labeling is costly and expensive, and in some practical application areas, such as biomedicine, sufficiently large labeled datasets are often unavailable. Therefore, researching small-sample NER methods with limited labeled data has significant theoretical and practical value. These methods can generally be categorized into two types:

[0003] 1. Fine-tuning-based methods

[0004] Fine-tuning-based methods are a traditional approach to few-shot learning. This method trains the model on the source dataset, then fine-tunes it on the training set of the target dataset, and finally evaluates it on the target dataset's test set. This approach fully leverages the powerful generalization capabilities of deep learning neural networks, avoids redesigning a new model, eliminates the need for training from scratch, and achieves faster convergence and better results, making it suitable for few-shot named entity recognition tasks. However, when the target and source datasets differ significantly, fine-tuning-based methods require adjusting model parameters on a smaller training set within the target dataset, which may lead to overfitting.

[0005] 2. Metric-based learning methods

[0006] Metric-based learning methods utilize prototype networks to compute prototypes, thereby measuring token-level similarity between the support set and query set for prediction, assigning a label to each token. In named entity recognition tasks, this approach outperforms fine-tuning-based methods in small sample scenarios. However, named entities typically consist of text spans, i.e., multiple tokens, with several adjacent tokens belonging to the same entity. Choosing a single token as the metric granularity may truncate semantic information, impacting the performance of named entity recognition.

[0007] In summary, existing fine-tuning-based few-shot named entity recognition methods require optimization on a small training set without a validation set in the target domain, leading to overfitting. Furthermore, most existing methods train at the token level, neglecting the textual span of sentences, resulting in truncated semantic information. Therefore, this invention proposes a multi-attention mechanism span-level few-shot named entity recognition method. Summary of the Invention

[0008] The purpose of this invention is to address the problems that fine-tuning-based methods may overfit and that choosing token-level metric granularity in few-shot named entity recognition may truncate semantic information. This invention proposes a multi-attention mechanism span-level few-shot named entity recognition method to improve the accuracy of few-shot named entity recognition.

[0009] The design principle of this invention is as follows: First, the sentence is enumerated for spans and encoded using a BERT encoder to generate a span coherence representation; second, an intra-span attention mechanism is proposed to focus on important character information, and an inter-span attention mechanism is introduced to focus on other spans that are highly correlated with the target span; then, a span boundary attention mechanism is used to focus on the common features of the span boundaries of entities in the support set, and an instance-level span attention mechanism is used to focus on the spans of support set samples that are highly correlated with the spans of the query set samples, thereby generating entity category prototypes; finally, entity category prediction is performed by calculating the distance between the support set and the query set.

[0010] The technical solution of the present invention is achieved through the following steps:

[0011] Step 1: Enumerate the span of the sentence and generate a span coherence representation by encoding it with a BERT encoder.

[0012] Step 1.1: Enumerate all possible spans in the support set and query set.

[0013] Step 1.2: Encode the sentences for the support set and query set using the BERT encoder.

[0014] Step 1.3: Obtain the initial representation of the span using the span coherent representation method.

[0015] Step 2 proposes an intra-span attention mechanism to focus on important character information, and introduces an inter-span attention mechanism to focus on other spans that are highly relevant to the target span.

[0016] Step 2.1: Apply attention to the span to enhance the representation of the overall meaning of the span by the information of each character within the span.

[0017] Step 2.2: Use other spans within the sentence and the target span for attention to enhance the representation of the meaning of the target span by other span information within the sentence.

[0018] Step 2.3: Input the span representation into a feedforward neural network with residual connections and layer normalization to obtain the final span-enhanced representation.

[0019] Step 3: Use the span boundary attention mechanism to focus on the common features of the span boundaries of entities in the support set, and use the instance-level span attention mechanism to focus on the span of support set samples that are highly correlated with the span of the query set samples, and generate entity category prototypes.

[0020] Step 3.1: Utilize span boundary attention to focus on the common features of the span boundaries of entities in the support set, and obtain the common entity span boundary feature vectors among the entity samples.

[0021] Step 3.2: Use the instance-level span attention mechanism to focus on important spans in the support set samples that are related to the span of the query set samples, and generate entity category prototypes.

[0022] Step 4: Entity category prediction is performed by calculating the distance between the support set and the query set.

[0023] Step 4.1: Calculate the predicted probability by using the span augmentation representation and the distance between all category prototypes to obtain the prediction result.

[0024] Beneficial effects

[0025] Compared to fine-tuning-based methods, this invention avoids the problem of overfitting.

[0026] Compared to metric-based learning methods, this invention proposes a span coherence representation method to obtain the initial representation of the span, storing the coherence information of the span. It enhances the representation of the overall meaning of the span by individual characters within the span through span-internal attention, enhances the representation of the target span's meaning by other span information within the sentence through attention to other spans within the sentence, enhances the entity span boundary information through entity span boundary attention, and generates entity category prototypes by using instance-level span attention to important spans in the support set samples related to the query set sample spans. Finally, it predicts entity categories by calculating the distance between the support set and the query set, thereby improving the problem of truncated semantic information in token-level metric-based learning methods and further improving the accuracy of few-sample named entity recognition tasks. Attached Figure Description

[0027] Figure 1 A schematic diagram of the principle of a multi-attention mechanism span-level small sample named entity recognition method.

[0028] Figure 2 Schematic diagram of feature-level attention extractor Detailed Implementation

[0029] To better illustrate the purpose and advantages of the present invention, the implementation methods of the present invention will be further described in detail below with reference to examples.

[0030] The specific process is as follows:

[0031] Step 1: Enumerate the span of the sentence and generate a span coherence representation by encoding it with a BERT encoder.

[0032] Step 1.1: For a sentence containing n words, enumerate all spans of length 1 to n in the support set and query set, resulting in n(n+1) / 2 spans. For a span s = [l, r], where l and r are the start and end indices of the tokens contained in the span within the sentence. Obtain the span label and span mask based on the span. The rule for obtaining the span label is: if the label corresponding to each word contained in the span is equal, then the span label of the span is assigned to this same label; otherwise, the span label of the span is assigned to class O (other classes). The span mask is used to distinguish between the actual span and the padding items; the value corresponding to the actual span is 1, and the value corresponding to the padding items is 0.

[0033] Step 1.2: Encode the sentences in the support set and query set using the BERT encoder, and use the sum of the last four hidden layers to represent the context embedding vector of each token in the sentence. Since the BERT-base model is used, the vector size obtained from this model is 768. Therefore, for a sentence x = (x1, x2, ..., x...) with N tokens... N ), which can yield the vector representation [e1,…,e N ] and position vector representation [p1,…,p N ].

[0034] Step 1.3: Obtain the initial representation of the span using the span coherent representation method. Step 1.1 yields a given span s = [i,j], where i and j are the start and end indices of the tokens contained within the span in the sentence, and the corresponding context word embeddings [e] are obtained through the BERT encoder. i ,…,e j ], where e k ∈R d d = 768. To obtain the initial representation of the span, the vector of each token in the span is first divided into 4 parts, as shown in formula (1):

[0035]

[0036] in, And 2a + 2b = d, a = 352, v = 32.

[0037] The initial representation of the span consists of three parts: the span start character vector, the span end character vector, and a scalar value used to measure the relevance between the start and end characters. The scalar value measuring the relevance between the start and end characters can store span coherence information. The initial representation of the span is shown in formula (2):

[0038]

[0039] Part 3 is and dot product, s ij ∈R 2a+1 .

[0040] Step 2 proposes an intra-span attention mechanism to focus on important character information, and introduces an inter-span attention mechanism to focus on other spans that are highly relevant to the target span.

[0041] Step 2.1: Perform attention within the span to enhance the representation of the overall meaning of the span by the information of each character within the span. Given a span s = [i,j] and its context word embeddings [e...]... i ,…,e j ], where e k ∈R d The position vector represents [p1,…,p N Attention is applied to the interior of the span to enhance the span representation, and this is concatenated with the initial span representation to obtain the span representation s. p The calculation formulas are shown in formulas (3), (4), (5), and (6):

[0042]

[0043]

[0044] a k =softmax(α) k (5)

[0045]

[0046] Where v is the learned parameter vector, i and j are the start and end indices of the tokens contained in the sentence, and p k-i+1 It is the relative position encoding of the span start boundary token.

[0047] Step 2.2: Use other spans within the sentence and the target span for attention to enhance the representation of the target span's meaning through other span information. Multiply the span representations within the sentence that have received intra-span attention by their relevance score to the target span (calculated by the attention mechanism) and sum them to the target span representation to obtain the target span representation enhanced by other spans within the sentence. For all span representations S∈R obtained in Step 2.1... B×d (B is the number of spans, and d is the span representing the dimension size), the p-th row of S is the p-th span s in the sentence. p For s p Use other spans within the sentence and the target span for attention to enhance the target span and connect it with the span representation s p The span is obtained by splicing. The calculation formulas are shown in formulas (7) and (8):

[0048]

[0049]

[0050] Where i and j are the spans s p The start and end indices of the included token within the sentence.

[0051] Step 2.3: Input the span representation into a feedforward neural network with residual connections and layer normalization to obtain the final span representation after internal span enhancement and inter-sentence span enhancement. The calculation formulas are shown in formulas (9) and (10):

[0052]

[0053] FFN(s p ) = (GeLU(s) p W 1 +b 1 ))W 2 +b 2 (10)

[0054] Where LayerNorm represents the layer normalization function, GeLU represents the activation function, and W 1 and W 2 These are the weight matrices for the first and second linear layers of the FFN, respectively. 1 and b 2 These are the bias matrices for the first and second linear layers of the FFN, respectively.

[0055] Step 3: Use the span boundary attention mechanism to focus on the common features of the span boundaries of entities in the support set, and use the instance-level span attention mechanism to focus on the important spans in the support set samples that are related to the span of the query set samples, and generate entity category prototypes.

[0056] Step 3.1 utilizes span boundary attention to focus on the common features of entity span boundaries in the support set, obtaining the entity span boundary feature vector common to the entity samples. When enhancing the entity span representation, all spans are enumerated, and the introduced "O" class (non-entity) spans blur the entity span boundaries. Span boundary attention focuses more on the common aspects of all entity spans in the support set (i.e., entity features and entity span boundary features), thereby enhancing the entity span boundary information in the entity prototype vector. This enhances the representation of all entity spans in the support set. n represents the number of entity samples in the support set, which is input into a feature-level attention extractor. Extractor such as Figure 2 As shown, the extractor calculates the linear separability of each dimension of the feature based on the distribution represented by each entity span, thereby obtaining the entity span boundary feature vector b common to the entity samples. The calculation formula is shown in formula (11):

[0057]

[0058] The entity span boundary attention layer first takes all entity span augmented representations in the support set as input to the convolutional neural network. After three convolutional layers, it outputs an entity span boundary feature vector *b* with the same dimensions as the entity span augmented representations. *b* will be used during entity prototype computation. The specific structure of the convolutional neural network is as follows: the first convolutional layer has 32 channels, a kernel of K×1, and a stride of 1×1; the second convolutional layer has 64 channels, a kernel of K×1, and a stride of 1×1; the third convolutional layer has 1 channel, a kernel of K×1, and a stride of K×1, and uses the ReLU activation function after each convolution computation.

[0059] Step 3.2 utilizes an instance-level span attention mechanism to focus on important spans in the support set samples that are related to the span of the query set samples, generating entity category prototypes. Different support set samples have different importance in the query sample representation. During prototype calculation, different weight coefficients are assigned to the spans in the support set based on the correlation between the query span and the spans in the support set, generating different category prototypes for different query spans. For the i-th class in the support set, it contains a span-enhanced representation with K annotations. For the m-th span augmentation representation among all M span augmentation representations in the query set The corresponding prototype representation of the i-th type entity is obtained by calculation, and the calculation formulas are shown in formulas (12) and (13):

[0060]

[0061]

[0062] Here, softmax represents the normalized exponential function.

[0063] Step 4: Entity category prediction is performed by calculating the distance between the support set and the query set.

[0064] Step 4.1: Calculate the predicted probability using the distance between the span augmentation representation and all category prototypes to obtain the prediction result. During the training step, the parameters are updated in each meta-task. In the testing step, the prediction result is the label of the category prototype closest to the span augmentation representation. That is, given a task T = (S, q), where S is the support set, q is the query set, N represents the types, and n is the number of types, for the m-th span in the query set q, its span augmentation representation is: And the corresponding category prototype vector is The prediction process is shown in formulas (14) and (15):

[0065]

[0066]

[0067] Among them, y m And i is a type in N, It is a class prototype vector of type i.

[0068] The experimental data comes from the Few-NERD dataset, a fine-grained dataset specifically designed for few-shot named entity recognition scenarios, including 188,238 sentences and 66 fine-grained entity types. Experiments were conducted in INTRA and INTER modes, using the cross-entropy loss function and the AdamW optimization method with a learning rate of 1e4.

[0069] The experiment uses precision, recall, and F1 score to evaluate the results of named entity recognition. The calculation formulas are shown in formulas (16), (17), and (18):

[0070]

[0071]

[0072]

[0073] Where tp and fp are the number of positive and negative samples that are classified as positive, respectively, and fn is the number of positive samples that are classified as negative.

[0074] This experiment was conducted on one computer and one server. The computer's specific configuration was: Intel i5-8250, 8GB RAM, and Windows 10 64-bit operating system; the server's specific configuration was: NVIDIA GeForce RTX 2080Ti graphics card and Linux Ubuntu 64-bit operating system.

[0075] Test Results: Based on the multi-attention mechanism span-level few-shot named entity recognition method, the experiment was conducted on the Few-NERD dataset. The highest F1 score of 64.98% was achieved in the 5-way-5 to 10-shot settings of the Few-NERD(INTER) dataset. This invention can effectively improve the accuracy on few-shot named entity recognition tasks.

[0076] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-attention mechanism span-level few-shot named entity recognition method, characterized in that... The method includes the following steps: Step 1: First, enumerate all feasible spans of length 1 to n in the support set and query set, and obtain... The sentence is encoded using a BERT encoder for the support set and query set, and the context embedding vector representation of each token in the sentence is represented by the sum of the last four hidden layers. Finally, the initial representation of the span is obtained through the span coherent representation method. Step 2: First, attention is applied within the span to enhance the representation of the overall meaning of the span by the information of each character within the span. Attention is then applied between the target span and other spans within the sentence to enhance the representation of the target span by the information of other spans within the sentence. The span representations within the sentence that have been enhanced with attention are multiplied by the relevance score of the target span calculated by the attention mechanism and accumulated onto the target span representation to obtain the target span representation enhanced by other spans within the sentence. Finally, the span representation is input into a feedforward neural network with residual connections and layer normalization to obtain the final span representation enhanced by both internal span enhancement and mutual enhancement between spans within the sentence. Step 3: First, using span boundary attention, focus on the common features of the span boundaries of entities in the support set to obtain the common entity span boundary feature vectors in the entity samples. Then, using the instance-level span attention mechanism, focus on the important spans in the support set samples that are related to the span of the query set samples. During the prototype calculation process, assign different weight coefficients to the spans in the support set according to the correlation between the query span and the spans in the support set, and generate different category prototypes for different query spans. Step 4: Entity category prediction is performed by calculating the distance between the support set and the query set, the distance between the span augmentation representation and all category prototypes is calculated, the prediction probability is calculated, and the prediction result is obtained.

2. The multi-attention mechanism span-level few-shot named entity recognition method according to claim 1, characterized in that: In step 1, regarding the span , where i and j represent the start and end indices of the tokens contained in the span within the sentence, respectively; to obtain the initial representation of the span, the vector of each token in the span is first divided into 4 parts: ,in, , ,and , , The initialization of the span consists of three parts: a vector of the span's starting character, a vector of the span's ending character, and a scalar value used to measure the relevance between the starting and ending vectors. The initialization of the span is as follows: Part 3 is and dot product, , .

3. The multi-attention mechanism span-level few-shot named entity recognition method according to claim 1, characterized in that: In step 2, attention is applied to the span to enhance the span representation. Given a span... and its context word embedding ,in Position vector representation Attention is applied to the interior of the span to enhance the span representation, and this is concatenated with the initial span representation to obtain the final span representation. for: ; ; ; ,in, It is the learned parameter vector. and These are the start and end indices of the token contained within the sentence. It is the relative position encoding of the span start boundary token.

4. The multi-attention mechanism span-level few-shot named entity recognition method according to claim 1, characterized in that: Step 2 uses attention between other spans within the sentence and the target span to enhance the target span, representing all the spans within the sentence. Where B is the number of spans, and d is the span representing the size of the dimension. The The line is the first line in a sentence. span ,for Use other spans within the sentence and the target span for attention to enhance the target span and its representation. The span is obtained by splicing. for: ; ,in, and It is the span The start and end indices of the included token within the sentence.

5. The multi-attention mechanism span-level few-sample named entity recognition method according to claim 1, characterized in that: Step 3 utilizes span boundary attention to focus on the common features of the entity span boundaries in the support set, obtaining the entity span boundary feature vector common to the entity samples, thus enhancing the span representation of all entities in the support set. , This represents the number of entity samples in the support set, which is then input into a feature-level attention extractor. The extractor calculates the linear separability of each dimension of the feature based on the distribution represented by each entity span, thereby obtaining the entity span boundary feature vector common to the entity samples. , The entity span boundary attention mechanism first uses all entity span augmented representations in the support set as input to a convolutional neural network. After passing through three convolutional layers, it outputs an entity span boundary feature vector with the same dimension as the entity span representation. , The convolutional neural network, which will be used in the calculation of the physical prototype, has the following structure: the first layer has 32 convolutional channels, and the convolutional kernel... Step length The second convolutional layer has 64 channels and a certain kernel size. Step length The third convolutional layer has 1 channel and a convolutional kernel. Step length And the ReLU activation function is used after each convolution calculation.

Images