Nested named entity recognition method based on multi-head three- affine attention mechanism

Abstract

The application discloses a nested named entity recognition method based on a multi-head three- affine attention mechanism. The method can capture deep word semantic information, solve the problem that a span-based method has high requirements for nested entity boundary recognition and has large calculation amount for learning all candidate spans. The span representation is obtained by using the multi-head three- affine attention mechanism, the span boundary and internal information are fused, and long-span nested named entities are captured. The entity boundary is smoothed by using a regularization technique, the accuracy of model boundary recognition is improved, and a nested named entity recognition task is carried out. The method can capture deep word semantic information, solve the problem that a span-based method has high requirements for nested entity boundary recognition and has large calculation amount for learning all candidate spans.

Description

Technical Field

[0001] This invention belongs to the field of named entity recognition technology, and specifically relates to a nested named entity recognition method based on a multi-head three-affine attention mechanism. Background Technology

[0002] Nested named entities are a special type of named entity where one or more other entities exist within a single entity. For example, "Three Gorges University" is an organization name type entity, while "Three Gorges" in "Three Gorges University" is also a place name type entity. Traditional sequence labeling-based named entity models struggle to handle such nested named entities directly and effectively. Generally, non-nested named entities are called Flat NERs, and nested named entities are called Nested NERs.

[0003] Currently, many technical solutions have been proposed for named entity recognition. For example, the "Named Entity Recognition Method and Apparatus" disclosed in Chinese patent document CN114330348A incorporates prior knowledge to enhance the model's learning ability in this domain, assisting the pre-trained model in making predictions in this domain and improving recognition accuracy. However, the recognition of nested named entities remains quite difficult. Therefore, it is necessary to design a recognition method to address the problem of large computational demands across such large sums. Summary of the Invention

[0004] In view of the technical problems existing in the background technology, the nested named entity recognition method based on multi-head three-affine attention mechanism provided by the present invention can capture deep word semantic information and solve the problems of high requirements for nested entity boundary recognition and large computational amount for learning all candidate spans in span-based methods.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A nested named entity recognition method based on a multi-head tri-affine attention mechanism is proposed. The method employs a nested entity recognition model architecture based on this mechanism, consisting of an encoding layer, a tri-affine mechanism layer, and a prediction layer. The recognition method is as follows:

[0007] Step 1: Encode the sentence by combining character features, part-of-speech embedding, word embedding vectors, and a pre-trained language model to obtain the semantic information of the sentence;

[0008] Step 2: Use a multi-head three-affine attention mechanism to interactively learn the above information to obtain a label-level span representation. Map the label-level span representation to a higher embedding dimension to obtain a more complete span representation.

[0009] Step 3: Smooth the boundaries of real entities and then perform a span-nested named entity recognition task based on this.

[0010] Preferably, in step 1, in the encoding layer, the pre-trained language models BERT and Bi-LSTM are used as encoders to aggregate the character features, part-of-speech features and word features of words from the input sentence to obtain the encoding vector of the sentence;

[0011] In step 2, at the three-affine mechanism layer, the first stage uses the three-affine attention mechanism to generate the span representation, and the second stage uses the three-affine to interact with the span boundary and the span representation, and the span boundary and the fused information interact in a higher order to obtain a new span representation, which provides a basis for subsequent span classification.

[0012] In step 3, the true boundary is regularized and smoothed in the prediction layer, and classified in combination with the entity span score.

[0013] Preferably, the operation steps of step 1 are as follows:

[0014] Step 1.1: For a given sentence, BERT is used as the word embedding model. The input vector of BERT is composed of character feature vectors, part-of-speech feature vectors, and word feature vectors. After BERT calculation, max pooling is used to obtain the word vector of each word.

[0015] After the BERT output, Bi-LSTM is used to generate the final word representation, i.e.:

[0016] x = LSTM(S) = {x1, x2, ..., x} n}∈R n×d (1)

[0017] Where n represents the length of the sentence and d represents the dimension of the word.

[0018] Preferably, step 2 is performed as follows:

[0019] Step 2.1: After obtaining word representations from Bi-LSTM, three independent MLPs are applied to represent the boundary information h of span-nested named entities. i ,h j The representation of nested named entities with spans h k ; where h i The header and h represent the span. j The tail of the span, h k h represents the span itself. i ,h j ,h k ∈R d ;;

[0020] To incorporate more information about entities and obtain better information representation, a three-affine attention mechanism is used for computation:

[0021] TirAff(u,v,w,ω)=ω×u′×w′×v′

[0022] =ω×[MLP(u)1]×[MLP(w)]×[MLP(v)1] (2)

[0023] h i ,h j ,h k =MLP(x) (3)

[0024] s i,j,h,r =TirAff(h i ,h j ,h k ,w r (4)

[0025]

[0026]

[0027] P i,j,k =TirAff(h i ,h j ,h i,j,r VR ) (7)

[0028] Among them, W r ∈R d+1 ×R d ×R d+1 It is a parameter vector, boundary information (h) i ,h j ) and label parameter W r It can be viewed as an attention query vector Q, with span represented by h. k It can be viewed as a key vector K and a value vector V.

[0029] h i,j,k It is h i ,h j ,h k The calculation shows that when MLP is 0,

[0030]

[0031] make:

[0032]

[0033] have to:

[0034]

[0035] Pre-trained language models are used to learn more knowledge independently and prevent overfitting. They utilize multi-head parallel computing to effectively avoid significant increases in computational cost and the number of parameters.

[0036] Preferably, the operation steps of step 3 are as follows:

[0037] Step 3.1: Boundary smoothing. Each span is assigned a full probability of becoming an entity, while all unlabeled spans are assigned zero probability; this probability assignment is called hard boundary. With a smoothing size of D, all spans with a Manhattan distance d (d≤D) to the labeled entity share an equal probability ε / D, i.e.:

[0038]

[0039] y ij It is a true span nested named entity (x) i ,x j The vector representation of ε is the entity boundary smoothing coefficient, and D is the smoothing dimension, which is the Manhattan distance from the annotation span to the surrounding area, indicating first-order or second-order smoothing of the entity's surroundings. It is the entity vector representation after boundary smoothing;

[0040] Step 3.2: Loss Function: For each sentence x = {x1, x2, ..., x...} n Our objective function for training is to minimize the log-likelihood loss with respect to the corresponding label, as shown in the formula:

[0041] y i,j,k =softmax(P i,j,k ) (10)

[0043]

[0044] Where N represents the number of words in the sentence. It means (x) i ,x j The vector of the true labels after span smoothing, y i,j,k This represents the probability vector of the prediction.

[0045] This patent can achieve the following beneficial effects:

[0046] 1. This invention can capture deep word semantic information and solve the problems of high requirements for nested entity boundary recognition and large computational cost of learning all candidate spans in span-based methods.

[0047] 2. Experiments were conducted on two large-scale benchmark nested NER datasets, ACE2004 and ACE2005, and a real-world medical nested NER dataset, GENIA. The results achieved state-of-the-art F1 scores. Furthermore, with the same system hyperparameters, the proposed model has fewer parameters, consumes fewer resources, and takes less time to train, demonstrating the effectiveness of the proposed method. Attached Figure Description

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0049] Figure 1 This is a diagram illustrating nested named entity instances from Embodiment 1 of the present invention;

[0050] Figure 2 This is a span representation diagram based on three affines according to Embodiment 1 of the present invention;

[0051] Figure 3 This is a model framework diagram of Embodiment 1 of the present invention;

[0052] Figure 4 This is a diagram of the three-affine attention mechanism in Embodiment 1 of the present invention;

[0053] Figure 5 This is a score calculation diagram based on the three-affine attention mechanism in Embodiment 1 of the present invention;

[0054] Figure 6 This is an example diagram of hard and smooth boundaries in Embodiment 1 of the present invention. Detailed Implementation

[0055] Example 1:

[0056] Preferred solutions include Figures 1 to 6 As shown, a nested named entity recognition method based on a multi-head tri-affine attention mechanism is presented. The method employs a nested entity recognition model architecture based on this mechanism, consisting of an encoding layer, a tri-affine mechanism layer, and a prediction layer. The recognition method is as follows:

[0057] Step 1: Encode the sentence by combining character features, part-of-speech embedding, word embedding vectors, and a pre-trained language model to obtain the semantic information of the sentence;

[0058] Step 2: Use a multi-head three-affine attention mechanism to interactively learn the above information to obtain a label-level span representation. Map the label-level span representation to a higher embedding dimension to obtain a more complete span representation.

[0059] Step 3: Smooth the boundaries of real entities and then perform a span-nested named entity recognition task based on this.

[0060] Specifically, at the encoding layer, pre-trained language models BERT and Bi-LSTM are used as encoders to aggregate character features, part-of-speech features, and word features from the input sentence to obtain the sentence's encoding vector;

[0061] In the three-affine mechanism layer, the first stage uses the three-affine attention mechanism to generate the span representation. The second stage will use the three-affine to interact with the span boundary and the span representation, and the span boundary and the fused information will interact in a higher order to obtain a new span representation, which will provide a basis for subsequent span classification.

[0062] In the prediction layer, the true boundary is regularized and smoothed, and then classified by combining the entity span score.

[0063] Preferably, the operation steps of step 1 are as follows:

[0064] Step 1.1: For a given sentence, BERT is used as the word embedding model. The input vector of BERT is composed of character feature vectors, part-of-speech feature vectors, and word feature vectors. After BERT calculation, max pooling is used to obtain the word vector of each word.

[0065] To further enhance the retention of important information in the coding layer, Bi-LSTM is used to generate the final word representation after the BERT output, i.e.:

[0066] x = LSTM(S) = {x1, x2, ..., x} n}∈R n×d (1)

[0067] Where n represents the length of the sentence and d represents the dimension of the word.

[0068] Preferably, step 2 is performed as follows:

[0069] Step 2.1: After obtaining word representations from Bi-LSTM, three independent MLPs are applied to represent the boundary information h of span-nested named entities. i ,h j The representation of nested named entities with spans h k ; where h i The header and h represent the span. j The tail of the span, h k h represents the span itself. i ,h j ,h k ∈R d ;

[0070] Compared to models that directly use LSTM output, this effectively reduces data dimensionality, while the independent representation of span-nested named entities and their boundaries also improves the accuracy of the model's representation.

[0071] To incorporate more information about entities and obtain better information representation, a three-affine attention mechanism is used for computation, such as... Figure 4 .

[0072] TirAff(u,v,w,ω)=ω×u′×w′×v′

[0073] =ω×[MLP(u)1]×[MLP(w)]×[MLP(v)1] (2)

[0074] h i ,h j ,h k =MLP(x) (3)

[0075] s i,j,k,r =TirAff(h i ,h j ,h k ,w r (4)

[0076]

[0077]

[0078] P i,j,k =TirAff(h i ,h j ,h i,j,r ,v r (7)

[0079] Among them, W r ∈R d+1 ×R d ×R d+1 It is a parameter vector, boundary information (h) i ,h j ) and label parameter W r It can be viewed as an attention query vector Q, with span representing h. k It can be viewed as a key vector K and a value vector V.

[0080] Compared to general attention frameworks (additive or multiplicative attention), triaffine attention allows for higher-order interactions between all heterogeneous matrices K and V, meaning that in addition to learning the boundaries and representations of span-nested named entities, it also learns the representations of span-nested named entities quite well.

[0081] h i,j,k It is hi ,h j ,h k The calculation shows that when MLP is 0,

[0082]

[0083] make:

[0084]

[0085] have to:

[0086]

[0087] Multi-head attention integrates different knowledge generated from the same attention pooling, where the differences stem from different subspace representations of the same query, key, and value. We reference the multi-head attention mechanism and utilize multi-head tri-affine attention for computation, such as... Figure 5 On the one hand, the model can learn more knowledge independently and prevent overfitting. On the other hand, by using multi-head parallel computing, significant increases in computational cost and the number of parameters can be effectively avoided.

[0088] Preferably, the operation steps of step 3 are as follows:

[0089] Step 3.1: Smooth the boundaries, as shown below. Figure 6 Each span is assigned a full probability of becoming an entity, while all unlabeled spans are assigned zero probability; this probability assignment is called a hard boundary. In real-world data, entity boundaries may be ambiguous and inconsistent due to data labeling or quality issues, but this ambiguity and inconsistency significantly impacts the model's recognition performance. This invention proposes the concept of boundary smoothing, where, given an annotated entity, a portion of the probability ε is assigned to its surrounding spans, and the remaining probability, 1-ε, is assigned to the original spans. With a smoothing size of D, all spans with a Manhattan distance d (d≤D) to the annotated entity receive an equal probability ε / D, i.e.:

[0090]

[0091] y ij It is a true span nested named entity (x) i ,x j The vector representation of ε is the entity boundary smoothing coefficient, and D is the smoothing dimension, which is the Manhattan distance from the annotation span to the surrounding area. It is usually taken as 1 or 2, indicating that the first or second order smoothing is applied to the area around the entity. It is the entity vector representation after boundary smoothing;

[0092] Step 3.2: Loss Function: For each sentence x = {x1, x2, ..., x...}n The objective function of} is to minimize the log-likelihood loss with respect to the corresponding label, as shown in the formula:

[0093] y i,j,k =softmax(P i,j,k ) (10)

[0095]

[0096] Where N represents the number of words in the sentence. It means (x) i ,x j The vector of the true labels after span smoothing, y i,j,k This represents the probability vector for the prediction. Since the span table is only valid for the upper triangular region, the loss is calculated only for the upper triangular region.

[0097] Experimental results and analysis:

[0098] This invention was tested on the public English datasets ACE2004, ACE2005, and GENIA, which have nested named entity annotations. For ACE2004 and ACE2005, the datasets were partitioned into train / dev / test in an 8:1:1 ratio according to the method of Shibuya and Hovy (2020). For GENIA, the dataset publicly available in the code of Zheng Yuan et al. (2022) was used. The entity statistics results for the ACE2004, ACE2005, and GENIA datasets are shown in Table 7.

[0099]

[0100] Table 7

[0101] ACE2004 and ACE2005 English datasets

[21] : have 7 types of entities, including people (PER), geographical locations (LOC), vehicles (VEH), political entities (GPE), weapons (WEA), organizations (ORG), and facilities (FAC). This invention follows the data partitioning method of Lu et al., which was adopted by most people before, and divides the dataset into training set, validation set and test set in a ratio of 80%:10%:10%. GENIA dataset

[22] : has 39 fine-grained categories of biological entities. This invention follows the work of Manning et al., classifies protein and its subclass entities as Protein, and uses the same processing method for RNA, DNA, CellType, CellLine and their subclasses. Finally, it only contains entities of 5 categories: Protein, RNA, DNA, CellType and CellLine, and divides the dataset in a ratio of 81%:9%:10%.

[0102] Ablation experiments; as shown in Table 8, ablation experiments were conducted on the ACE2005 dataset to demonstrate the performance improvement brought by multi-head tri-affine attention and boundary smoothing. The results are shown in the table. We removed the Bi-LSTM layer and directly fed the output of the pre-trained language model into the tri-affine decoder.

[0103]

[0104]

[0105] Table 8

[0106] The results show that the lack of a Bi-LSTM layer leads to a decrease in F1 scores of approximately 0.3, 0.5, and 0.1 percentage points across the three datasets. Without the triaffine attention mechanism, directly utilizing biaffine prediction after obtaining the context embedding from BERT results in a significant performance drop, decreasing by 1.67%. Boundary smoothing contributes a 0.58% gain to the model.

Claims

1. A nested named entity recognition method based on a multi-head three-affine attention mechanism, characterized in that: A nested entity recognition model architecture based on a multi-head tri-affine attention mechanism is adopted: the nested entity recognition model architecture consists of three parts: an encoding layer, a tri-affine mechanism layer, and a prediction layer; the recognition method is as follows: Step 1: Encode the sentence by combining character features, part-of-speech embedding, word embedding vectors, and a pre-trained language model to obtain the semantic information of the sentence; Step 2: Use a multi-head three-affine attention mechanism to interactively learn the above information to obtain a label-level span representation. Map the label-level span representation to a higher embedding dimension to obtain a more complete span representation. Step 3: Smooth the boundaries of real entities and perform a span-nested named entity recognition task based on this. The steps for step 1 are as follows: Step 1.1: For a given sentence, BERT is used as the word embedding model. The input vector of BERT is composed of character feature vectors, part-of-speech feature vectors, and word feature vectors. After BERT calculation, max pooling is used to obtain the word vector of each word. To further enhance the retention of important information in the coding layer, Bi-LSTM is used to generate the final word representation after the BERT output, i.e.: （1）； Where n represents the length of the sentence and d represents the dimension of the word; Step 2 is performed as follows: Step 2.1: After obtaining word representations from Bi-LSTM, three independent MLPs are applied to represent the boundary information of span-nested named entities. , Representation of nested named entities ;in, The header indicating the span The tail of the span, Indicates the span itself, ; To incorporate more information about entities and obtain better information representation, a three-affine attention mechanism is used for computation: （2）； （3）； （4）； （5）； （6）； （7）； in, It is a parameter vector, boundary information. and label parameters It can be viewed as an attention query vector Q, with span representing... It can be viewed as a key vector K and a value vector V; It is by , , The calculation shows that when MLP is 0, （8）； make: （9）； have to: （10）； Pre-trained language models are used to learn more knowledge independently and prevent overfitting. They utilize multi-head parallel computing to effectively avoid significant increases in computational cost and the number of parameters.

2. The nested named entity recognition method based on multi-head three-affine attention mechanism according to claim 1, characterized in that: In step 1, at the encoding layer, the pre-trained language models BERT and Bi-LSTM are used as encoders to aggregate the character features, part-of-speech features, and word features of words from the input sentence to obtain the sentence's encoding vector; In step 2, at the three-affine mechanism layer, the first stage uses the three-affine attention mechanism to generate the span representation, and the second stage uses the three-affine to interact with the span boundary and the span representation, and the span boundary and the fused information interact in a higher order to obtain a new span representation, which provides a basis for subsequent span classification. In step 3, the true boundary is regularized and smoothed in the prediction layer, and classified in combination with the entity span score.

3. The nested named entity recognition method based on multi-head three-affine attention mechanism according to claim 1, characterized in that: The steps for step 3 are as follows: Step 3.1: Boundary smoothing, each span is assigned a full probability of becoming an entity, while all unlabeled spans are assigned zero probability; in the case of smoothing size D: （11）； It is a real span nested named entity. The vector representation of , It is the entity boundary smoothing coefficient, and D is the smoothing size, which is the Manhattan distance from the annotation span to the surrounding area, indicating that the first or second order smoothing is applied to the entity's surrounding area. Step 3.2: Loss Function: For each sentence Our objective function for training is to minimize the log-likelihood loss with respect to the corresponding label, as shown in the formula: （12）； （13）； Where N represents the number of words in the sentence. express The vector of the true label after span smoothing. This represents the probability vector of the prediction.

Images