A nested named entity recognition method based on span boundary perception

Abstract

The application discloses the technical field of information extraction, and particularly relates to a nested named entity recognition method based on span boundary perception. The application aims to solve the problem of fuzzy semantic boundary between nested entities. Firstly, the application uses a double affine attention mechanism to capture the semantic correlation between word units, and then generates a span semantic representation matrix. Secondly, a second-order diagonal neighborhood difference operator is designed to establish a span semantic difference mechanism to extract the semantic difference information between spans. In addition, a span boundary perception mechanism is introduced to use the local feature extraction capability of a sliding window to strengthen the boundary semantic difference of the span, so as to accurately locate the entity span position. Finally, a linear decoder is used to predict the named entity type.

Description

Technical Field

[0001] This invention relates to the field of information extraction technology, specifically to a nested named entity recognition method based on span boundary awareness. Background Technology

[0002] With the rapid increase in the amount of text information, named entity recognition (NER) has become one of the most important tasks in the field of natural language processing (NLP). In traditional research, NER is often modeled as a sequence labeling task, using sequence models to output the sequence of labels with the highest probability. However, traditional sequence labeling models cannot handle nested structures. Nested named entity recognition, as an extension of NER, aims to identify two or more nested named entities. Therefore, nested named entity recognition is a more complex and challenging task within NER.

[0003] Current methods for span-based nested named entity recognition typically employ exhaustive span enumeration and simple aggregation strategies (such as pooling or concatenating start and end words) to construct span representations. However, due to the high degree of word overlap between nested entities, and even the sharing of the same start and end words, the similarity between nested entity representations generated by existing methods is high. This makes it difficult for classifiers to directly capture the semantic differences between spans, leading to the problem of ambiguous nested entity boundaries. To address this issue, this invention proposes a span-boundary-aware nested named entity recognition model. The model uses the semantic differences between spans as explicit input features, effectively alleviating the problem of ambiguous nested entity boundaries. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a nested named entity recognition method based on span boundary awareness. By introducing a span semantic difference mechanism and a span boundary awareness mechanism, semantic differences can be extracted from nested entities. Through gradient backpropagation, the model can learn appropriate internal representations, thereby increasing the distance between nested entities in the span semantic representation space, and effectively overcoming the problem of ambiguous semantic boundaries between nested entities.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] This invention provides a nested named entity recognition method based on span boundary awareness. The specific steps of the session recommendation method are as follows:

[0007] The specific steps of the session recommendation method are as follows:

[0008] Step S10: Construct a span semantic encoder. Based on the word-level feature encoding of the pre-trained language model BERT, introduce a double affine mechanism to construct a span semantic representation matrix.

[0009] Step S20: Design a semantic difference module. By designing a second-order diagonal neighborhood difference operator, the span semantic representation is transformed into a semantic difference representation between spans.

[0010] Step S30: Design a boundary awareness module, construct a learnable sliding convolutional kernel to enhance the semantic differences between spans, thereby perceiving the boundaries of potential named entities;

[0011] Step S40: Construct a span semantic decoder to classify potential named entities based on a linear decoder.

[0012] In the above technical solution, the specific steps of step S10 are as follows:

[0013] Step S101, given the sentence X = {x1, x2, ..., x...} l Using the pre-trained model BERT to process word units x i Vectorization yields the word-level feature representation vector H of the sentence;

[0014] Step S102: Design two feedforward neural networks to perform lexical mapping transformation, and obtain the semantic representation vectors (h) of the first and last lexical units of the span. s h e The calculation formula is as follows:

[0015] h s =GELU(HW s )

[0016] h e =GELU(HW e )

[0017] in, This indicates that the dimensions of the real matrix are l and h, respectively. The superscript h represents the number of hidden layers; Let the dimensions of the real matrix be h and h', both of which are trainable parameter matrices, r be the number of biaffine features, and GELU be the activation function;

[0018] Step S103: Use a dual affine model to fuse the span head term and tail term (h s h e And the width representation vector w corresponding to the span. ij Capture the semantic correlation between the initial and final word meta-semantic representations to construct a span representation matrix. in, The dimensions of this real matrix are l, l, and d, where l is the sentence length, d is the feature vector dimension of the span semantics, and S is the span with the position index i of the first word and the position index j of the last word. ij The corresponding span representation matrix M0 The representation vector in The calculation formula is as follows:

[0019] [i]Uh e [j] T

[0020] Among them, h s [i] means the semantic representation vector of the head word at position index i, h e [j] represents the semantic representation vector of the head word at position index j, h e [j] T h is the semantic representation vector of the head word at position index j. e The transpose of [j]; For span S ij The width of the embedding, where c is the number of embeddings for the span width; and This indicates that the dimensions of the real matrix are (2h+c) and r, respectively. Let the dimensions of this real matrix be h, r, and h', respectively, where h is the trainable parameter matrix, and r is the number of biaffine features. The operation represents the splicing of tensors, when M located at the output of the dual affine model 0 When the matrix is ​​in a non-diagonal position, the span S ij The representation of has two forms, namely S ij and S ji Furthermore, it is symmetrical along the diagonal, and H is the word-level feature representation vector of the sentence.

[0021] In the above technical solution, the specific steps of step S20 are as follows:

[0022] Step S201: Design a lightweight diagonal neighborhood difference operator by calculating the span representation matrix M. 0 Each span S ij Semantic distance between spans along the main diagonal and secondary diagonal directions and Construct the semantic difference representation matrix M between spans 1 and M 1′ , It is M 1 For a specific element in the matrix, the difference process of the operator is implemented using a convolution kernel. The convolution kernel parameters and calculation process are shown below:

[0023]

[0024]

[0025]

[0026] Step S202: Select the opposite direction of the main and secondary diagonals as the calculation direction, and further calculate the second-order span semantic difference representation matrix of the above span. and It is M 2 The vector in the i-th row and j-th column of the matrix, with specific convolution kernel parameters and calculation process, is shown below:

[0027]

[0028]

[0029]

[0030] Step S203: The feature representations calculated by each difference operator are processed using a normalization layer and activation function to obtain semantic difference features with stable distribution. This leads to the construction of a feature matrix representing the semantic difference information between spans, specifically the first-order span semantic difference representation matrix M. 1 The semantic difference representation matrix M of the second span 2 The specific calculation process is shown in the following formula. Similarly, M is calculated. 1′ and M 2 :

[0031] M 1 =D p Bolck(M 0 ) = GELU(LN(D p (M 0 ))),

[0032] M 2 =D Rp Bolck(M 1 ) = GELU(LN(D Rp (M 1 ))),

[0033] Among them, D p Block represents the semantic difference block along the main diagonal, D Rp Block represents the semantic difference block in the secondary diagonal direction, and LN is the normalization layer, which is the layer that normalizes the feature dimension d.

[0034] In the above technical solution, the specific steps of step S30 are as follows:

[0035] Step S301: Process the semantic difference information representation matrix M obtained by the semantic difference module along the main diagonal and secondary diagonal directions using a sliding convolution kernel. 3 and M 3′ :

[0036] M 3 =GELU(LN(Conv(M) 2 )))

[0037] M 3′ =GELU(LN(Conv(M) 2′ ))),

[0038] Step S302: Using a linear layer to fuse semantic difference information in the main diagonal and secondary diagonal directions, the representation of semantic boundary information between spans is enhanced, and a span boundary matrix is ​​obtained. This helps the model locate potential named entities more effectively. The specific calculation process is as follows:

[0039]

[0040] in, For a trainable weight matrix, This indicates that the dimensions of the real matrix are (d×2) and d, respectively. Let be a trainable bias matrix. This indicates that the dimension of the real matrix is ​​d. To ensure the consistency of the training results, all convolutional kernels do not contain bias terms and are filled with 0.

[0041] In the above technical solution, the specific steps of step S40 are as follows:

[0042] To preserve the complete semantic information of the span, M 0 Added as a residual term to M 3 In this process, feature representations that retain complete semantic information are obtained. Finally, a linear decoder outputs the class probabilities of the latent entities.

[0043]

[0044] in, l and t, where t is the number of predefined entity categories. This indicates that the dimensions of the real matrix are l, t All of these are trainable parameter matrices. This indicates that the dimension of the real matrix is ​​t. Let the dimensions of the real matrix be d and t, respectively.

[0045] The beneficial effects of this invention are as follows: This invention introduces a second-order diagonal neighborhood difference operator, enabling the effective capture and extraction of semantic differences between spans. This helps to better understand and distinguish the semantic differences between different spans, thereby improving the accuracy of named entity recognition. Furthermore, this invention utilizes a span boundary awareness mechanism, combined with the local feature extraction capability of a sliding window, to effectively enhance the semantic differences at span boundaries. This is particularly important in the case of complex nested structures, where it plays a crucial role in accurately locating the span position of entities. Attached Figure Description

[0046] Figure 1 This is a flowchart of the method steps of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Please see Figure 1 The present invention provides a technical solution:

[0049] For news texts in the field of breaking news, this invention provides the following technical solution: a nested named entity recognition method based on span boundary awareness. The specific steps of the named entity recognition method are as follows:

[0050] Step 1: Input news text sentences and construct a span semantic encoder. Based on the word-level feature encoding of the pre-trained language model BERT, a dual affine mechanism is introduced to construct a span semantic representation matrix;

[0051] Step 1-1: Given a news text sentence X = {he, then, read, at, New York, University}, use the pre-trained model BERT to analyze the word units x. i Vectorization yields the word-level feature representation vector H of sentence X.

[0052] Steps 1-2 involve designing two feed-forward neural networks (FNNs) to perform lexical mapping transformations, obtaining semantic representation vectors (h) of the first and last words across the span. s h e ).

[0053] Steps 1-3: Use a dual affine model to fuse the span head term and tail term (h s h e And the width representation vector w corresponding to the span.ij Capture the semantic correlation between the initial and final word meta-semantic representations to construct a span representation matrix. Where l is the sentence length and d is the feature vector dimension of the span semantics. Compared to directly concatenating or pooling the first and last words, the dual-affine mechanism used in this model effectively captures the semantic connections between the first and last words by calculating the correlation between them. Furthermore, the dual-affine mechanism outputs the semantic features of the span formed by each word and other words (e.g., the span formed by "he" and other words in the sentence: "he then", "he reads", and "he reads at"), constructing a span semantic matrix, which facilitates subsequent boundary information extraction and entity decoding. For a span S with the position index i of the first word and the position index j of the last word... ij Its representation vector The calculation formula is as follows:

[0054]

[0055] Among them, h s [i] means the semantic representation vector of the head word at position index i, h e [j] represents the semantic representation vector of the head word at position index j, h e [j] T h is the semantic representation vector of the head word at position index j. e The transpose of [j]; For span S ij The width of the embedding, where c is the number of embeddings for the span width; and This indicates that the dimensions of the real matrix are (2h+c) and r, respectively. Let the dimensions of this real matrix be h, r, and h', respectively, where h is the trainable parameter matrix, and r is the number of biaffine features. The operation represents the splicing of tensors, when M located at the output of the dual affine model 0 When the matrix is ​​in a non-diagonal position, the span S ij The representation of has two forms, namely S ij and S ji Furthermore, it is symmetrical along the diagonal, and H is the word-level feature representation vector of the sentence.

[0056] Step 2: Design a semantic difference module. By designing a second-order diagonal neighborhood difference operator, the semantic representation of span is transformed into a semantic difference representation between spans.

[0057] Step 2-1: Design a lightweight diagonal neighborhood difference operator to calculate the semantic distance between the spans along the principal diagonal and counter diagonal directions. and A semantic difference representation matrix between spans is constructed. In the model, the difference process of the operator is specifically implemented using convolution kernels. The convolution kernel parameters and calculation process are shown below:

[0058]

[0059]

[0060]

[0061] Step 2-2: Select the opposite direction i of the main and secondary diagonals as the calculation direction, and further calculate the second-order span semantic difference representation of the above span. and The specific convolution kernel parameters and calculation process are shown below:

[0062]

[0063]

[0064]

[0065] Compared to the Roberts operator, the diagonal neighborhood difference operator proposed in this invention uses a 3×3 template, avoiding the span feature drift problem that occurs in the Roberts operator during computation. Secondly, to avoid difference object drift when computing higher-order semantic difference information, the span semantic difference mechanism also designs a difference operator in the opposite direction, rather than simply reusing the original difference operator.

[0066] Steps 2-3, to ensure the stability of the training process, use Layer Normalization (LN) and GELU activation functions to process the feature representations calculated by each difference operator, in order to obtain semantic difference features with stable distribution, and then construct a feature matrix to represent semantic difference information between spans. 1 and M 2 The specific calculation process is shown in the following formula, M 1′ and M 2′ The calculation process is similar.

[0067] M 1 =D p Bolck(M 0 ) = GELU(LN(D p (M0 ))),

[0068] M 2 =D Rp Bolck(M 1 ) = GELU(LN(D Rp (M 1 ))),

[0069] Among them, D p Block represents the semantic difference block along the main diagonal. D Rp Block represents the semantic difference block along the secondary diagonal. LN is the normalization layer, which normalizes the feature dimension d.

[0070] Step 3: Design a boundary awareness module, construct a learnable sliding convolutional kernel, enhance the semantic differences between spans, and output the span boundary matrix;

[0071] Step 3-1: Use a sliding convolution kernel to process the span semantic difference information representation matrix obtained by the semantic difference module along the main and secondary diagonal directions.

[0072] Step 3-2: Then, the semantic difference information in the main and secondary diagonal directions is fused using a linear layer to enhance the representation of semantic boundary information between spans, obtaining a span boundary matrix, thereby more effectively helping the model locate potential named entities. The specific calculation process is as follows:

[0073] M 3 =GELU(LN(Conv(M) 2 )))

[0074] M 3′ =GELU(LN(Conv(M) 2′ ))),

[0075]

[0076] in, To ensure the consistency of training results, all convolutional kernels in this module do not contain bias terms and are filled with 0

[30] .

[0077] Step 4: Construct a span semantic decoder, classify latent named entities based on the linear decoder, and output the predicted entities ("New York" and "New York University");

[0078] To preserve the complete semantic information of the span, M 0 Added as a residual term to M 3 In this process, feature representations that retain complete semantic information are obtained. Finally, a linear decoder outputs the class probabilities for all spans (e.g., the latent entities “NYU”, “New York”, and “NYU”).

[0079]

[0080] in, l and t, where t is the number of predefined entity categories. This indicates that the dimensions of the real matrix are l, and All of these are trainable parameter matrices. This indicates that the dimension of the real matrix is ​​t. Let the dimensions of the real matrix be d and t, respectively.

[0081] The nested named entity recognition method based on span boundary awareness includes a span semantic encoder, a semantic difference module, a boundary awareness module, and a span semantic decoder, specifically:

[0082] A span semantic encoder is constructed by introducing a dual affine mechanism on the basis of word-level feature encoding of the pre-trained language model BERT to construct a span semantic representation matrix.

[0083] By designing a second-order diagonal neighborhood difference operator, the span semantic representation is transformed into a semantic difference representation between spans;

[0084] Construct a learnable sliding convolutional kernel to enhance semantic differences between spans, thereby perceiving the boundaries of potential named entities;

[0085] Classify potential named entities based on a linear decoder.

[0086] The method and system of this invention first utilize a dual affine attention mechanism to capture semantic relevance between lexical units, thereby generating a span semantic representation matrix. Second, a span semantic difference mechanism is established by designing a second-order diagonal neighborhood difference operator to extract semantic differences between spans. Furthermore, a span boundary awareness mechanism is introduced, leveraging the local feature extraction capability of a sliding window to enhance the semantic differences at the span boundaries, thus accurately locating the entity span position. Finally, a linear decoder is used to predict the named entity type.

[0087] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A nested named entity recognition method based on span boundary awareness, characterized in that... This includes the following steps: Step S10: Construct a span semantic encoder. Based on the word-level feature encoding of the pre-trained language model BERT, introduce a double affine mechanism to construct a span semantic representation matrix. Step S20: Design a semantic difference module. By designing a second-order diagonal neighborhood difference operator, the span semantic representation is transformed into a semantic difference representation between spans. Step S30: Design a boundary awareness module, construct a learnable sliding convolutional kernel to enhance the semantic differences between spans, thereby perceiving the boundaries of potential named entities; Step S40: Construct a span semantic decoder and classify potential named entities based on a linear decoder; The specific steps of step S10 are as follows: Step S101, given the sentence is Using the pre-trained model BERT to extract word units Vectorization yields word-level feature representation vectors of the sentence. ; Step S102: Design two feedforward neural networks to perform lexical mapping transformation, and obtain the semantic representation vectors of the first and last lexical units of the span. The calculation formula is as follows: in, , This indicates that the dimensions of the real matrix are l and h, respectively. superscript The number of hidden layers; , Let the dimensions of this real matrix be h and h', both of which are trainable parameter matrices; r is the number of biaffine features; GELU is the activation function. Step S103, use a dual affine model to fuse the span head term and tail term ( ) and the width representation vector corresponding to the span Capture the semantic correlation between the initial and final word meta-semantic representations to construct a span representation matrix. ,in, , Let the dimensions of this real matrix be l, l, and d, respectively. Let d be the length of the sentence, and d be the dimension of the feature vector for the span semantics. The span is defined by the position index i of the first word and the position index j of the last word. The corresponding span represents the matrix. The representation vector in The calculation formula is as follows: in, The meaning is the semantic representation vector of the head word at position index i. The meaning is the semantic representation vector of the head word element with position index j. for The transpose of the matrix; For span The embedding width, where c is the number of embeddings for the span width; and , The dimensions of this real matrix are respectively and r, The dimensions of this real matrix are h, r, and h', which are trainable parameter matrices, and r is the number of biaffine features. The operation represents the splicing of tensors, when Located at the output of the double affine model When the matrix is ​​in a non-diagonal position, the span The representation of has two forms, namely and And it is symmetrical along the diagonal. This is a vector representing the word-level features of the sentence; The specific steps of step S20 are as follows: Step S201: Design a lightweight diagonal neighborhood difference operator, and compute the span representation matrix. Each span Semantic distance between spans along the main diagonal and secondary diagonal directions and Construct a semantic difference representation matrix across spans and , yes For a specific element in the matrix, the difference process of the operator is implemented using a convolution kernel. The convolution kernel parameters and calculation process are shown below: , Step S202: Select the opposite direction of the main and secondary diagonals as the calculation direction, and further calculate the second-order span semantic difference representation matrix of the above span. and , yes The vector in the i-th row and j-th column of the matrix, with specific convolution kernel parameters and calculation process, is shown below: , Step S203 involves processing the feature representations calculated by each difference operator using a normalization layer and activation function to obtain semantic difference features with stable distribution. This process then constructs a feature matrix representing the semantic difference information between spans, and a first-order span semantic difference representation matrix. And second-order span semantic difference matrix The specific calculation process is shown in the following formula, and the calculation is similar. and : , , in, This represents the semantic difference block along the main diagonal. This represents a semantic difference block along the secondary diagonal, and LN is a normalization layer, meaning that this layer normalizes the feature dimension d.

2. The nested named entity recognition method based on span boundary awareness according to claim 1, characterized in that: The specific steps of step S30 are as follows: Step S301: Process the semantic difference information representation matrix obtained by the semantic difference module along the main diagonal and secondary diagonal directions using a sliding convolution kernel. and : , Step S302: Using a linear layer to fuse semantic difference information in the main diagonal and secondary diagonal directions, the representation of semantic boundary information between spans is enhanced, and a span boundary matrix is ​​obtained. This helps the model locate potential named entities more effectively. The specific calculation process is as follows: , in, , where is a trainable weight matrix. The dimensions of this real matrix are respectively and d, , is a trainable bias matrix. This indicates that the dimension of the real matrix is ​​d. To ensure the consistency of the training results, all convolutional kernels do not contain bias terms and are filled with 0.

3. The nested named entity recognition method based on span boundary awareness according to claim 2, characterized in that: The specific steps of step S40 are as follows: To preserve the complete semantic information of the span, Added as a residual term In this process, feature representations that retain complete semantic information are obtained. Finally, a linear decoder outputs the class probabilities of the latent entities. in, l and t, where t is the number of predefined entity categories. This indicates that the dimensions of the real matrix are l, and All of these are trainable parameter matrices. This indicates that the dimension of the real matrix is ​​t. Let the dimensions of the real matrix be d and t, respectively.

Images