A nested named entity recognition method based on multi-directional gradient feature extraction

Abstract

This invention belongs to the field of natural language processing technology and discloses a nested named entity recognition method based on multi-directional gradient feature extraction. The method includes the following steps: preprocessing a text dataset; inputting the preprocessed sentences into a model to obtain contextual features of word vectors; representing sentences with contextual information features in a planar format; extracting multi-directional entity semantic edge features by combining channel-wise convolution with an extended edge gradient operator; using pointwise convolution to perform spatial concatenation to obtain higher-order features, feeding them into a multilayer perceptron, performing residual concatenation with the planar sentence representation, and using Softmax and Argmax to predict classification and return index values, thus completing candidate entity selection. This invention employs the aforementioned nested named entity recognition method based on multi-directional gradient feature extraction, using an extended eight-directional Sobel operator in the planar sentence representation to extract entity semantic edge features, making them more complete and discriminative.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a nested named entity recognition method for multi-directional gradient feature extraction. Background Technology

[0002] Information extraction aims to extract structured information from large-scale unstructured or semi-structured natural language text. Its main tasks include entity recognition, relation extraction, and event extraction. Among these, Named Entity Recognition (NER) aims to identify predefined noun phrases with specific meanings in a given sentence, such as people, locations, and geographic locations. As a fundamental task in many applications, including knowledge graphs, question answering, and machine translation, NER has received widespread attention and research in the field of natural language processing.

[0003] Early named entity recognition was typically implemented as a sequence labeling task, assigning a label to each element in a sentence to indicate its semantic role within the named entity framework. However, a problem with sequence models is the difficulty in identifying nested named entities within a sentence. Nested named entities refer to the phenomenon where an entity contains one or more other entities, with the same element corresponding to multiple labels. For example, in "Peking University," "Beijing" is labeled as a geographic entity, and "Peking University" is labeled as an organizational entity, where the entity "Peking University" contains another entity, "Beijing."

[0004] For nested named entity recognition tasks, relevant research models can be broadly categorized into hierarchical sequence-based models and span-based classification models. Hierarchical sequence-based models identify nested entities by dynamically stacking planar sequence recognition layers. However, this approach introduces the problem of error cascading and only guarantees unidirectional information transmission, preventing inner layers from utilizing information from outer layers. Span-based models treat entity recognition as a span classification task, enumerating overlapping spans to unfold nested semantic structures and fully utilizing the labeled characteristics within spans. However, existing span models primarily classify based on independent, fragmented spans, disrupting their interaction and failing to encode the global semantic features of the sentence.

[0005] To overcome the shortcomings of span-based classification models, some studies have structured the exhaustive spans into two-dimensional sentence representations, which can unfold nested structures within sentences and encode interactions between words. However, adjacent spans share the same context, leading to semantic overlap of entities in the planar sentence representation. Therefore, edge detection techniques from image processing are combined, using gradient operators to enhance and extract semantic edge features of entities in the planar sentence representation, and employing the Laplace operator to capture semantic gradients between adjacent spans in the top, bottom, left, and right directions. However, the above studies only consider edge information in four directions between adjacent spans, ignoring feature information in other directions, resulting in missing semantic edge features of the extracted entities. Summary of the Invention

[0006] The purpose of this invention is to provide a nested named entity recognition method based on multi-directional gradient feature extraction. Building upon edge detection methods in the field of image processing, this invention employs an extended eight-directional Sobel operator in planar sentence representation to address the problem of insufficient extraction of entity semantic edge features, thereby extracting more complete and discriminative entity semantic edge features in two-dimensional representation.

[0007] To achieve the above objectives, this invention provides a nested named entity recognition method based on multi-directional gradient feature extraction, comprising the following steps:

[0008] Step S1: Preprocess the text dataset, that is, process the raw data into data suitable for the entity model;

[0009] Step S2: Input the preprocessed sentence from Step S1 into the pre-trained BERT model to obtain the contextual features of the word vectors;

[0010] Step S3: Perform a flattened sentence representation on the sentences with contextual information features obtained in step S2;

[0011] Step S4: Extract multi-directional entity semantic edge features from the sentence after the planarization of the sentence in step S3 by combining channel-wise convolution with the extended edge gradient operator; then, spatially connect the extracted features using point-wise convolution to obtain higher-order features.

[0012] Step S5: First, the high-order features obtained in step S4 are fed into a multilayer perceptron, and then residual connections are made with the sentence represented by the planarized sentence obtained in step S3. Finally, Softmax and Argmax are used to predict the classification and return the index value, thus completing the screening of candidate entities.

[0013] Preferably, in step S1, the text dataset is preprocessed, that is, the original data is processed into data suitable for the entity model; the position of the entity in the sentence is marked, from the beginning to the end, and the entity type is marked with type; and the sentence structure is obtained through pre-training.

[0014] Preferably, in step S2, the sentence preprocessed in step S1 is input into the pre-trained BERT model to obtain the contextual features of the word vectors. The specific process is as follows:

[0015] First, for a dataset containing N words x = [x i For sentences where 1 ≤ i ≤ N, each word x i Convert them into word fragments and feed them into a pre-trained BERT model;

[0016] Then, a Bi-LSTM network is used to train the word embedding vectors, running both forward and backward to fuse the information from both ends. This allows the model to comprehensively consider the contextual information at each time step in the input sequence, resulting in the final word representation H.

[0017] H = {h1, h2, ..., h} N};

[0018] Among them, h i ,1≤i≤N represents the concatenation of the bidirectional representations of the i-th word.

[0019] Preferably, in step S3, the sentence with contextual information features obtained in step S2 is represented in a flattened form, as follows:

[0020] The input to a multi-headed biaffine is two matrices H. s H e ∈Γ N×h The output is R∈Γ N×N×r The planarized sentence representation R is obtained using a multi-head dual-affine decoder, as shown below:

[0021]

[0022] R = MHBiaffine(H s H e );

[0023] Where N, h, and r represent the number of words in the sentence, the size of the hidden vector, and the size of the feature, respectively; W s and W e Let Γ represent the word embedding vectors at the start and end positions of each word in the sentence, respectively; each cell in the output R represents the feature vector v∈Γ representing the span. r .

[0024] Preferably, in step S4, the sentence after planarization in step S3 is processed by combining channel-wise convolution with the extended edge gradient operator to extract multi-directional entity semantic edge features; the sentence after feature extraction is then spatially connected using pointwise convolution to obtain higher-order features, as detailed below:

[0025] Step S41: Use depthwise separable convolution to support the multi-directional Sobel gradient operator; wherein, depthwise separable convolution is decomposed into depthwise convolution and 1×1 convolution, i.e. pointwise convolution.

[0026] Depthwise convolution performs individual convolution operations on each channel of the planarized sentence representation R using the gradient operator to extract semantic edge features of entities, as shown below:

[0027]

[0028] in, This represents a flattened sentence representation after depthwise convolution; w (d) Represents the depthwise convolution operation function; G∈Γ K×K The derivative operator in filter mask form is represented by K, where K represents the size of the derivative operator.

[0029] Step S42: Perform a standard 1×1 convolution operation using pointwise convolution to learn the interaction between different channels, as shown below:

[0030]

[0031] in, This represents a flattened sentence representation after pointwise convolution; w (p) C represents the pointwise convolution operation function; C represents the convolution kernel;

[0032] Step S43: Combine the above two convolution formulas into a unified form, as shown below:

[0033] w (s) (R,G,C)=w (p) (w (d) (R,G),C);

[0034] The specific formulas for the eight different directions are expressed as follows:

[0035]

[0036] Among them, w (0) w (45) w (90) ···w (315)Let G0 and G1 represent the gradients of the elements at 0°, 45°, 90°, ..., 135° respectively; 45 G 90 ···G 315 These represent the derivative operators corresponding to the 0° directional filter mask form, the 45° directional filter mask form, the 90° directional filter mask form, ..., the 135° directional filter mask form, respectively.

[0037] Step S44: The gradient of each element is the maximum absolute value of the gradient values ​​in eight directions, as shown below:

[0038] G[w (0,45,···,315) ]≈max{|w0|,|w 45 |,|w 90 |,···,|w 315 |};

[0039] Finally, the sentence after entity semantic edge feature extraction is represented as W.

[0040] Preferably, in step S5, the high-order features obtained in step S4 are first fed into a multilayer perceptron, then residually connected with the planarized representation of the sentence obtained in step S3, and finally Softmax and Argmax are used to predict the classification and return the index value, thus completing the selection of candidate entities. The specific process is as follows:

[0041] Step S51: First, W is directly fed into the MLP layer to further mix semantic information;

[0042] Step S51: Then, combine W and R to represent A to predict the distribution of named entities, as shown below:

[0043] P θ (C|A) = Softmax(Linear(A));

[0044] Where C represents the predicted entity label, which is an N×N entity label matrix; element C i,j It corresponds to the entity span A i,j The entity labels; θ is the parameter that the model needs to learn during training;

[0045] During training, the cross-entropy function is used as the loss function, given the input X and the true label matrix. The formula for calculating the loss function is as follows:

[0046]

[0047] Where z is the number of NER categories + 1, representing non-entities; y ij It is A ij Output entity labels; N 2 It is a normalization factor;

[0048] Step S53: Finally, predict the entity label for each span by selecting the category with the highest probability, as shown below:

[0049]

[0050] Among them, P θ Y represents the probability distribution of the model's prediction of a certain entity label given parameters θ; i,j represents the entity tag value corresponding to the span from the i-th word to the j-th word; e represents the specific entity tag category.

[0051] Therefore, this invention adopts the above-mentioned nested named entity recognition method with multi-directional gradient feature extraction. Compared with the prior art, this invention makes full use of the complete information of the sentence text and uses the extended eight-directional Sobel operator in the planar sentence representation to solve the problem of insufficient semantic edge feature extraction. This allows for the extraction of more complete and discriminative entity semantic edge features in the planar sentence representation, thereby improving the performance of the nested named entity recognition task and achieving excellent results in entity recognition.

[0052] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0053] Figure 1 This is the technical approach of the nested named entity recognition method for multi-directional gradient feature extraction in this invention;

[0054] Figure 2 This is an entity recognition model diagram of a nested named entity recognition method for multi-directional gradient feature extraction according to the present invention;

[0055] Figure 3 This is a schematic diagram of the extended eight-direction Sobel operator of the present invention. Detailed Implementation

[0056] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0057] like Figure 1 and Figure 2 As shown, a nested named entity recognition method using multi-directional gradient feature extraction includes the following steps:

[0058] Step S1: Preprocess the text dataset, that is, process the raw data into data suitable for the entity model.

[0059] The entity is marked in the sentence from beginning to end, and the entity type is marked with type. The sentence structure is obtained through pre-training.

[0060] Step S2: Input the preprocessed sentence from step S1 into the model to obtain the contextual features of the word vectors.

[0061] For a dataset containing N words x = [x i For sentences where 1 ≤ i ≤ N, each word x i The words are converted into word fragments and fed into a pre-trained BERT module. After BERT computation, each word in a sentence may involve vector representations of several fragments.

[0062] Next, a Bi-LSTM network is used to train the word embedding vectors, running both forward and backward to fuse the information from both ends. This allows the model to comprehensively consider the contextual information at each time step in the input sequence, resulting in the final word representation H.

[0063] H = {h1, h2, ..., h} N};

[0064] Among them, h i ,1≤i≤N represents the concatenation of the bidirectional representations of the i-th word.

[0065] Step S3: Perform a flattened sentence representation on the sentences with contextual information features obtained in step S2.

[0066] The H obtained in step S2 is input into the multi-head biaffine to obtain a planarized sentence vector representation. The input to the multi-head biaffine is two matrices H. s H e ∈Γ N×h The output is R∈Γ N×N×r Therefore, a planarized sentence representation R is obtained using a multi-head dual-affine decoder, as shown below:

[0067]

[0068] R = MHBiaffine(H s H e );

[0069] Where N, h, and r represent the number of words in the sentence, the size of the hidden vector, and the size of the feature, respectively; W s and W eLet Γ represent the word embedding vectors at the start and end positions of each word in the sentence, respectively; each cell (i,j) in the output R can be viewed as a feature vector v∈Γ representing the span. r .

[0070] Step S4: Extract multi-directional entity semantic edge features from the sentence after the planarization of the sentence in step S3 by combining channel-wise convolution with the extended edge gradient operator; then, spatially connect the extracted features using point-wise convolution to obtain higher-order features.

[0071] Step S41: Use depthwise separable convolution to support the multi-directional Sobel gradient operator, such as... Figure 3 As shown.

[0072] Depthwise separable convolution can be decomposed into depthwise convolution and 1×1 convolution (also known as pointwise convolution). Depthwise convolution performs individual convolution operations on each channel of the planarized sentence representation R using the gradient operator to extract semantic edge features of entities, as shown below:

[0073]

[0074] in, This represents a flattened sentence representation after depthwise convolution; w (d) It can be viewed as a depthwise convolution operation function; G∈Γ K×K This represents the derivative operator in filter mask form, where K represents the size of the derivative operator.

[0075] Step S42: Perform a standard 1×1 convolution operation using pointwise convolution to learn the interaction between different channels, as shown below:

[0076]

[0077] in, This represents a flattened sentence representation after pointwise convolution; w( p ) can be viewed as a pointwise convolution operation function; C represents the convolution kernel.

[0078] Step S43: Combine the above two convolution formulas into a unified form, as shown below:

[0079] w (s) (R,G,C)=w (p) (w (d) (R,G),C);

[0080] The specific formulas for the eight different directions are expressed as follows:

[0081]

[0082] Among them, w (0) w (45) w (90) ···w (315) Let G0 and G1 represent the gradients of the elements at 0°, 45°, 90°, ..., 135° respectively; 45 G 90 ···G 315 These represent the derivative operators corresponding to the 0° direction filter mask form, the 45° direction filter mask form, the 90° direction filter mask form, ..., the 135° direction filter mask form, respectively.

[0083] Step S44: The gradient of each element is the maximum absolute value of the gradient values ​​in eight directions, as shown below:

[0084] G[w (0,45,···,315) ]≈max{|w0|,|w 45 |,|w 90 |,···,|w 315 |};

[0085] Finally, the sentence after entity semantic edge feature extraction is represented as W.

[0086] Step S5: First, the high-order features obtained in step S4 are fed into a multilayer perceptron, then residual connections are made with the planarized representation of the sentence obtained in step S3, and finally Softmax and Argmax are used to predict the classification and return the index value to complete the screening of candidate entities.

[0087] Step S51: First, W is directly fed into the MLP layer to further mix semantic information.

[0088] Step S51: Then, combine W and R to represent A to predict the distribution of named entities, as shown below:

[0089] P θ (C|A) = Softmax(Linear(A));

[0090] Where C represents the predicted entity label, which is an N×N entity label matrix; element C i,j It corresponds to the entity span A i,j The entity labels; θ is the parameter that the model needs to learn during training.

[0091] During training, the cross-entropy function is used as the loss function, given the input X and the true label matrix. The formula for calculating the loss function is as follows:

[0092]

[0093] Where z is the number of NER categories + 1 (representing non-entities); y ij It is A ij Output entity labels; N 2 It is the normalization factor.

[0094] Step S53: Finally, predict the entity label for each span by selecting the category with the highest probability, as shown below:

[0095]

[0096] Among them, P θ Y represents the probability distribution of the model's prediction of a certain entity label given parameters θ; i,j represents the entity tag value corresponding to the span from the i-th word to the j-th word; e represents the specific entity tag category.

[0097] Example 1

[0098] This embodiment implements the method of the present invention, mainly applied to nested entity data, using the GENIA and ACE2005 datasets. First, the position of the entity within the sentence is marked, from the beginning to the end, and the entity type is labeled with `type`. The sentence structure is then obtained through pre-training.

[0099] Then, for a given sentence, such as "Mr. A of a certain country unfortunately passed away", the vector representation of each word or character in the sentence is obtained through the pre-trained model BERT. The Bi-LSTM network is then used to train the character embedding vectors, running from front to back and from back to front to fuse the information before and after, so that the model can fully consider the contextual information before and after each time step in the input sequence, and obtain the final word representation.

[0100] Next, the word representation H is planarized into a sentence representation using a multi-head biaffine operation, resulting in a matrix-form sentence. The planarized sentence representation is then used to extract multi-directional entity semantic edge features by combining channel-wise convolution with an extended edge gradient operator. Finally, the extracted features are spatially concatenated using pointwise convolution to obtain higher-order features.

[0101] Finally, the obtained flattened sentence representation is residually connected with the sentence representation after feature extraction to predict the distribution of named entities. The final prediction results are the four entities: "a country", "A", "Mr. A", and "Mr".

[0102] Example 2

[0103] This embodiment utilizes the GENIA and ACE2005 datasets for evaluation. The ACE2005 dataset is a Chinese nested named entity dataset containing documents collected from news agencies, broadcasts, and blogs. It defines seven entity types: people, organizations, locations, facilities, weapons, vehicles, and geopolitical entities, totaling 45,112 entities. In processing the dataset, this invention divides it into training, validation, and test sets in an 8:1:1 ratio. The GENIA dataset contains 32 nested biomedical entity categories. This invention focuses on five entity types: DNA, RNA, protein, cell lineage, and cell type, and splits the training, validation, and test sets in an 8.1:0.9:1.0 ratio.

[0104] This invention compares current mainstream entity recognition models on these two datasets, using P-value, R-value, and F1 value to evaluate the performance of the model methods. As shown in Table 1, the experimental results of this invention are higher than those of other model methods, demonstrating the effectiveness of this invention in improving the performance of named entity recognition tasks.

[0105] Table 1. Experimental Comparison Results

[0106]

[0107] As shown in Table 2, ablation experiments were conducted on the key modules of this civilization on the GENIA dataset. Without residual connections, performance significantly decreased, demonstrating that removing residual connections reduces the model's fault tolerance. Removing the multi-directional gradient operator resulted in a 0.34% performance decrease, indicating that the multi-directional gradient operator has a positive impact on nested NER tasks. Using the multi-directional gradient operator in depthwise convolution effectively enhances the semantic representation of the actual corpus, extracting multi-directional semantic edge features between adjacent spans and improving the accuracy of named entity extraction from sentences. Removing the pointwise convolution module resulted in a 0.27% performance decrease, verifying the impact of 1×1 convolution operations on model performance, which achieves the fusion of cross-channel information.

[0108] Table 2 Ablation Experiment Results

[0109]

[0110] Therefore, this invention adopts the above-mentioned nested named entity recognition method with multi-directional gradient feature extraction. Compared with the prior art, this invention makes full use of the complete information of the sentence text and uses the extended eight-directional Sobel operator in the planar sentence representation to solve the problem of insufficient semantic edge feature extraction. This allows for the extraction of more complete and discriminative entity semantic edge features in the planar sentence representation, thereby improving the performance of the nested named entity recognition task and achieving excellent results in entity recognition.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A nested named entity recognition method for multi-directional gradient feature extraction, characterized in that, Includes the following steps: Step S1: Preprocess the text dataset, that is, process the raw data into data suitable for the entity model; Step S2: Input the preprocessed sentence from Step S1 into the pre-trained BERT model to obtain the contextual features of the word vectors; Step S3: Perform a flattened sentence representation on the sentences with contextual information features obtained in step S2; Step S4: Extract multi-directional entity semantic edge features from the planarized sentence representation obtained in Step S3 by combining channel-wise convolution with the extended edge gradient operator; then, spatially connect the extracted features using pointwise convolution to obtain higher-order features. The specific process is as follows: Step S41: Use depthwise separable convolution to support the multi-directional Sobel gradient operator; wherein, depthwise separable convolution is decomposed into depthwise convolution and 1×1 convolution, i.e. pointwise convolution. Depth convolution for planar sentence representation Each channel is convolved separately using the gradient operator to extract semantic edge features of the entity, as shown below: ； in, Indicates the output of a multi-head dual affine circuit; This represents a flattened sentence representation after depthwise convolution. This represents the depthwise convolution operation function; The derivative operator represents the filter mask form. This indicates the size of the derivative operator; Step S42: Perform pointwise convolution. The standard convolution operation learns the interaction between different channels, as shown below: ； in, This represents a flattened sentence representation that has undergone pointwise convolution. This represents the pointwise convolution operation function; Represents the convolution kernel; Step S43: Combine the above two convolution formulas into a unified form, as shown below: ； The specific formulas for the eight different directions are expressed as follows: ； in, , , These represent the gradients of the element at 0°, 45°, 90°, ..., 135° respectively. , , These represent the derivative operators corresponding to the 0° directional filter mask form, the 45° directional filter mask form, the 90° directional filter mask form, ..., the 135° directional filter mask form, respectively. Step S44: The gradient of each element is the maximum absolute value of the gradient values ​​in eight directions, as shown below: ； Finally, the sentence after entity semantic edge feature extraction is represented as: ; Step S5: First, the high-order features obtained in step S4 are fed into a multilayer perceptron, and then residual connections are made with the sentence represented by the planarized sentence obtained in step S3. Finally, Softmax and Argmax are used to predict the classification and return the index value, thus completing the screening of candidate entities.

2. The nested named entity recognition method for multi-directional gradient feature extraction according to claim 1, characterized in that: In step S1, the text dataset is preprocessed, that is, the raw data is processed into data suitable for the entity model; the position of the entity in the sentence is marked, from the beginning to the end, and the entity type is marked with type; the sentence structure is obtained through pre-training.

3. The nested named entity recognition method for multi-directional gradient feature extraction according to claim 1, characterized in that, In step S2, the sentence preprocessed in step S1 is input into the pre-trained BERT model to obtain the contextual features of the word vectors. The specific process is as follows: First, for datasets containing N one word The sentence, each word Convert them into word fragments and feed them into a pre-trained BERT model; Then, a Bi-LSTM network is used to train the word embedding vectors, running both forward and backward to fuse the information from both ends. This allows the model to comprehensively consider the contextual information at each time step of the input sequence, resulting in the final word representation. : ； in, Indicates the first The concatenation of bidirectional representations of words.

4. The nested named entity recognition method for multi-directional gradient feature extraction according to claim 1, characterized in that, In step S3, the sentence with contextual information features obtained in step S2 is represented in a flattened form. The specific process is as follows: The input to a multi-headed biaffine is two matrices. The output is A planarized sentence representation is obtained using a multi-head dual affine decoder. As shown below: ； ； in, , and These represent the number of words in the sentence, the size of the hidden vector, and the size of the feature, respectively. and Let each word in the sentence have a word embedding vector representing the start position and an end position, respectively; output the word embedding vector. Each cell in the table represents a feature vector of the span. .

5. The nested named entity recognition method for multi-directional gradient feature extraction according to claim 1, characterized in that, In step S5, the high-order features obtained in step S4 are first fed into a multilayer perceptron, then residually connected with the planarized representation of the sentence obtained in step S3, and finally Softmax and Argmax are used to predict the classification and return the index value, thus completing the selection of candidate entities. The specific process is as follows: Step S51: First, [the text abruptly ends here, likely due to an incomplete sentence or a formatting error.] It is directly fed into the MLP layer for further mixing of semantic information; among which, This is the sentence representation after entity semantic edge feature extraction. Step S52, then, and Combined, it can be expressed as To predict the distribution of named entities, as shown below: ； in, Indicates the output of a multi-head dual affine circuit; Represents the predicted entity labels; it is an N×N entity label matrix; elements It corresponds to the span of the entity. Entity tags; These are the parameters that the model needs to learn during the training process; During training, the cross-entropy function is used as the loss function, given the input X and the true label matrix. The formula for calculating the loss function is as follows: ； in, It is the number of NER categories + 1, indicating a non-entity; yes Output entity labels; It is a normalization factor; Step S53: Finally, predict the entity label for each span by selecting the category with the highest probability, as shown below: ； in, This indicates that the model is in response to given parameters. Predict the probability distribution of a certain entity label; Indicates from the first The word to the 1 The entity tag value corresponding to the span of each word; Indicates a specific entity label category.

Images