Text named entity recognition model and method

Abstract

The invention relates to the technical field of named entity recognition, and discloses a text named entity recognition model and method. Wherein the model comprises an embedded layer, a multi-layer perceptron, a global context enhancement module, a word pair relative position coding attention module, a layered feature fusion module and a prediction layer; the embedding layer is used for generating a text embedding representation according to the text data; the multi-layer perceptron is used for converting the text embedded representation into a two-dimensional word pair matrix; the global context enhancement module is used for generating global context enhancement representation according to the two-dimensional word pair matrix; the word pair relative position coding attention module is used for generating word pair relative position coding attention representation according to the two-dimensional word pair matrix; the hierarchical feature fusion module is used for generating a score matrix according to the global context enhancement representation, the word pair relative position coding attention representation and the two-dimensional word pair matrix; and the prediction layer is used for outputting a named entity prediction result according to the score matrix. According to the invention, the accuracy of text named entity recognition is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of named entity recognition, and in particular to a text named entity recognition model and method. Background Art

[0002] Named Entity Recognition (NER) is a key task in the field of natural language processing, which aims to automatically identify and classify named entities such as names of people, places, organizations, etc. from text. Traditional NER methods mainly rely on manual feature engineering, rule definition, or machine learning methods based on statistical models. However, with the development of deep learning technology, neural network-based models, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformer models based on attention mechanisms, have become the mainstream methods for NER tasks. Although deep learning models have made significant progress in NER tasks in recent years, there are still some significant shortcomings.

[0003] First, many existing NER models have limitations in capturing the global context information of the text. Generally speaking, the models focus on the local context, i.e., a small number of words around the named entity, but ignore the role of the global semantic background. For example, in long texts or complex semantic environments, relying solely on local context information may not fully capture the true meaning of the named entity.

[0004] Secondly, in the prior art, the model often ignores the relative position relationship between the named entity and other words in its context. Relative position information is crucial for accurately identifying named entities, because the boundaries and categories of named entities may be affected by the relative positions of their adjacent words. For example, in the sentence "John met Mary at the park", the relative positions of "John" and "Mary" can provide valuable clues to the model to help distinguish whether these entities exist independently or are related to each other. However, most existing NER models cannot effectively incorporate this relative position information into the entity recognition process. This limitation may cause the model to be unable to accurately judge the boundaries of named entities when faced with complex sentences, or even make recognition errors. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a text named entity recognition model and method.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] In a first aspect, the present application provides a text named entity recognition model, including an embedding layer, a multi-layer perceptron, a global context enhancement module, a word pair relative position encoding attention module, a hierarchical feature fusion module and a prediction layer;

[0008] The embedding layer is used to generate a text embedding representation based on the text data;

[0009] The multi-layer perceptron is used to convert the text embedding representation into a two-dimensional word pair matrix;

[0010] The global context enhancement module is used to generate a global context enhancement representation according to the two-dimensional word pair matrix;

[0011] The word pair relative position encoding attention module is used to generate a word pair relative position encoding attention representation according to a two-dimensional word pair matrix;

[0012] The hierarchical feature fusion module is used to generate a score matrix based on the global context enhancement representation, the word pair relative position encoding attention representation and the two-dimensional word pair matrix;

[0013] The prediction layer is used to output the named entity prediction result according to the score matrix.

[0014] In a possible implementation of the first aspect, the embedding layer includes a BERT layer, a Word2Vec layer, a bidirectional long short-term memory network layer, and a concatenation layer;

[0015] The BERT layer is used to generate a BERT embedding representation of each word based on the text data;

[0016] The Word2Vec layer is used to generate a word embedding representation of each word based on text data;

[0017] The bidirectional long short-term memory network layer is used to generate a character embedding representation of each word according to the text data;

[0018] The concatenation layer is used to add the BERT embedding representation of each word, the word embedding representation of each word, and the character embedding representation of each word to form a text embedding representation.

[0019] In a possible implementation manner of the first aspect, the global context enhancement module includes a mask matrix, a sparse convolution layer, and a normalization layer;

[0020] The mask matrix is ​​used to control the calculation area of ​​the sparse convolutional layer;

[0021] The sparse convolution layer is used to capture context at different positions of the two-dimensional word pair matrix according to the mask matrix to obtain context information between word pairs;

[0022] The normalization layer is used to globally enhance the context information between word pairs using the two-dimensional word pair matrix as a condition to obtain a global context enhanced representation.

[0023] In a possible implementation manner of the first aspect, the mask matrix is ​​specifically:

[0024]

[0025] Among them, H is the mask matrix, σ is the sigmoid function, S is the mask parameter matrix, g1 and g2 are both random Gumbel noise, and τ is the temperature parameter.

[0026] In a possible implementation manner of the first aspect, the sparse convolutional layer is specifically:

[0027] L ij =W sc ·M ij ·H ij

[0028] Among them, L ij is the context information of the i-th row and j-th column in the context information between word pairs, W sc is the convolution kernel of sparse convolution, M ij is the position of the i-th row and j-th column in the two-dimensional word pair matrix, H ij is the mask element in the i-th row and j-th column of the mask matrix.

[0029] In a possible implementation manner of the first aspect, the normalization layer is specifically:

[0030]

[0031] Among them, C ij is the global context enhancement information of the i-th row and j-th column in the global context enhancement representation, L ij is the context information of the i-th row and j-th column in the context information between word pairs, mean[M] is the mean of the two-dimensional word pair matrix M, std[M] is the standard deviation of the two-dimensional word pair matrix M, and W and b are both training parameters.

[0032] In a possible implementation of the first aspect, the word pair relative position encoding attention module includes an embedding table and an attention mechanism layer;

[0033] The embedding table is used to store the relative position codes of the word pairs assigned by the relative position differences between each word pair;

[0034] The attention mechanism layer is used to calculate the relative position difference between word pairs according to the two-dimensional word pair matrix, and query the corresponding word pair relative position encoding from the embedding table according to the relative position difference between the word pairs, and generate the word pair relative position encoding attention representation according to the word pair relative position encoding.

[0035] In a possible implementation manner of the first aspect, the attention mechanism layer is specifically:

[0036]

[0037] A ij =α ij (W l[i,j] M ij )

[0038] Among them, A ij is the relative position attention information of the word pair in the i-th row and j-th column in the word pair relative position encoding attention representation, α ij is the attention weight of the i-th row and j-th column in the two-dimensional word pair matrix, W l[i,j] ,W a ,W h[i,j] are weight training parameters, M ij is the position of the i-th row and j-th column in the two-dimensional word pair matrix, n is the total number of all candidate entities, k is the traversal number of the control candidate entity, v is the training weight vector, T is the transpose, is the relative position difference between the word pairs in the i-th row and j-th column of the two-dimensional word pair matrix.

[0039] In a possible implementation manner of the first aspect, the hierarchical feature fusion module is specifically:

[0040]

[0041] Among them, S is the score matrix, M is the two-dimensional word pair matrix, C is the global context enhancement representation, σ is the sigmoid function, V_Pool is the vertical pooling operation, H_Pool is the horizontal pooling operation, ⊙ is the Hadamard product, For serial operation, It is the outer product operation.

[0042] In a second aspect, the present application provides a text named entity recognition method, which is applied to a text named entity recognition model as described in any one of the first aspects, comprising the following steps:

[0043] Obtain text data to be recognized;

[0044] Input the text data to be recognized into the text named entity recognition model to obtain the text named entity recognition results.

[0045] The present invention has the following beneficial effects:

[0046] The present invention converts a text into an embedded one-dimensional vector representation in an embedding layer, and constructs the vector into a two-dimensional word pair matrix, which contains global word pair information and relative positions of word pairs in the text; sparse convolution and global context enhancement normalization layers are set in a global context enhancement module, wherein the sparse convolution improves the calculation efficiency by only calculating the position of non-zero weights compared with traditional CNN, saving memory and computing resources; the normalization operation uses the two-dimensional word pair matrix as a conditional input to normalize the context information captured by the sparse convolution to achieve global enhancement of the context; in the word pair relative position attention module, the relative position of the word pair is introduced into the attention mechanism, so that when calculating the attention weight, the relative position embedding of each word pair can be combined, and the relative position of the word pair is used to pay attention to the entity position in the text; thereby improving the global context understanding ability of the model through global context enhancement and using the relative position of the word pairs in the text to help the model pay attention to the position of the entity, so as to improve the accuracy of text named entity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a structural diagram of a text named entity recognition model;

[0048] Figure 2 It is a schematic diagram of the structure of the global context enhancement module;

[0049] Figure 3 Schematic diagram of the structure of the attention module for encoding the relative position of word pairs;

[0050] Figure 4 A flowchart of a text named entity recognition method. DETAILED DESCRIPTION

[0051] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0052] like Figure 1 As shown, an embodiment of the present application provides a text named entity recognition model, including an embedding layer, a multi-layer perceptron, a global context enhancement module, a word pair relative position encoding attention module, a hierarchical feature fusion module and a prediction layer;

[0053] The embedding layer is used to generate a text embedding representation h according to the text data;

[0054] The multi-layer perceptron is used to convert the text embedding representation h into a two-dimensional word pair matrix M;

[0055] The global context enhancement module is used to generate a global context enhancement representation C according to the two-dimensional word pair matrix M;

[0056] The word pair relative position encoding attention module is used to generate a word pair relative position encoding attention representation A according to a two-dimensional word pair matrix M;

[0057] The hierarchical feature fusion module is used to generate a score matrix S according to the global context enhancement representation C, the word pair relative position encoding attention representation A and the two-dimensional word pair matrix M;

[0058] The prediction layer is used to output the named entity prediction result according to the score matrix S.

[0059] In an optional embodiment of the present invention, the embedding layer includes a BERT layer, a Word2Vec layer, a bidirectional long short-term memory network layer and a concatenation layer;

[0060] The BERT layer is used to generate a BERT embedding representation of each word based on the text data;

[0061] The Word2Vec layer is used to generate a word embedding representation of each word based on text data;

[0062] The bidirectional long short-term memory network layer is used to generate a character embedding representation of each word according to the text data;

[0063] The concatenation layer is used to add the BERT embedding representation of each word, the word embedding representation of each word, and the character embedding representation of each word to form a text embedding representation.

[0064] This example uses BERT to obtain the BERT embedding representation of each word Use pre-trained word vectors (Word2Vec) to generate fixed word embeddings for each word Generate character embeddings for each word using a BiLSTM network Then embed the BERT generated above into Word Embedding Representation and character embedding representation Add and form a text embedding representation, specifically:

[0065]

[0066] in, Represents a concatenation operation.

[0067] In an optional embodiment of the present invention, the multi-layer perceptron uses a group of MLPs to take the text embedding representation h obtained by the embedding layer as input to generate a projection representation h mlp . Then, hmlp Perform an outer product with itself to obtain the two-dimensional word pair matrix M of the text, which is:

[0068] h mlp =MLP(h)

[0069]

[0070] in, Represents the outer product operation.

[0071] In an optional embodiment of the present invention, Figure 2 As shown, the global context enhancement module includes a mask matrix, a sparse convolution layer and a normalization layer;

[0072] The mask matrix is ​​used to control the calculation area of ​​the sparse convolutional layer;

[0073] The sparse convolution layer is used to capture context at different positions of the two-dimensional word pair matrix according to the mask matrix to obtain context information between word pairs;

[0074] The normalization layer is used to globally enhance the context information between word pairs using the two-dimensional word pair matrix as a condition to obtain a global context enhanced representation.

[0075] This embodiment uses the mask matrix H∈{0,1} B×1×n×n Used to control the calculation area of ​​sparse convolution, specifically:

[0076]

[0077] Among them, H is the mask matrix, σ is the sigmoid function, S is a learnable mask parameter matrix, g1 and g2 are both random Gumbel noise, and τ is the corresponding temperature parameter in Gumbel-Softmax.

[0078] This embodiment uses sparse convolution to capture context at different positions of the two-dimensional word pair matrix M obtained by the multi-layer perceptron, and obtains the context information L between the word pairs, specifically:

[0079] L ij =W sc ·M ij ·H ij

[0080] Among them, L ij is the context information of the i-th row and j-th column in the context information between word pairs, W sc is the convolution kernel of sparse convolution, M ij is the position of the i-th row and j-th column in the two-dimensional word pair matrix, H ijis the mask element in the i-th row and j-th column of the mask matrix.

[0081] In this embodiment, the two-dimensional word pair matrix M obtained by the multi-layer perceptron is used as a conditional input in the normalization operation to achieve global enhancement of the context information L, thereby obtaining a global context enhancement representation C, which is specifically:

[0082]

[0083] Among them, C ij is the global context enhancement information of the i-th row and j-th column in the global context enhancement representation, L ij is the context information of the i-th row and j-th column in the context information between word pairs, mean[M] is the mean of the two-dimensional word pair matrix M, std[M] is the standard deviation of the two-dimensional word pair matrix M, and W and b are both training parameters.

[0084] In an optional embodiment of the present invention, Figure 3 As shown, the word pair relative position encoding attention module includes an embedding table and an attention mechanism layer;

[0085] The embedding table is used to store the relative position codes of the word pairs assigned by the relative position differences between each word pair;

[0086] The attention mechanism layer is used to calculate the relative position difference between word pairs according to the two-dimensional word pair matrix, and query the corresponding word pair relative position encoding from the embedding table according to the relative position difference between the word pairs, and generate the word pair relative position encoding attention representation according to the word pair relative position encoding.

[0087] This embodiment allocates a position embedding vector (the vector is randomly initialized by normal distribution) to each relative position difference, and stores the vector in an embedding table P of size 2m+1. Here m is the predefined maximum relative position distance, and relative positions beyond the range of [-m, +m] are mapped to the same position code.

[0088] This embodiment is for relative position coding. The relative position difference Δ of each word pair in the input text is calculated through the relative position information between the word pairs provided by the word pair matrix. Then, in the embedding table P, for each position difference Δ ij , find the corresponding encoding vector Finally, the relative position encoding of each word pair in the obtained two-dimensional word pair matrix is ​​introduced into the attention mechanism to obtain the relative position encoding attention representation A of the word pair, which is specifically:

[0089]

[0090] A ij =α ij (W l[i,j] Mij )

[0091] Among them, A ij is the relative position attention information of the word pair in the i-th row and j-th column in the word pair relative position encoding attention representation, α ij is the attention weight of the i-th row and j-th column in the two-dimensional word pair matrix, W l[i,j] ,W a ,W h[i,j] are weight training parameters, M ij is the position of the i-th row and j-th column in the two-dimensional word pair matrix, n is the total number of all candidate entities, k is the traversal number that controls the candidate entity, v is a trainable weight vector, T is the transpose, is the relative position difference between the word pairs in the i-th row and j-th column of the two-dimensional word pair matrix.

[0092] In an optional embodiment of the present invention, this embodiment uses a hierarchical feature fusion module to integrate the two-dimensional word pair matrix M obtained by the multi-layer perceptron, the global context enhancement representation C obtained by the global context enhancement module, and the word pair relative position attention representation A obtained by the word pair relative position encoding attention module to obtain a score matrix S, which is specifically:

[0093]

[0094] Among them, S is the score matrix, M is the two-dimensional word pair matrix, C is the global context enhancement representation, Sig is the sigmoid function, V_Pool is the vertical pooling operation, H_Pool is the horizontal pooling operation, ⊙ is the Hadamard product, For serial operation, is the outer product operation. The score matrix S contains the scores of each word pair.

[0095] In an optional embodiment of the present invention, this embodiment uses the prediction layer to use each word pair in the score matrix S obtained by the hierarchical feature fusion module as the input of the softmax function to calculate its probability, specifically:

[0096]

[0097] in, is the probability that the word pair (i, j) belongs to the kth type.

[0098] like Figure 4 As shown, the embodiment of the present application also provides a text named entity recognition method, comprising the following steps S1 and S2:

[0099] S1, obtaining text data to be recognized;

[0100] S2. Input the text data to be recognized into the text named entity recognition model to obtain the text named entity recognition result.

[0101] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0104] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0105] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A text named entity recognition model, characterized in that: It includes embedding layer, multi-layer perceptron, global context enhancement module, word pair relative position encoding attention module, hierarchical feature fusion module and prediction layer; The embedding layer is used to generate a text embedding representation based on the text data; The multi-layer perceptron is used to convert the text embedding representation into a two-dimensional word pair matrix; The global context enhancement module is used to generate a global context enhancement representation according to the two-dimensional word pair matrix; The word pair relative position encoding attention module is used to generate a word pair relative position encoding attention representation according to a two-dimensional word pair matrix; The hierarchical feature fusion module is used to generate a score matrix based on the global context enhancement representation, the word pair relative position encoding attention representation and the two-dimensional word pair matrix; The prediction layer is used to output the named entity prediction result according to the score matrix.

2. A text named entity recognition model according to claim 1, characterized in that: The embedding layer includes a BERT layer, a Word2Vec layer, a bidirectional long short-term memory network layer, and a concatenation layer; The BERT layer is used to generate a BERT embedding representation of each word based on the text data; The Word2Vec layer is used to generate a word embedding representation of each word based on text data; The bidirectional long short-term memory network layer is used to generate a character embedding representation of each word according to the text data; The concatenation layer is used to add the BERT embedding representation of each word, the word embedding representation of each word, and the character embedding representation of each word to form a text embedding representation.

3. A text named entity recognition model according to claim 1, characterized in that: The global context enhancement module includes a mask matrix, a sparse convolutional layer and a normalization layer; The mask matrix is ​​used to control the calculation area of ​​the sparse convolutional layer; The sparse convolution layer is used to capture context at different positions of the two-dimensional word pair matrix according to the mask matrix to obtain context information between word pairs; The normalization layer is used to globally enhance the context information between word pairs using the two-dimensional word pair matrix as a condition to obtain a global context enhanced representation.

4. A text named entity recognition model according to claim 3, characterized in that: The mask matrix is ​​specifically: Among them, H is the mask matrix, σ is the sigmoid function, S is the mask parameter matrix, g1 and g2 are both random Gumbel noise, and τ is the temperature parameter.

5. A text named entity recognition model according to claim 3, characterized in that: The sparse convolutional layer is specifically: L ij =W sc ·M ij ·H ij Among them, L ij is the context information of the i-th row and j-th column in the context information between word pairs, W sc is the convolution kernel of sparse convolution, M ij is the position of the i-th row and j-th column in the two-dimensional word pair matrix, H ij is the mask element in the i-th row and j-th column of the mask matrix.

6. A text named entity recognition model according to claim 3, characterized in that: The normalization layer is specifically: Among them, C ij is the global context enhancement information of the i-th row and j-th column in the global context enhancement representation, L ij is the context information of the i-th row and j-th column in the context information between word pairs, mean[M] is the mean of the two-dimensional word pair matrix M, std[M] is the standard deviation of the two-dimensional word pair matrix M, and W and b are both training parameters.

7. A text named entity recognition model according to claim 1, characterized in that: The word pair relative position encoding attention module includes an embedding table and an attention mechanism layer; The embedding table is used to store the relative position codes of the word pairs assigned by the relative position differences between each word pair; The attention mechanism layer is used to calculate the relative position difference between word pairs according to the two-dimensional word pair matrix, and query the corresponding word pair relative position encoding from the embedding table according to the relative position difference between the word pairs, and generate the word pair relative position encoding attention representation according to the word pair relative position encoding.

8. A text named entity recognition model according to claim 7, characterized in that: The attention mechanism layer is specifically: A ij =a ij (W l[i,j] M ij ) Among them, A ij is the relative position attention information of the word pair in the i-th row and j-th column in the word pair relative position encoding attention representation, α ij is the attention weight of the i-th row and j-th column in the two-dimensional word pair matrix, W l[i,j] ,W a ,W h[i,j] are weight training parameters, M ij is the position of the i-th row and j-th column in the two-dimensional word pair matrix, n is the total number of all candidate entities, k is the traversal number of the control candidate entity, v is the training weight vector, T is the transpose, is the relative position difference between the word pairs in the i-th row and j-th column of the two-dimensional word pair matrix.

9. A text named entity recognition model according to claim 1, characterized in that: The hierarchical feature fusion module is specifically: Among them, S is the score matrix, M is the two-dimensional word pair matrix, C is the global context enhancement representation, σ is the sigmoid function, V_Pool is the vertical pooling operation, H_Pool is the horizontal pooling operation, ⊙ is the Hadamard product, For serial operation, It is the outer product operation.

10. A method for identifying named entities in text, characterized in that: A text named entity recognition model applied to any one of claims 1 to 8, comprising the following steps: Obtain text data to be recognized; Input the text data to be recognized into the text named entity recognition model to obtain the text named entity recognition results.

Images