A named entity recognition method based on label embedding and feature extraction

Abstract

The invention relates to a named entity recognition method based on label embedding and feature extraction, belonging to the technical field of named entity recognition methods; the technical problem to be solved is: to provide an improvement of the named entity recognition method based on label embedding and feature extraction; the technical scheme adopted to solve the above technical problem is: firstly, a text sequence is defined, and each word is encoded by a pre-training layer of BERT to obtain an embedded representation of each word; then, a word sequence context representation is performed on the recognized text, a Bi-LSTM model is adopted to extract local context features, a label vector is adopted to classify context feature vectors in the document when performing document-level representation, and when decoding and predicting the recognized text, a sequence feature generated by a second-layer Bi-LSTM model is linearly mapped to an emission score of each label, the emission score is combined with a state transition score learned by a conditional random field, the probability of the label sequence is calculated, and finally a log-likelihood function is adopted to calculate the loss: the invention is applied to named entity recognition.

Description

Technical Field

[0001] The invention relates to a named entity recognition method based on label embedding and feature extraction, and belongs to the technical field of named entity recognition methods. Background Art

[0002] Named Entity Recognition (NER) refers to the identification of entities with specific meanings in text, mainly including names of people, places, organizations or proper nouns. Named entity recognition is an important basic tool in application fields such as information extraction, question-answering systems, syntactic analysis, machine translation, and metadata annotation. It is the basis of some complex tasks such as relationship extraction, syntactic parsing, entity coreference resolution, and information retrieval. It occupies an important position in the process of natural language processing technology becoming practical.

[0003] At present, the NER methods based on traditional machine learning have the defects of relying on expert features and weak model generalization ability. The NER methods based on deep learning can effectively alleviate the above problems. Therefore, the current mainstream NER methods are mainly implemented based on deep learning technology; the NER model structure based on deep learning is mainly divided into three layers: embedding layer, encoding layer and decoding layer. In the embedding layer, the model needs to obtain the vector representation of the text input. Different methods are usually used to randomly initialize the vector according to a certain distribution or to use a pre-trained model to obtain a vector with semantic representation ability; the embedding layer, as the bottom layer of the model, plays a very important role in the upper-layer learning of the model. Researchers usually incorporate rich word information into the embedding layer of the model to improve the expressive ability of the model; the BERT model uses a bidirectional attention mechanism to learn the relationship between characters, words and sentences, and is usually used as an embedding layer to improve the representation ability of the model; because the sequential characteristics of the text match the inherent sequential input characteristics of the LSTM, the Bi-LSTM model is usually selected as the encoding layer. The code layer extracts context features; named entity recognition is a typical sequence labeling task, and the model needs to label the entities in the sentence according to the learned context features; the labeling strategy is usually "BIO" or "BMES". No matter which strategy is used, there is always a sequence relationship between the labels. For example, the label starting with "B" can only label the first word of the entity. At this time, the middle label starting with "I" or "M" can only appear after the "B" label, otherwise illegal labeling will occur, resulting in recognition errors; therefore, it is necessary to improve the current NER method based on deep learning to avoid the occurrence of illegal labeling sequences and improve recognition accuracy. Summary of the invention

[0004] In order to overcome the deficiencies in the prior art, the present invention aims to solve the technical problem of providing an improved method for named entity recognition based on tag embedding and feature extraction.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for named entity recognition based on label embedding and feature extraction, comprising the following steps:

[0006] Step 1: Perform word embedding representation on the recognized text:

[0007] Define the text sequence as X = {x1, x2, x3, ..., x n ), each word x i After encoding through the BERT pre-training layer, the embedding representation c of each word is obtained i ;

[0008] Step 2: Perform word sequence context representation on the recognized text:

[0009] The Bi-LSTM model is used to extract local context features. The Bi-LSTM model is composed of a combination of forward and backward LSTM. The calculation formula is:

[0010]

[0011]

[0012]

[0013] Where: θ represents the trainable parameter, then the calculation formula of unidirectional LSTM is:

[0014]

[0015] Where: i t represents the input gate, f t represents the forget gate, o t represents the output gate; W and b represent parameters; C t Represents the cell state, used to memorize long-distance information, h t is the truncated output of short-term memory;

[0016] Define the word embedding sequence as C = {c1, c2, c3, ..., c n}, each word is embedded in c i After forward and backward LSTM encoding, the context representation h of each word is obtained i ;

[0017] Step 3: Represent the recognized text at the document level: Use label vectors to classify the context feature vectors in the document so that the model can find information that is more similar to the context or helpful to the model for prediction during information fusion. The specific calculation process is as follows:

[0018] Step 3.1: Store document information: Use two sets to store word embedding vectors c i The context representation h corresponding to the set and word embedding ik Set, the above two sets establish a corresponding relationship through the serial number generated for each unique word when the model initializes the data, and store the word embedding vector and the context representation generated by the first layer of Bi-LSTM in the corresponding positions of the two sets respectively according to the word serial number;

[0019] Step 3.2: Initialize label embedding: Introduce the label vector to measure the relationship between the current word embedding vector and the word embedding vector in the storage module, and calculate the initial word embedding c i For the label vector L = {l1, l2, l3, ..., l m Initialize weights:

[0020] Step 3.2.1: Use BERT to preload the entire document training set to generate word embeddings. The calculation formula is:

[0021] C doc =BERT(Doc)#(10);

[0022] Step 3.2.2: Embed all words into C according to the label type doc Classification, the calculation formula is:

[0023]

[0024] In the formula, type(l j ) is a randomly assigned label type;

[0025] Step 3.2.3: Calculate each label vector l j The initial weight of is calculated as:

[0026]

[0027] Initialize word embedding based on BERT so that the initial word embedding contains rich semantic information. Initialize the label vector based on the word embedding vector to improve the correlation between the label vector and the word embedding vector.

[0028] Step 3.3: Compute document-level context representation:

[0029] Step 3.3.1: Use cosine similarity to measure the embedding vector c of the current word k With all labels L = {l1, l2, l3, ..., l m The correlation S between k , the calculation formula is:

[0030]

[0031] Step 3.3.2: According to the correlation S k Find the current word embedding vector c k The target label vector is calculated as:

[0032]

[0033] Step 3.3.3: Calculate the target label vector l based on formula (14) k Word embedding representation in the storage module The correlation score between Select the word embedding vector that is highly correlated with the target label vector, and then find the corresponding context representation based on the ordinal correspondence between the two sets.

[0034] Then two methods are used to select the context representation with higher relevance:

[0035] Or use a fixed number γ, then sort in descending order according to the relevance score, select the first γ vectors, and when the total number is less than γ, add a 0 vector;

[0036] Or use a fixed correlation score δ and select vectors with correlation scores greater than δ;

[0037] Step 3.3.4: Represent the selected context The corresponding relevance score is h kj The context representation of the current word h k The contribution of h k The document-level representation calculation formula is:

[0038]

[0039] Step 3.3.5: Where It is a combination of the current context and the calculated context information. At this time, the Bi-LSTM model is used again to extract features and input them into the decoding layer to enhance the association between the two sets of information.

[0040] Step 4: Decode and predict the recognized text. The decoder of the named entity recognition task uses a conditional random field. First, the sequence feature H′={h′1, h′2, ..., h′ generated by the second-layer Bi-LSTM model is n}After linear mapping, it is the emission score of each label, and the calculation formula is:

[0041] E=W e H′+b e #(20);

[0042] Where W e is the weight matrix, be is the bias matrix;

[0043] The emission score is then combined with the state transition score learned by the conditional random field to calculate the probability of the label sequence:

[0044]

[0045] In the formula, y′ represents the possible label sequence of the input sequence, Indicates o i-1 The emission fraction of the tag, Represents the last tag o i-1 Move to next tab i The state transition score of

[0046] Finally, the loss is calculated using the log-likelihood function:

[0047]

[0048] Based on the above loss data, the recognition text context is corrected to complete the recognition.

[0049] The beneficial effects of the present invention compared with the prior art are as follows: the algorithm model of document-level Chinese named entity recognition based on label embedding proposed by the present invention can effectively avoid the occurrence of illegal annotation sequences by learning the transfer rules between labels. In the named entity recognition task, Bi-LSTM can efficiently extract local information or sentence-level information according to the input sequence text, but the sequence characteristics of Bi-LSTM and the inability to simultaneously recognize multiple sentences make it unable to obtain global information; in view of this problem, the present invention proposes to assist Bi-LSTM in perceiving global information by extracting document-level global features, extracting the work content most relevant to the current word to be predicted in the document, classifying the document-level information more carefully according to the label vector obtained by model learning, and then searching the classified document information for the information most relevant to the current context feature, thereby reducing the noise during information fusion; the present invention can solve the problem of being limited by the annotation accuracy when classifying document information using the actual annotation type of the text through label embedding vectors, and can extract the high relevance information of the current word from the document information through label embedding vectors, and shows competitive F1 scores on three Chinese data sets MSRA, Resume and Weibo. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention will be further described below in conjunction with the accompanying drawings:

[0051] Figure 1 The figure is a flowchart of the steps of the named entity recognition method of the present invention. DETAILED DESCRIPTION

[0052] Named entity recognition refers to marking the type of entity in a piece of unstructured text, and is the basis for tasks such as relationship extraction and sentiment analysis. The named entity recognition method based on document-level information of the present invention is: integrating the information of the same word in different contexts into the current word information to be predicted, effectively improving the model's prediction ability for the same word in different sentences; however, different contexts may lead to completely different context information for the same word, so simply fusing the information in the document may introduce certain noise; to address this problem, the present invention proposes to use label embedding vectors to classify the information in the document, and select the information most relevant to the current context based on the correlation between the label embedding vector and the word embedding vector; the proposed method incorporates the model with high-correlation document-level information calculated by label embedding vectors, which has better recognition effect than the existing model.

[0053] like Figure 1 As shown, in the field of named entity recognition, the present invention utilizes information in the document to assist the model in predicting entities. In the process of model predicting entities, the context information of the current word plays a leading role, and other context information corresponding to the word in the document appears more in an auxiliary form. Therefore, how to find the information most relevant to the current word is crucial. To address this problem, the present invention extracts the information most relevant to the current word from the document in two steps.

[0054] First, the information in the document is classified using the label type, and this method prevents information between types from referencing each other. The recognition method proposed in the present invention does not use the actual annotation type of each word in the document to classify the word information, because this method is limited by the accuracy of the corpus annotation. If the wrong annotation is used to guide the model learning, it will inevitably lead to deviations in the model learning. Therefore, the present invention proposes to use label embedding vectors to classify the information in the document; specifically: by calculating the cosine similarity between the label embedding vector and the word embedding vector, find the target label vector corresponding to the current word. At this time, the target label vector already has type information, and then use the target label vector to classify the information in the document.

[0055] Secondly, the information in the document that is highly correlated with the target label vector is used to calculate the document-level features of the current word. This step separates the information that belongs to the same label category but has greatly different context information.

[0056] Based on the above steps, the most relevant information about the current word can be extracted from the document.

[0057] Specifically, the present invention provides a method for identifying cultural relics named entities based on label embedding to extract document-level information, which specifically includes the following steps:

[0058] Step 1: Perform word embedding representation; first, given a sequence X = {x1, x2, x3, ..., x n}, each word x i After encoding through the BERT pre-training layer, the embedding representation c of each word is obtained i , word embedding c i It contains rich semantic information, which helps the upper-level model extract more effective features.

[0059] Step 2: Perform word sequence context representation; Bi-LSTM is widely used in named entity recognition tasks because it can efficiently extract local context features. The model is composed of a combination of forward and backward LSTM. The specific calculation formula is as follows:

[0060]

[0061]

[0062]

[0063] Where θ represents a trainable parameter, and the calculation formula of the unidirectional LSTM is as follows:

[0064] i t =sigmoid(W i ·[h t-1 , x t ]+b i )#(4)

[0065] f t =sigmoid(W f ·[h t-1 , x t ]+b f )#(5)

[0066] o t =sigmoid(W o ·[h t-1 , x t ]+b o )#(6)

[0067]

[0068]

[0069] h t =o t *tanh(C t )#(9)

[0070] Where i t , f t , and ot Respectively represent input gate, forget gate and output gate; W and b represent parameters; C t represents the cell state, which is used to memorize long-distance information, and h t This produces a truncated output, called short-term memory.

[0071] For a word embedding sequence C = {c1, c2, c3, ..., c n}, each word is embedded in c i After forward and backward LSTM encoding, the context representation h of each word is obtained i .

[0072] Step 3: Perform document-level representation; each unique word will appear repeatedly in the document, but the corresponding context is not the same, so each unique word will produce a different context feature representation. If all the context representations generated by a word in the document are fused together, due to the asymmetric context information of the same word in different sentences, it is likely to introduce certain noise. Based on this, the present invention uses label vectors to classify the context feature vectors in the document, so that when the information is fused, the model can find information with more similar context or helpful to the model for prediction. The specific calculation steps are as follows:

[0073] (I) Storage document information:

[0074] The present invention uses two sets to store word embedding vectors c respectively i The context representation h corresponding to the word embedding ik The two sets establish a corresponding relationship through the serial number generated for each unique word when the model initializes the data, and the word embedding vector and the context representation generated by the first layer of Bi-LSTM are stored in the corresponding positions of the two sets respectively according to the word serial number; if all the information in the document is stored, it may lead to problems of excessive memory occupancy and excessive calculation time. Therefore, the present invention only stores the latest 500 word information for each word.

[0075] (II) Initialize tag embedding:

[0076] The label vector introduced in the present invention is used to measure the relationship between the current word embedding vector and the word embedding vector in the storage module. Both of these information have a certain relationship with the initial word embedding vector, so it is necessary to consider using the initial word embedding vector c i For the label vector L = {l1, l2, l3, ..., l m Initialize weights:

[0077] (1) Use BERT to preload the entire document (training set) to generate word embeddings:

[0078] C doc=BERT(Doc)#(10)

[0079] (2) Embed all words into C according to the tag type doc Classification:

[0080]

[0081] Where type(l j ) is a randomly assigned label type;

[0082] (3) Calculate each label vector l j The initial weight of:

[0083]

[0084] By initializing word embedding through BERT, the word embedding in the initial state contains rich semantic information, and the label vector is initialized based on the word embedding vector, which strengthens the correlation between the label vector and the word embedding vector.

[0085] (III) Computing document-level context representation:

[0086] (1) Use cosine similarity to measure the embedding vector c of the current word k With all labels L = {l1, l2, l3, ..., l m The correlation S between k :

[0087] S k =[s k1 ,s k2 ,s k3 , ..., s km ]#(13)

[0088]

[0089] (2) According to S k Find the current word embedding vector c k The target label vector is:

[0090] l k =L[loc k ]#(15)

[0091] loc k = argmax(S k )#(16)

[0092] (3) Use formula (14) to calculate the target label vector l k Word embedding representation in the storage module The correlation score between Select the word embedding vector that is highly correlated with the target label vector, and then find the corresponding context representation based on the ordinal correspondence between the two sets. Here, two methods are used to select context representations with higher relevance:

[0093] a. Fix the number γ, then sort in descending order according to the relevance score, select the first γ vectors, and when the total number is less than γ, add a 0 vector;

[0094] b. Fix the correlation score δ and select vectors with correlation scores greater than δ.

[0095] (4) The selected context is represented as The corresponding relevance score is h kj The context representation of the current word h k The contribution of h k The document-level representation of is calculated as follows:

[0096]

[0097]

[0098]

[0099] (5) It is a combination of the current context and the context information generated by calculation. In order to enhance the association between the two sets of information, the present invention again uses the Bi-LSTM model to extract features and inputs them into the decoding layer.

[0100] Step 4: Decode and predict; In the named entity recognition task, there is a certain correlation between the predicted labels. For example, the order of the labels "B-LOC", "M-LOC", "E-LOC" and "S-LOC" is relatively fixed. The label "M-" cannot be used as the first label of an entity, and "M-" and "E-" cannot appear after "S-". These rules can be used to calculate the transition probability of conditional random fields to learn the relationship between labels, thereby effectively avoiding the appearance of illegal label sequences; Based on the above reasons, conditional random fields have become a classic decoder for named entity recognition tasks.

[0101] The present invention converts the sequence feature H′={h′1, h′2, ..., h′ generated by the second layer Bi-LSTM model n}After linear mapping, the emission score for each label is:

[0102] E=W e H′+b e #(20)

[0103] Among them, W e is the weight matrix, be is the bias matrix.

[0104] The emission score is then combined with the state transition score learned by the conditional random field to calculate the probability of the label sequence:

[0105]

[0106] Among them, y′ represents the possible label sequence of the input sequence, Indicates o i-1 The emission fraction of the tag, Represents the last tag o i-1 Move to next tab i The state transition score of

[0107] Finally, the loss is calculated using the log-likelihood function:

[0108]

[0109] Furthermore, the above named entity recognition method is used to test the specific text to be detected. The experimental process is as follows:

[0110] 1. Preprocessing of the data set: The present invention can use the public data sets of MSRA, Resume and Weibo to verify the model, and process the raw data into executable data with a unified format.

[0111] 2. Experimental settings: The model is optimized using the stochastic gradient descent (SGD) algorithm. On the MSRA dataset, the batch size is set to 10, and the BERT embedding is fine-tuned with an initial learning rate of 8e-4. On the Resume and Weibo datasets, the batch size is set to 16 and 10, respectively, a fixed BERT embedding is used, and the model is trained with an initial learning rate of 0.01. All learning rates decay at a rate of 5%. The hidden layer dimension of the Bi-LSTM is set to 128. In order to avoid overfitting of the model, the dropout mechanism is used for each layer of features and word embedding representation. All experimental parameters are obtained through a large number of experimental comparisons.

[0112] 3. Evaluation: All experiments were run on an NVIDIA Tesla K80 with 11G memory, and the models were evaluated by F1 score. The BIOES annotation strategy was used in both model training and prediction.

[0113] like Figure 1 As shown, input the data into the model and use the prepared data to train the model, as follows:

[0114] 1) Use the BERT model to obtain the embedding representation of each word. At the same time, use the BERT model to load the training set to obtain the embedding representation of all words, and then initialize the label embedding according to the word embedding corresponding to each type.

[0115] 2) Input the embedding representation of each word into the Bi-LSTM encoding to obtain the contextual features.

[0116] 3) The information of each word is stored in the form of key-value, where the key is the word embedding corresponding to each word and the value is the contextual feature generated by the word in different contexts.

[0117] 4) Query the target label vector based on the embedding vector of the current word.

[0118] 5) According to the target label vector, the memory information is searched for information with higher relevance, and connected with the current context information to input the next layer of Bi-LSTM to extract new context features.

[0119] 6) Use CRF to decode, predict the accuracy of entity recognition, and calculate the F1 score.

[0120] According to the above steps, the model parameters are continuously updated and iterated, and the model performance is evaluated by taking the average value of 5 experiments on each data set. The comparative experiments show that the proposed model exhibits competitive model performance.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A named entity recognition method based on label embedding and feature extraction, characterized in that: The steps include: Step 1: Perform word embedding representation on the recognized text: Define the text sequence as X = {x1, x2, x3, ..., x n }, each word x i After encoding through the BERT pre-training layer, the embedding representation c of each word is obtained i ; Step 2: Perform word sequence context representation on the recognized text: The Bi-LSTM model is used to extract local context features. The Bi-LSTM model is composed of a combination of forward and backward LSTM. The calculation formula is: Where: θ represents the trainable parameter, then the calculation formula of unidirectional LSTM is: Where: i t represents the input gate, f t represents the forget gate, o t represents the output gate; W and b represent parameters; C t Represents the cell state, used to memorize long-distance information, h t is the truncated output of short-term memory; Define the word embedding sequence as C = {c1, c2, c3, ..., c n }, each word is embedded in c i After forward and backward LSTM encoding, the context representation h of each word is obtained i ; Step 3: Represent the recognized text at the document level: Use label vectors to classify the context feature vectors in the document so that the model can find information that is more similar to the context or helpful to the model for prediction during information fusion. The specific calculation process is as follows: Step 3.1: Store document information: Use two sets to store word embeddings c separately i The context representation h corresponding to the word embedding ik , c i and h ik The corresponding relationship is established by the serial number generated for each unique word when the model initializes the data, and the word embedding vector and the context representation generated by the first layer of Bi-LSTM are stored in c according to the word serial number. i and h ik The corresponding position of Step 3.2: Initialize label embedding: Introduce the label vector to measure the relationship between the current word embedding and the word embedding in the storage module, using the initial word embedding c i For the label vector L = {l1, l2, l3, ..., l m Initialize weights: Step 3.2.1: Use BERT to preload the entire document training set to generate word embeddings. The calculation formula is: C doc =BERT(Doc)#(10); Step 3.2.2: Embed all words into C according to the label type doc Classification, the calculation formula is: In the formula, type(l j ) is a randomly assigned label type; Step 3.2.3: Calculate each label vector l j The initial weight of is calculated as: Initialize word embedding based on BERT so that the initial word embedding contains rich semantic information. Initialize label vector based on word embedding vector to improve the correlation between label vector and word embedding vector. Step 3.3: Compute document-level context representation: Step 3.3.1: Use cosine similarity to measure the embedding vector c of the current word k With all labels L = {l1, l2, l3, ..., l m The correlation S between k , the calculation formula is: Step 3.3.2: According to the correlation S k Find the current word embedding vector c k The target label vector is calculated as: Step 3.3.3: Calculate the target label vector l based on formula (14) k Word embedding representation in the storage module The correlation score between Select the word embedding vector that is highly correlated with the target label vector, and then find the corresponding context representation based on the ordinal correspondence between the two sets. Then two methods are used to select the context representation with higher relevance: Or use a fixed number γ, then sort in descending order according to the relevance score, select the first γ vectors, and when the total number is less than γ, add a 0 vector; Or use a fixed correlation score δ and select vectors with correlation scores greater than δ; Step 3.3.4: Represent the selected context The corresponding relevance score is h ki The context representation of the current word h k The contribution of h k The document-level representation calculation formula is: Step 3.3.5: Where It is a combination of the current context and the calculated context information. At this time, the Bi-LSTM model is used again to extract features and input them into the decoding layer to enhance the association between the two sets of information. Step 4: Decode and predict the recognized text. The decoder of the named entity recognition task uses a conditional random field. First, the sequence feature H′={h′1, h′2, ..., h′ n }After linear mapping, it is the emission score of each label, and the calculation formula is: E=W e H′+b e #(20); Where W e is the weight matrix, b e is the bias matrix; The emission score is then combined with the state transition score learned by the conditional random field to calculate the probability of the label sequence: In the formula, y' represents the possible label sequence of the input sequence, E oi-1 Indicates o i-1 The emission fraction of the tag, T (oi-1,oi) Represents the last tag o i-1 Move to next tab i The state transition score of Finally, the loss is calculated using the log-likelihood function: Based on the above loss data, the recognition text context is corrected to complete the recognition.

Images