Chinese named entity recognition method based on pre-training model

Abstract

The invention discloses a Chinese named entity recognition method based on a pre-training model, which is used for a knowledge graph semi-automatic construction task. The method is based on a pre-training model BERT, firstly, a Trie structure is constructed for an input text character token to obtain a [character-word] pair sequence, and a marked training set is input into the BERT for encoding; a Lexcon Adapter structure is introduced into a Transform layer in the BERT, and a candidate vocabulary vector and a hidden state in the BERT are fused; and inputting the fused text features into an eLSTM network to mine a context relationship, and inputting the context relationship into a CRF layer to realize entity label output. According to the method, a word information feature fusion calculation method is improved, interactive attention is adopted to capture more nonlinear information, an eLSTM network is introduced to capture context features, and the performance of a named entity recognition model can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing knowledge graph construction in artificial intelligence, and specifically relates to a Chinese named entity recognition method based on a pre-training model. Background Art

[0002] In view of the wide distribution of online marketing channels and the uncontrollable compliance of business channels in the power business, it is necessary to sort out and study national, local and personal data security laws, regulations and policies, grasp the compliance needs of open business data in all channels of online marketing cooperation, and semi-automatically construct a safety regulation knowledge graph based on knowledge reasoning of relevant regulations and expert knowledge.

[0003] Among them, named entity recognition is the most critical task in the information extraction link of semi-automatic construction of knowledge graphs. From the early dictionary-based and rule-based methods to the later traditional machine learning algorithms such as HMM and SVM, they all have shortcomings such as low recognition accuracy and limited application scope. The research on named entity recognition methods based on deep learning has become a hot research topic in this field.

[0004] Research based on large language models is now the forefront of natural language processing technology. Models such as GPT and BERT have learned rich language representations through pre-training on large-scale text data, enabling them to perform well on various language tasks. For named entity recognition, there are currently basic BERT models and larger-scale RoBERTa models, as well as BERT-wwm and RoBERTa-wwm for full-word masking tasks, and model variants such as ALBERT have also been born. These pre-trained language models have been widely used in named entity recognition tasks and have achieved good results.

[0005] However, these existing methods have the following problems: 1) There is a lack of further exploration and utilization of text features, and the local vocabulary information and grammatical features of the Chinese named entity recognition task are not effectively utilized; 2) Some variants of large language models load too many parameters in actual model training, which requires huge computing resources; 3) There is no further modeling of global contextual relationships. Summary of the invention

[0006] Purpose of the invention: The present invention proposes a Chinese named entity recognition method based on a pre-training model.

[0007] Technical solution: The Chinese named entity recognition method based on the pre-training model proposed in the present invention comprises the following steps: (1) Build a Trie dictionary tree structure based on the dictionary, then traverse all character subsequences in the text data sentence, match the dictionary to obtain all potential words, and then assign words to each character it contains to obtain a [character-word] pair sequence; (2) Input the training set into the BERT pre-trained model, perform [character-word] matching on each character token in the input text based on the Trie dictionary tree structure obtained in step (1), and then connect it to the inside of BERT through the word information matcher LexiconAdapter to perform [character-word] character hidden state and vocabulary feature fusion, and then input it into the subsequent Transformer layer; (3) After the operation in step (2), the text embedding output by BERT is input into the eLSTM layer to further model and capture context information, and then input into the CRF layer to implement the entity sequence labeling task and decoding process, thus completing the construction of a Chinese named entity recognition model based on a pre-trained model that integrates vocabulary information. (4) Train the Chinese entity recognition model based on the training model on the dataset. For constructing a dictionary tree through a dictionary to perform a self-mapping function, thereby forming a [character-word] pair sequence, the step (1) should include the following steps: (11) Given a Chinese dictionary D, construct a Trie dictionary tree, traverse the nodes of the tree, and compare it with a Chinese sentence s with a character length of n. c ={c1, c2, ..., c n} to match and find the words that contain these characters in the sequence; (12) Then, these matching words are assigned to the characters they contain, each character is paired with the specified word, and the Chinese sentence s c Convert to a sequence of [character-word] pairs, s cw ={(c1,ws1), (c2,ws2),..., (c n , ws n )},c i Represents the i-th character, ws i Represents the word matched by the i-th character. After the dictionary tree is constructed, the text is input into the BERT model to obtain the hidden state vector of the text, which is then fused with the candidate words through the word information matcher Lexicon Adapter. The step (2) should include the following steps: (21) After the sentences in the training set are input into the BERT pre-trained model, the hidden state vector corresponding to each character token is obtained at the output of the first layer Transformer of BERT. The corresponding [character-word] pair sequence is expressed as Lexicon Adapter accepts input The jth word in is represented as follows: Among them, e w is the pre-trained word embedding lookup table, w ij is ws in step (22) i The jth word of ; (22) By setting a nonlinear transformation, The word vector dimension and character hidden state vector Align the dimensions of the word vectors to obtain the aligned word vectors There are two sets of linear layers W1∈(d c ×d w ), W2∈(d c ×d c ), b1 and b2 are the biases of the corresponding linear layers, d w is the dimension size of the training word vector, d c is the dimension size of the character hidden state inside BERT; (23) For each character, set the matching number of m candidate words, then there is a candidate word vector sequence Aiming at the original bilinear transformation attention calculation strategy, a method for interactive attention scores of two sets of vectors of the input word information matcher Lexicon Adapter is designed: K=X i .W k V=X i .W v a i =softmax(QK T ) Where W q ∈(d c ×d c )),W k ∈(d c ×d c )),W v ∈(d c ×d c )) is a learnable attention weight matrix, a i represents the attention scores of the m candidate words matched by the i-th input character, so the weighted sum of the m candidate words can be expressed as: The character hidden state and the candidate word vector weight value are fused through residual connection: Finally, it is processed through a dropout layer and a normalization layer, and then input into the subsequent Transformer layer inside BERT: G=LN(H l-1 +MHAttn(H l-1 )) H l =LN(G+FFN(G)) H l-1 represents the hidden state of the output of the i-1th layer, LN represents layer normalization, MHAtn represents the multi-head attention mechanism, G is the vector of the normalized output of the multi-head attention module, and FFN represents the feedforward neural network. After integrating vocabulary information into the BERT internal network through step (2), the output text embedding needs to be further mined through a deep neural network for contextual features, and then entity annotation and decoding are performed. The step (3) should include the following steps: (31) Obtain the text hidden state of BERT output after vocabulary feature fusion represents the contextual semantic feature representation of the character ci in the Chinese sentence input in step (21). Input into a bidirectional eLSTM network, the unit structure of eLSTM is: c t =f t c t-1 +i t z t n t =f t n t-1 +i t c t Represents the update of cell state, n t represents the normalized cell state, h t represents the hidden state, z t represents cell input, i t , f t , o t They represent the input gate, forget gate and output gate respectively. represents the calculation of the pre-activation value of each item, w and r represent the weight matrix, x tRepresents the input of the current time step. Capture the hidden semantic feature information of the sentence in both directions: that is, get the hidden state of the forward eLSTM unit output The output of the backward eLSTM unit is expressed as Then the final hidden state is obtained by splicing (32) By using the CRF layer to model the dependencies between tag sequences, given the last hidden layer of the eLSTM network output H = {h1,h2,…,h n}, first calculate the emission fraction P: P=W0H+b0 W0 represents the learnable weight matrix, b0 represents the bias; For a label sequence y = {y1, y2, ..., y n}, for sentence s={s1,s2,...,s n}, its probability can be defined as: T represents the transfer score matrix, represents all possible tag sequences; Given N labeled data The model is trained by minimizing the sentence-level negative log-likelihood loss: During the decoding process, the Viterbi algorithm is used to find the tag sequence with the highest score. Finally, in step (4), the model is trained on multiple public datasets. Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: (1) By designing a new method for calculating attention scores for two groups of different sequences, the character features and their corresponding candidate word information are fused using the word information matcher Lexicon Adapter to fully utilize the word boundary features in the text; (2) The basic BERT Chinese pre-training model is selected, and the number of loaded parameters is far less than that of models with huge parameter amounts such as RoBERTa and BERT-wwm, which saves computing power resources and has small effect fluctuations; (3) The text context features are further modeled through the eLSTM layer, thereby improving the global generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 It is a flow chart of the present invention; Figure 2 This is a schematic diagram of the structure of the word information matcher Lexicon Adapter; DETAILED DESCRIPTION The present invention will now be described in further detail with reference to the accompanying drawings and tables. It should be noted that the terms such as "upper", "lower", "left", "right", "front", "back", etc. cited in the invention are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments in their relative relationships should also be regarded as the scope of implementation of the present invention without substantially changing the technical content. The present invention relates to a Chinese named entity recognition method based on a pre-trained model. The overall operation process is as follows: Figure 1 The specific steps are as follows: 1. Build a Trie dictionary tree structure based on the dictionary, then traverse all character subsequences in the text data sentence, match the dictionary to get all potential words, and then assign the word to each character it contains to get a [character-word] pair sequence. The specific steps are as follows: 1) Given a Chinese dictionary D, construct a Trie dictionary tree, traverse the nodes of the tree, and compare it with a Chinese sentence s with a character length of n c ={c1,c2,…,c n} to match and find the words that contain these characters in the sequence, that is, candidate words. For the dictionary tree, each node has two parameters: one is the vocabulary flag, the value of this flag is a Boolean value, which is used to mark whether the character of the node is the last word of the vocabulary. If the flag is False, it means that the flag is a single word or a word in the middle of the vocabulary; if the flag is True, it means that the character of the flag and all its parent nodes can form a vocabulary. The second is the child node sequence, which uses the hash method, that is, one character corresponds to one tree node. ; 2) Then, assign these matching words to the characters they contain, pair each character with the specified word, and convert the Chinese sentence into a sentence. c Convert to a sequence of [character-word] pairs, s cw ={(c1,ws1),(c2,ws2),...,(c n ,ws n )},c i Represents the i-th character, ws iRepresents the words matched by the i-th character. The specific matching process of the dictionary tree is as follows: First, given the address string and the maximum depth d of the dictionary tree (that is, the length of the longest word in the dictionary tree), then, with d as the sliding window size of the address string, the string of length d is intercepted in order. Next, this string of length d is matched word by word using the constructed dictionary tree. If a child node is marked as True during the matching process, the vocabulary formed by the child node and all its parent nodes is added to the vocabulary matching table. When the sliding window is at the end of the sentence, the sliding window size needs to be continuously reduced from left to right to ensure that the last few characters of the address string can also be matched. Finally, the matched vocabulary matching table is returned. 2. Input the training set into the BERT pre-training model, match each character token in the input text with a [character-word] pair based on the Trie dictionary tree structure, and then connect the word information matcher Lexicon Adapter to the BERT internal [character-word] to fuse the character hidden state and vocabulary features. The structural diagram of Lexicon Adapter is shown below: Figure 2 As shown in Figure 1 As shown in the flowchart, the operations in step 2 are all performed after the first layer Transformer of BERT. The specific steps are as follows: 1) After the sentences in the training set are input into the BERT pre-trained model, the hidden state vector corresponding to each character token is outputted in the first layer of BERT Transformer. The corresponding [character-word] pair sequence is expressed as Lexicon Adapter accepts input The jth word in is represented as follows: Among them, e w is a pre-trained word embedding lookup table. The embedding vector used in this invention comes from the word embedding vector set of Tencent AI Laboratory, which contains 2 million pre-trained word vectors with a vector dimension size of 200. ij is ws in step 2) i The jth word of ; 2) By setting nonlinear transformation, The word vector dimension and character hidden state vector Align the dimensions of the word vectors to obtain the aligned word vectors There are two sets of linear layers W1∈(d c ×dw ), W2∈(d c ×d c ), b1 and b2 are the biases of the corresponding linear layers, d w is the dimension size of the training word vector, d c is the dimension size of the hidden state of the character inside BERT, in the parameter table of the BERT model, d c Set to 768; 3) For each character, set m candidate words to match. In the present invention, m is set to 3, that is, each character is assigned three candidate words. Vector sequence Aiming at the original bilinear transformation attention calculation strategy, a method for interactive attention scores of two sets of vectors input into Lexicon Adapter is designed: K=X i .W k V=X i .W v a i =softmax(QK T ) Where W q ∈(d c ×d c )),W k ∈(d c ×d c )),W v ∈(d c ×d c )) is a learnable attention weight matrix, a i represents the attention scores of the m candidate words matched by the i-th input character, so the weighted sum of the m candidate words can be expressed as: exist Figure 2 As shown in the additive layer normalization module, the character hidden state and the candidate word vector weight value are fused in this way: Finally, it is processed through a dropout layer and a normalization layer, and then input into the subsequent Transformer layer inside BERT: G=LN(H l-1 +MHAttn(H l-1 )) H l =LN(G+FFN(G)) H l-1represents the hidden state of the output of the i-1th layer, where H 0 represents the initial embedding E, LN represents layer normalization, MHAtn represents the multi-head attention mechanism, G is the vector of the normalized output of the multi-head attention module, and FFN represents the feedforward neural network. 3. The text embedding output by BERT is input into the eLSTM layer to further capture contextual information, and then input into the CRF layer to implement the entity sequence labeling task and decoding process, and build a Chinese named entity recognition model based on the pre-trained model that integrates vocabulary information. The specific steps of eLSTM to capture contextual features are as follows: 1) Obtain the text hidden state of BERT output after vocabulary feature fusion Represents the character c in the Chinese sentence entered in step 1 i The contextual semantic feature representation of Input into a bidirectional eLSTM network, the vector dimension size of eLSTM is set to 128. The unit structure of the eLSTM network is: c t =f t c t-1 +i t z t n t =f t n t-1 +i t c t Represents the update of cell state, n t represents the normalized cell state, h t represents the hidden state, z t represents cell input, i t , f t , o t They represent the input gate, forget gate and output gate respectively. represents the calculation of the pre-activation value of each item, w and r represent the weight matrix, x t Represents the input of the current time step. Capture the hidden semantic feature information of the sentence in both directions: that is, get the hidden state of the forward eLSTM unit output The output of the backward eLSTM unit is expressed as Then the final hidden state is obtained by splicing 2) The decoding process of the CRF network is as follows: by using the CRF layer to model the dependencies between tag sequences, given the last hidden state of the network output H = {h1, h2, ..., h n}, first calculate the emission fraction P: O=W0H+b0 W0 represents the learnable weight matrix, b0 represents the bias; For a label sequence y = {y1, y2, ..., y n}, for sentence s={s1,s2,…,s n}, its probability can be defined as: T represents the transfer score matrix, represents all possible tag sequences; Given N labeled data The model is trained by minimizing the sentence-level negative log-likelihood loss: During the decoding process, the Viterbi algorithm is used to find the tag sequence with the highest score. 4. Metrics Here, the commonly used metric for classification models, namely the F1 value, is used to compare the recognition accuracy of other models and the present invention. The relevant parameters in the calculation of the F1 value are as follows: TP, namely True Positives, indicates that the sample is classified as a positive sample and the allocation is correct; TN, namely True Negatives, indicates that the sample is classified as a sample and the allocation is correct; FP, namely False Positives, indicates that the sample is classified as a positive sample but the allocation is wrong; FN, namely False Negatives, indicates that the sample is classified as a negative sample but the allocation is wrong. Precision, or accuracy, represents the ratio of the number of correctly assigned positive samples to the total number of assigned positive samples. The formula is as follows: Recall, or recall rate, indicates the ratio of correctly assigned positive samples to the total number of positive samples. The formula is as follows: F1-Score is the weighted harmonic mean of Precision and Recall. The closer the F1 score is to 1, the higher the comprehensive quality of the model in terms of recall and precision. However, if the Precision and Recall values ​​differ greatly, the F1 score will deteriorate. The calculation formula is as follows: 5. The named entity recognition method based on the pre-training model described in the present invention designs a word information matcher Lexicon Adapter to calculate the relevance attention score of the hidden state of the internal characters of BERT and the candidate related words corresponding to the characters. The improved long short-term memory network eLSTM network is further introduced to improve the model's ability to capture contextual features. The above improvements can improve the performance of the model in the named entity recognition task. The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and is characterized in that when the computer program is loaded into the processor, the small-scale pedestrian detection method based on feature fusion is implemented. The computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the small-scale pedestrian detection method based on feature fusion is implemented. The computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. The processor is used to execute the computer program stored in the memory to implement each step of the method involved in the above embodiment.

Claims

1. A Chinese named entity recognition method based on a pre-trained model, characterized in that: The following steps are involved: (1) Build a Trie structure based on the dictionary, then traverse all character subsequences in the text data sentence, match the dictionary to obtain all potential words, and then assign words to each character it contains to obtain a [character-word] pair sequence; (2) Input the training set into the BERT pre-trained model, perform [character-word] matching on each character token in the input text based on the Trie dictionary tree structure obtained in step (1), and then connect it to the inside of BERT through the word information matcher LexiconAdapter to perform [character-word] character hidden state and vocabulary feature fusion, and then input it into the subsequent Transformer layer; (3) After the operation in step (2), the text embedding output by BERT is input into the eLSTM layer to further model and capture context information, and then input into the CRF layer to implement the entity sequence labeling task and decoding process, thus completing the construction of a Chinese named entity recognition model based on a pre-trained model that integrates vocabulary information; (4) Train the Chinese entity recognition model based on the training model on the dataset.

2. The Chinese named entity recognition method based on the pre-training model according to claim 1 is characterized in that: The step (1) should include the following steps: (21) Given a Chinese dictionary, construct a dictionary tree, traverse the nodes of the tree, and compare it with a Chinese sentence s with a character length of n c ={c1,c2,…,c n } to match and find the words that contain these characters in the sequence; (22) Then, these matching words are assigned to the characters they contain, each character is paired with the specified word, and the Chinese sentence s c Convert to a sequence of [character-word] pairs, s cw ={(c1,ws1), (c2,ws2),..., (c n ,ws n )},c i Represents the i-th character, ws i Represents the word matched by the i-th character.

3. The Chinese named entity recognition method based on the pre-training model according to claim 1 is characterized in that: The step (2) should include the following steps: (31) After the sentences in the training set are input into the BERT pre-trained model, the hidden state vector corresponding to each character token is obtained at the output of the first layer Transformer of BERT. The corresponding [character-word] pair sequence is expressed as Lexicon Adapter accepts input Lookup table e through word embedding w Query the corresponding embedding vector; (32) By setting a nonlinear transformation, The word vector dimension and character hidden state vector The dimensions of align; (33) For each character, set the matching number of m candidate words, then there is a candidate word vector sequence Aiming at the original bilinear transformation attention calculation strategy, a method for calculating the interactive attention scores of two sets of vectors was redesigned in the word information matcher Lexicon Adapter, and then the character hidden state and the weighted value of the candidate word vector were fused through the residual connection. Finally, it was processed through a dropout layer and a normalization layer, and then input into the subsequent Transformer layer inside BERT.

4. The Chinese named entity recognition method based on the pre-training model according to claim 1 is characterized in that: After integrating vocabulary information into the BERT internal network through step (2), the output text embedding needs to be further mined through a deep neural network for contextual features, and then entity annotation and decoding are performed. The step (3) should include the following steps: (41) Obtain the text embedding vector output by BERT after vocabulary feature fusion Will Input into the eLSTM network to capture the bidirectional hidden semantic feature information of the sentence: that is, the hidden state of the forward eLSTM unit output is obtained The output of the backward eLSTM unit is expressed as Then the final hidden state is obtained by splicing Represents the contextual semantic feature representation of the character ci in the input Chinese sentence in step (21); (42) By using the CRF layer to model the dependencies between tag sequences, the last hidden layer of the given network output is H = {h1,h2,…,h n }, first calculate the emission score P, based on a tag sequence y = {y1, y2, ..., y n }, for s={s1,s2,…,s n }Calculate its label probability, given N labeled data The model is trained by minimizing the sentence-level negative log-likelihood function loss. During the decoding process, the Viterbi algorithm is used to find the label sequence with the highest score.

Images