An entity recognition method based on optimized word encoding under Bert model

Abstract

The present invention relates to the technical field of text processing, and discloses an entity recognition method based on optimizing word encoding under the Bert model, using a vocabulary library prepared in advance to perform full word matching on each character in the example sentence, extracting K possible words, and for each extracted word, recording the index position of its first character and last character in the sentence, designing an information fusion device, using a bilinear attention mechanism to fuse vocabulary features, vocabulary position features and character features, and using the structural advantages of BERT, after the information fusion device is embedded in the 0th layer of BERT, the training expression of BERT can be better utilized. The entity recognition method based on optimizing word encoding under the Bert model integrates vocabulary features and vocabulary position features on the basis of the BERT model, maximizes the increase of the input features of the model, and further improves the accuracy of Chinese named entity recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing, and in particular to an entity recognition method based on optimizing word encoding under a Bert model. Background Art

[0002] Chinese named entity recognition is a basic task in many downstream natural language processing (NLP) tasks. The research on Chinese NER has always received widespread attention. Compared with languages ​​such as English that have space segmentation, Chinese NER is more difficult because Chinese has the concept of words. Before performing Chinese NER, word segmentation is required. The quality of word segmentation determines the quality of Chinese NER. Therefore, some studies have shown that character-based Chinese NER methods perform better. However, if some methods can be found to better utilize part-of-speech features, the effect of Chinese NER will be greatly improved.

[0003] The full name of the BERT model is: Bidirectional Encoder Representations from Transformer. The goal of the BERT model is to use large-scale unlabeled corpus training to obtain the semantic representation of text containing rich semantic information, and then fine-tune the semantic representation of the text in a specific NLP task, and finally apply it to the NLP task.

[0004] BERT's input embedding is the sum of three embeddings:

[0005] 1. Token Embeddings are word vectors, which are character vectors in Chinese. The first word is the CLS flag, which can be used for subsequent classification tasks;

[0006] 2. Segment Embeddings is used to distinguish the two sentences, because pre-training not only does LM but also does a classification task with two sentences as input;

[0007] 3. Position Embeddings are calculated in a similar way to word embeddings.

[0008] BERT has made great achievements in the field of NLP. Traditional methods use the BERT model to perform text analysis. Generally, the initial input data is to convert the text into a word vector (referred to as word encoding), and then embed a position encoding information. However, the previous BERT model, its word encoding is generated by the Word2Vec model, and then its position information is based on the sin or cos function. Finally, the two are combined as the input data of the Bert model. However, the improvement brought by the word-based BERT in the field of Chinese NLP is limited. Because Chinese has the concept of vocabulary, the input vector and position vector of BERT are both for a single character, and there is no way to make good use of the vocabulary information of Chinese. To this end, a method for entity recognition based on optimizing word encoding under the Bert model is proposed. Summary of the invention

[0009] In view of the shortcomings of the prior art, the present invention provides an entity recognition method based on optimized word encoding under the Bert model, which has a position encoding method using head and tail indexes, and uses an attention mechanism to screen matching words for characters. It can very well integrate character features, vocabulary features, and vocabulary position features, thereby greatly improving the accuracy of Chinese NER and solving the defect that BERT cannot utilize vocabulary information.

[0010] The present invention provides the following technical solution: an entity recognition method based on optimizing word encoding under the Bert model, characterized in that it comprises the following steps:

[0011] Step 1: Prepare vocabulary in advance;

[0012] Step 2: Enter a sample sentence;

[0013] Step 3: Use the pre-prepared vocabulary to perform a full word match on each character in the example sentence, extract K possible words, and for each extracted word, record the index position of its first and last characters in the sentence;

[0014] Step 4: Get the ternary sequence of the example sentence according to step 3

[0015] ;

[0016] The method for obtaining the ternary sequence of the example sentence is: the example sentence is S, S has n characters, and the character is c, then the example sentence is: , each character k matching words have been found, then the word sequence of the i-th character is , each word has a corresponding first and last character index, then the word position sequence of the i-th character is , the initial ternary sequence is ;

[0017] in: is the lexical position of the character, The first character index position of the k-th word that matches the i-th character, The index position of the last character of the k-th word that matches the i-th character;

[0018] Step 5: Get the input triplet of the information adapter : They are the output of the 0th layer of Transformer, the vocabulary vector corresponding to the character, and the position encoding of the vocabulary vector corresponding to the character;

[0019] Step 6: Insert n information fusers between the 0th layer and the first layer of BERT;

[0020] Step 7: Perform bilinear attention calculation on the character vector and word vector to obtain the attention weight vector of each word , , Wattn is the weight matrix of bilinear attention calculation;

[0021] Step 8: Add the word vector and position vector corresponding to the character to fuse the information and get , ;

[0022] Step 9: Add the weights from step 7 The fusion information in step eight Perform weighted summation and get , ;

[0023] Step 10: The obtained features and layer 0 Add, integrate the features into the character vector, and get , ;

[0024] Step 11: The fused features Input to the first layer of BERT, and then train normally according to the BERT structure.

[0025] The BERT model used consists of L Transformer layers.

[0026] Input triples for the message adapter The method of obtaining is: input V and Ec into the 0th layer of BERT, and the output of the 0th layer, that is, the input character vector of the first layer, is obtained. , the first layer input of the i-th character in the example sentence S is ; Let Tencent word vector be , for each character Look up the table to get the word vector sequence ; right Perform dimension mapping and obtain .

[0027] The information fuser in step six takes as input the character vector, the corresponding word vector and the corresponding vocabulary position vector.

[0028] The position feature of a word is recorded using the first and last character index of the word.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. This entity recognition method based on optimizing word encoding under the Bert model incorporates vocabulary features and vocabulary position features on the basis of the BERT model, maximizes the input features of the model, and further improves the accuracy of Chinese named entity recognition.

[0031] 2. This entity recognition method based on optimizing word encoding under the Bert model designs an information fuser, adopts a bilinear attention mechanism to fuse vocabulary features, vocabulary position features and character features, and uses the structural advantages of BERT. After the information fuser is embedded into the 0th layer of BERT, it can better utilize BERT's training expression, thereby greatly improving the model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is the structural diagram of the i-th information fusion device;

[0033] Figure 2 Schematic diagram of the word encoding model structure. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] See also Figure 1 and 2 , an entity recognition method based on optimizing word encoding under the Bert model, comprising the following steps:

[0036] Step 1: Prepare vocabulary in advance;

[0037] Step 2: Enter a sample sentence;

[0038] Step 3: Use the pre-prepared vocabulary to perform a full word match on each character in the example sentence, extract K possible words, and for each extracted word, record the index position of its first and last characters in the sentence;

[0039] Step 4: Get the ternary sequence of the example sentence according to step 3

[0040] ;

[0041] The method for obtaining the ternary sequence of the example sentence is: the example sentence is S, S has n characters, and the character is c, then the example sentence is: , each character k matching words have been found, then the word sequence of the i-th character is , each word has a corresponding first and last character index, then the word position sequence of the i-th character is , the initial ternary sequence is ;

[0042] in: is the lexical position of the character, The first character index position of the k-th word that matches the i-th character, The index position of the last character of the k-th word that matches the i-th character;

[0043] Step 5: Get the input triplet of the information adapter : They are the output of the 0th layer of Transformer, the vocabulary vector corresponding to the character, and the position encoding of the vocabulary vector corresponding to the character. Through this adapter, the vocabulary information and the position information of the vocabulary are integrated into the BERT model, thereby enhancing the input features of BERT.

[0044] Input triples for the message adapter The method of obtaining is: input V and Ec into the 0th layer of BERT, and the output of the 0th layer, that is, the input character vector of the first layer, is obtained. ; The first layer input of the i-th character in the example sentence S is , let Tencent word vector be , for each character Look up the table to get the word vector sequence ; right Perform dimension mapping and obtain ;

[0045] Step 6: Insert n information fusers between the 0th layer and the first layer of BERT. The information fuser takes the character vector, the corresponding word vector and the corresponding vocabulary position vector as input. Through the setting of the information fuser, the vocabulary features, vocabulary position features and character features are integrated, which can better utilize the training expression of BERT and greatly improve the model performance.

[0046] Step 7: Perform bilinear attention calculation on the character vector and word vector to obtain the attention weight vector of each word , , Wattn is the weight matrix of bilinear attention calculation;

[0047] Step 8: Add the word vector and position vector corresponding to the character to fuse the information and get , ;

[0048] Step 9: Add the weights from step 7 The fusion information in step eight Perform weighted summation and get , ;

[0049] Step 10: The obtained features and layer 0 Add, integrate the features into the character vector, and get , ;

[0050] Step 11: The fused features Input to the first layer of BERT, and then train according to the BERT structure. The overall structure of the model is as follows Figure 2 .

[0051] The positional features of words are recorded using the first and last character indexes of the words. The method of integrating character, vocabulary, and positional features is a bilinear attention mechanism and an addition method.

[0052] The electrical components involved in this application are all available on the market and are all prior art. Although the embodiments of the present invention have been shown and described, it is understandable to those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An entity recognition method based on optimized word encoding under the Bert model, characterized in that: The following steps are involved: Step 1: Prepare vocabulary in advance; Step 2: Enter a sample sentence; Step 3: Use the pre-prepared vocabulary to perform a full word match on each character in the example sentence, extract K possible words, and for each extracted word, record the index position of its first and last characters in the sentence; Step 4: Get the ternary sequence of the example sentence according to step 3 ； The method for obtaining the ternary sequence of the example sentence is: the example sentence is S, S has n characters, and the character is c, then the example sentence is: , each character k matching words have been found, then the word sequence of the i-th character is , each word has a corresponding first and last character index, then the word position sequence of the i-th character is , the initial ternary sequence is ; in: is the lexical position of the character, The first character index position of the k-th word that matches the i-th character, The index position of the last character of the k-th word that matches the i-th character; Step 5: Get the input triplet of the information adapter : They are the output of the 0th layer of Transformer, the vocabulary vector corresponding to the character, and the position encoding of the vocabulary vector corresponding to the character; Step 6: Insert n information fusers between the 0th layer and the first layer of BERT; Step 7: Perform bilinear attention calculation on the character vector and word vector to obtain the attention weight vector of each word , , Wattn is the weight matrix of bilinear attention calculation; Step 8: Add the word vector and position vector corresponding to the character to fuse the information and get , ; Step 9: Add the weights from step 7 The fusion information in step eight Perform weighted summation and get , ; Step 10: The obtained features and layer 0 Add, integrate the features into the character vector, and get , ; Step 11: The fused features Input to the first layer of BERT, and then train normally according to the BERT structure.

2. According to claim 1, the entity recognition method based on optimized word encoding under the Bert model is characterized in that: The BERT model used consists of L Transformer layers.

3. The entity recognition method based on optimized word encoding under the Bert model according to claim 1 is characterized in that: Input triples for the message adapter The method of obtaining is: input V and Ec into the 0th layer of BERT, and the output of the 0th layer, that is, the input character vector of the first layer, is obtained. , the first layer input of the i-th character in the example sentence S is ; Let Tencent word vector be , for each character Look up the table to get the word vector sequence ; right Perform dimension mapping and obtain .

4. The entity recognition method based on optimized word encoding under the Bert model according to claim 3 is characterized in that: The information fuser in step six takes as input the character vector, the corresponding word vector and the corresponding vocabulary position vector.

5. The entity recognition method based on optimized word encoding under the Bert model according to claim 1, characterized in that: The position feature of a word is recorded using the first and last character index of the word.

Images