A Chinese named entity recognition method based on improved embedding distribution

Abstract

At present, with the development of machine translation, information extraction, conditional search and other fields, named entity recognition as the basic technology in these fields has also made further progress. At present, the commonly used word embeddings are divided into static word embeddings and dynamic word embeddings based on contextual semantic information (such as BERT word embeddings), but both of these word embeddings have certain shortcomings. Static word embeddings use fixed word vectors to express word units, without considering the different semantics of word units in different sentences; and BERT word embeddings that consider contextual semantics have the problem of representation degradation. In view of this situation, this method proposes a simple and effective word embedding method, which improves the distribution of static word embeddings and dynamic word embeddings to make them have isotropic distribution characteristics, so as to improve the semantic expression ability of word embeddings. At the same time, in order to better utilize semantic embeddings, the calculation method of the attention mechanism is improved. Finally, a Chinese named entity recognition model based on improved embedding distribution is constructed based on the transformer network architecture to solve the problem of naming recognition errors caused by the anisotropy of embedding distribution.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and natural language processing, and in particular to a Chinese named entity recognition method based on improved embedding distribution. Background Art

[0002] With the rapid development of machine translation, information extraction, conditional search and other fields, natural language processing is becoming more and more widely used in practical applications. Named Entity Recognition (NER) is one of the most important tasks in information extraction, machine translation and other fields. Its purpose is to automatically detect named entities in a given text and identify their categories. Since Chinese does not have similar space separators as in English text, Chinese NER is more difficult than English NER.

[0003] In the early stage of research, the Chinese NER task was decomposed into two independent serial tasks, namely word segmentation and word order tagging. This method has a serious flaw: if there is a word segmentation error, it will cause the word segmentation error to propagate in the subsequent network. In subsequent research, researchers proposed a character-based model to avoid word segmentation errors, but this completely discards word information and abandons the deep semantic information of words, which also reduces the semantic expression ability of embedded words in disguise. Therefore, researchers proposed to integrate word information into the character-based NER model. In recent years, researchers have focused on modeling the relationship between words and characters so that the model can learn some latent semantic information. For example, the (Xiaonan Li, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020. FLAT: Chinese NER using flat-lattice transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6836–6842, Online. Association for Computational Linguistics.) model proposes to model the relationship between word units based on positional encoding of span information; however, few people pay attention to the original word embedding in the Chinese NER task. If there is a problem with the original word embedding, it will produce the same error as the error propagation of word segmentation information. For static word embedding, (Gong C, He D, Tan X, et al. Frage: Frequency-agnostic word representation [J]. Advances in neural information processing systems, 2018, 31.) explored the impact of word frequency information on word embedding information, and proposed to remove word frequency information from the embedding information to solve the problem of weak semantic expression ability of low-frequency words; (Mu J, Bhat S, Viswanath P. All-but-the-top: Simple and effective postprocessing for word representations [J]. arXiv preprint arXiv: 1702.01417, 2017.) proposed a word embedding post-processing technique to make the existing representation more powerful by eliminating the common average vector and several top dominant directions.Word embeddings are distributed in a narrow cone area, which greatly limits the expressive power of word embeddings. (Gao J, He D, Tan X, et al. Representation degeneration problem in training natural language generation models [J]. arXiv preprint arXiv: 1907.1 2009, 2019.) proved this phenomenon and proposed a new regularization method to solve this problem. On the other hand, with the development of pre-training models, contextualized word embedding representations have become the mainstream trend instead of static word embeddings. In recent years, researchers have often used BERT pre-training models to obtain word embeddings as network input. However, researchers in named entity recognition rarely consider the reliability of word embeddings. (Ethayarajh K. How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings [J]. arXiv preprint arXiv: 1909.00512, 2019.) studied the word embeddings of each layer of the Elmo, BERT and GPT-2 pre-trained models, and proved that their distribution is not isotropic; (Li B, Zhou H, He J, et al. On the sentence embeddings from pre-trained language models [J]. arXiv preprint arXiv: 2011.05864, 2020.) Through theoretical discussions on embeddings, it was found that the BERT sentence vector space is semantically non-smooth. Therefore, the present invention improves the anisotropic distribution of word embeddings, and then proposes a simple and effective method to change the distribution characteristics of embeddings to solve this problem.On the other hand, enhancement methods based on embedded information have also been a research hotspot in recent years, (Aiguo Chen, Chenglong Yin. CRW-NER: Exploiting Multiple Embeddings for Chinese Named Entity Recognition [C] / / Proceedings of 2021 4th International Conference on Artificial Intelligence and Big Data (ICAI BD), 2021: 520-524.) and (Shuang Wu, Xiaoning Song, Zhenhua Feng. MECT: Multi-Metadata Embedding based Cross-Transformer for Chinese Named Entity Recognition [C] / / Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, ACL, 2021: 1529-1539.) The root information is integrated into the input layer to enhance the semantic information, which also shows that the auxiliary input information can improve the semantic richness to a certain extent. Therefore, the present invention adds part-of-speech information to the model in the embedding layer to achieve semantic enhancement.

[0004] To summarize, considering that researchers currently generally use static word embeddings and dynamic word embeddings based on semantic information, and static word embeddings have natural deficiencies in expressing polysemous words and the anisotropic distribution of dynamic word embeddings leads to the problem of expression degradation, the present invention designs a Chinese named entity recognition method based on improved embedding distribution, by performing different transformations on static word embeddings and dynamic word embeddings respectively, so that the input distribution of the model presents an isotropic characteristic, allowing the model to learn richer semantic information, thereby improving the accuracy of Chinese named entity recognition. Summary of the invention

[0005] The purpose of the present invention is to design a Chinese named entity recognition method based on improved embedding distribution to accurately identify entities in text, and on the basis of the Chinese named entity recognition method based on improved embedding distribution, fine-tune the pre-training model for specific fields of named entity recognition such as medical text to achieve the best implementation effect.

[0006] The present invention provides a Chinese named entity recognition method based on improved embedding distribution, comprising: an input information preprocessing module, which is used to preprocess the input text, add part-of-speech information to the input text through a natural language processing tool, and then perform embedding space transformation on static word embedding and BERT-based contextual semantic word embedding to make their distribution isotropic, and finally send them to a self-attention mechanism for modeling, and learn label constraints through conditional random fields to obtain the final prediction of entities and entity categories.

[0007] The content of this invention is mainly divided into three parts: static word embedding processing method, dynamic word embedding processing method and attention mechanism improvement method.

[0008] The Chinese named entity recognition method based on improved embedding distribution is the main content of the present invention. The Chinese named entity recognition method based on improved embedding distribution and attention mechanism proposed in the present invention obtains the final input of the network model by preprocessing the input text, and then performs long-distance dependency modeling through the attention mechanism and further feature extraction through the feedforward neural network. Finally, the output of the model is sent to the conditional random field for constraint learning of label information. Specifically, it includes the following steps:

[0009] 1. Preprocess the input text: In the embedding layer, the present invention divides the input into three parts. The first part uses the open source natural language processing library to extract the part-of-speech information and transfer it to the character level, and then uses the pre-trained vocabulary for vector mapping as input; the second part uses the BERT pre-trained model to obtain dynamic word embedding as input; the third part uses both static word vectors and dynamic word vectors as input. The processing method of static word vector is as follows: the word vector used is 50-dimensional, and the elements are enlarged first. First, the average value c of two adjacent values ​​a and b is calculated, and then a, b, and c are sorted. If the smallest values ​​among them are greater than 0, then let a add the absolute value of the difference between a and c, and let b add the absolute value of the difference between b and c. If the largest values ​​among them are less than 0, then let a subtract the absolute value of the difference between a and c, and let b subtract the absolute value of the difference between b and c. After that, after all the elements of the word vector are enlarged, the excessively large values ​​are scaled and reduced, and the excessively small values ​​are scaled and enlarged. The average value of 50 values ​​is calculated, the number of values ​​greater than the average and less than the average is counted, and then the value of the number of their differences is constrained so that they occupy more space in the embedding space. The dynamic word embedding adopts the BERT model with strong contextualization ability, so that the model can learn several meanings of the same word in different sentences. Therefore, the representation vector is only concentrated in a certain cluster in the vector space, which is expressed as anisotropy. This situation will reduce the generalization ability of the model. That is to say, the distribution of the same word in different sentences should be different, rather than having a conical distribution in the word embedding space with a high cosine similarity. The present invention first performs a random mask operation on the original BERT word embedding, and then passes it through a linear layer to make it have an isotropic distribution.

[0010] 2. Construct a Chinese named entity recognition network based on improved embedding distribution: In the calculation of the attention mechanism, the transformerXL calculation method is used, and the encoding method of the FLAT network is used for the position encoding part. For the input of the attention mechanism, the present invention modifies the original single input into three inputs, that is, the present invention performs three kinds of processing on the input in the embedding layer. The calculation method of the attention module is as follows:

[0011]

[0012] Att(A,V)=softmax(A)V

[0013] Among them, i represents the i-th word unit, ij represents the relationship between the i-th word unit and the j-th word unit. Q is the dynamic word embedding information obtained using the BERT pre-training model, K is the part-of-speech information obtained using natural language processing tools, and V is the representation of the static word vector and dynamic word vector information used simultaneously. u and v are learnable hyperparameters, and the position information encoding module in the attention mechanism It is the position information encoding in the attention mechanism, which is used to model the position information between words in the input sentence. The normalized exponential function softmax normalizes the attention value. for:

[0014]

[0015] In the above formula, h in i -h j represent Similarly, t i -t j represent and The calculation process is as follows:

[0016]

[0017]

[0018] In the above formula, d model is the dimension of the model, and the position d is calculated as follows:

[0019]

[0020] Where hh represents the distance from head[i] to head[j], i represents the i-th word, j represents the j-th word, and tt represents the distance from tail[i] to tail[j].

[0021] 3. Use open source data sets such as Weibo to pre-train the network and obtain a pre-trained model;

[0022] 4. Transfer the pre-trained model to the target domain through transfer learning;

[0023] 5. Obtain a pre-trained model suitable for the target domain through fine-tuning, and use this model to perform named entity recognition detection on the text in this domain to obtain the target entity and entity type.

[0024] Due to the adoption of the above technical solution, the present invention has the following advantages:

[0025] 1. Word embedding technology has brought great changes to natural language processing applications. Thanks to geometric expressions, word vectors can better capture language rules. Since Word2Vec was proposed, it has often been used as the basic input mapping in natural language processing tasks. For example, for the input sequence S = {s1, s2, ...s n}, often through Word2Vec to get the final input S ′ = Word2Vec (S). This kind of static mapping to obtain word embedding has a very obvious shortcoming, that is, the lack of flexible contextual expression. The different semantics of "leadership" can be clearly understood in the following two sentences: "under the leadership of ××" and "real leader". For such multi-semantic words, the expression of static word embedding cannot meet the requirements. Therefore, with the introduction of the BERT pre-training model, more researchers tend to use BERT-like pre-training models to obtain contextualized word embedding expressions. However, even if a large-scale corpus is used to train a neural network, most of the word vectors will still degenerate and be distributed in a small conical area of ​​the embedding space, which leads to a non-smooth anisotropic semantic space, that is, such a distribution impairs its semantic expression performance. In the present invention, the distribution of BERT word embedding is first processed to change its distribution from anisotropic to isotropic, and then it is fused with the static word embedding to achieve the complementary advantages of the two, thereby achieving better expression of semantic features.

[0026] 2. The present invention proposes an improvement on the self-attention mechanism. Different from the traditional transformerXL calculation method, the traditional self-attention mechanism calculation method adopts different linear transformations of the same input to implement it. Here, semantic embedding containing word information and contextual semantic information based on the BERT pre-trained model are used to interact. Finally, the static word embedding and the input representation of the BERT word embedding are combined through dot multiplication to perform information recovery, so that the model can pay attention to the semantic relationship and position relationship between word units. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0028] Figure 1 It is a flow chart of the Chinese named entity recognition method based on improved embedding distribution of the present invention;

[0029] Figure 2 is a static word embedding and a dynamic word embedding distribution graph of the present invention;

[0030] Figure 3 It is a distribution diagram after the word embedding feature is processed by the present invention.

[0031] Specific implementation plan

[0032] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0033] This paper proposes a Chinese named entity recognition method based on improved embedding distribution, such as Figure 1 , specifically including the following steps:

[0034] Step 1: Perform preprocessing operations such as word matching, part-of-speech matching, and word vector mapping on the input text;

[0035] Step 2: Build a neural network that integrates part-of-speech information and word information, and send the preprocessed vectors into the network for learning.

[0036] Step 3: Use the improved self-attention mechanism to train the attention of the three parts of the input obtained after preprocessing, so that when such information appears again in the future, it can automatically pay attention to this area;

[0037] Step 4: Send the output of the self-attention mechanism to the linear layer for feature learning;

[0038] Step 5: Send the output of the encoder into the conditional random field for label constraint learning to obtain the final predicted entity. DETAILED DESCRIPTION

[0039] Step 1: Receive input text, use the input preprocessing module to perform word matching and part-of-speech matching, then use the pre-trained vector table to map the word vectors, and linearly and uniformly amplify the static word vectors. First, measure the original static word vectors as follows: Figure 2 As shown in the figure (left), it is found that the distribution of static word embedding is close to the zero domain, which makes its distribution unable to express semantic features well. The word vector used here is 50-dimensional. The elements are first amplified. The specific implementation method is: calculate the average value c of two adjacent values ​​a and b, and then sort a, b, and c. If the smallest value among them is greater than 0, then let a add the absolute value of the difference between a and c, and let b add the absolute value of the difference between b and c. If the largest value among them is less than 0, then let a subtract the absolute value of the difference between a and c, and let b subtract the absolute value of the difference between b and c. After amplifying the elements of all word vectors, the overly large values ​​are scaled and reduced, and the undersized values ​​are scaled and amplified. The average value of 50 values ​​is calculated, and the number of values ​​greater than and less than the average value is counted, and then the value of the number of their differences is constrained. After processing the static word vector, the BERT word embedding needs to be processed. Here, the BERT word vector is measured as follows Figure 2As shown in the figure (right), its distribution is found to be conical, so the BERT word vector is added to the static word embedding after uniform amplification, and then subjected to dropout operation to obtain Figure 3 (middle) representation, and then through a linear layer to perform isotropic spatial transformation to obtain Figure 3 (Right) The vector representation shown.

[0040] Step 2: Send the preprocessed text to the self-attention mechanism module for modeling. Use the PyTorch framework to build this Chinese named entity recognition network. The position of the multi-head attention module in the overall framework is as follows: Figure 1 As shown, the overall calculation formula is as follows:

[0041]

[0042] Att(A,V)=softmax(A)V

[0043] Among them, i represents the i-th word unit, ij represents the relationship between the i-th word unit and the j-th word unit. Q is the dynamic word embedding information obtained using the BERT pre-training model, K is the part-of-speech information obtained using natural language processing tools, and V is the representation of the static word vector and dynamic word vector information used simultaneously. u and v are learnable hyperparameters, and the position information encoding module in the attention mechanism It is the position information encoding in the attention mechanism, which is used to model the position information between words in the input sentence. The normalized exponential function softmax normalizes the attention value. for:

[0044]

[0045] In the above formula, h in i -h j represent Similarly, t i -t j represent and The calculation process is as follows:

[0046]

[0047]

[0048] In the above formula, d model is the dimension of the model, and the position d is calculated as follows:

[0049]

[0050] Where hh represents the distance from head[i] to head[j], i represents the i-th word, j represents the j-th word, and tt represents the distance from tail[i] to tail[j].

[0051] Step 3: Send the output of the encoding part to the CRF layer for calculation, and obtain the final predicted entity through the constraint learning of the conditional random field on the label information.

[0052] Step 4: Train the constructed Chinese named entity recognition network. Through transfer learning, first use the open source data in related fields to pre-train the network, and then use the self-made annotated Chinese entity recognition dataset to fine-tune the pre-trained network.

Claims

1. A Chinese named entity recognition method based on improved embedding distribution, characterized in that: The proper nouns that can be focused on in a given text content are achieved through the following four steps: Step 1: Get the text object to be recognized and preprocess the input text; Step 2: Map the input text into word vector representation and build a Chinese named entity recognition network based on improved embedding distribution, including input information processing module, self-attention mechanism module, feedforward neural network module and CRF label constraint module; Step 2.1 The input information processing module uses the proposed method based on static word embedding and dynamic word embedding to modify the distribution. First, part-of-speech information is added to the input sentence and the matching word information is added to the input sentence. Then, part-of-speech tagging information is added to the matching word and transferred to the character level. Finally, the input information is matched with the word vector. In the process of word vector matching, the proposed method based on static word embedding to make the elements occupy a larger space in the embedding space and the modified distribution method of dynamic word embedding are used to make the words have an isotropic distribution in the embedding space for mapping. This mapping method performs a linear transformation on the original static word matching vector, scales and reduces the excessive values, calculates the average value of all values, counts the number of values ​​greater than the average value and less than the average value, and then constrains the number of their differences to increase the distance between word vectors so that they occupy a larger space in the vector space. For the dynamic word embedding with contextual semantic information obtained by Bert, a random mask operation is first performed on it, and then it is passed through a linear layer to make it have an isotropic distribution. Step 2.2 The self-attention mechanism module obtains the final feature input by performing self-attention calculation on the embedding information matching the part-of-speech information and the embedding information matching the word information. In the improved attention mechanism, K uses the embedding matching the part-of-speech information, Q is the dynamic word embedding obtained using BERT, and V is the vector representation after combining the dynamic word embedding and the static word embedding. The following is the calculation method of the attention mechanism: Att(A,V)=softmax(A)V Among them, u and v are learnable hyperparameters; i represents the i-th word unit, ij represents the relationship between the i-th word unit and the j-th word unit; the fusion position information encoding here is The FLAT encoding method is used, and the calculation method is: where h i -h j represent Similarly i -tj stands for and The calculation process is: In the above formula, d model is the dimension of the model, and the position d is calculated as follows: Where hh represents the distance from head[i] to head[j], i represents the i-th word, j represents the j-th word, and tt represents the distance from tail[i] to tail[j]. Step 2.3: The feedforward neural network module relearns the features obtained using the self-attention mechanism to obtain deeper spatial feature mapping information; finally, the CRF conditional random field module is used to add some constraints to the final predicted labels to ensure the accuracy of the predicted labels. These constraints are learned through the CRF layer and the prediction results are finally output; Step 3: Use the pre-trained model to perform named entity recognition on the input text to obtain entities and entity types.

Images