Chinese named entity recognition method and system based on stroke convolution and word vector
Through the method based on stroke convolution and word vector, combined with the self-attention mechanism and BiLSTM network, the CRF model is used for Chinese named entity recognition, which solves the problem of multiple meanings and insufficient boundary identifiers in Chinese named entity recognition, and improves the Identify accuracy and semantic understanding.
Patent Information
- Application Number
- CN202111641955.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-12-29
AI Technical Summary
There are problems with Chinese named entity recognition, lack of entity boundary identifiers and insufficient labeling data sets. The existing methods affect entity recognition when word segmentation stage errors. The word-based sequence labeling method lacks semantics and the introduction of external word vectors reduces accuracy. Rate.
Using a method based on stroke convolution and word vector, by obtaining stroke sequences and character feature vectors of Chinese characters, combining self-attention mechanisms and BiLSTM networks, a CRF model is used to predict entity labels, and a classification loss function is added to improve the accuracy of model training.
It improves the accuracy and effect of naming entity recognition, is suitable for the number of strokes of Chinese characters, makes up for the problem of semantics, and avoids the decrease in prediction accuracy caused by the introduction of external vocabulary.
Smart Images

Figure CN114298047B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of named entity recognition, and in particular to a Chinese named entity recognition method and system based on stroke convolution and word vector. Background Art
[0002] With the rapid development of Internet technology, unstructured data is growing continuously. We are in an era of massive and unstructured data. How to efficiently manage data and extract effective information from unstructured data has become an urgent problem that we need to solve.
[0003] The purpose of named entity recognition (NER) is to identify defined named entities from unstructured text, such as names of people, places, and institutions. It is a basic core task of information retrieval and information extraction. Chinese NER is a division of NER in the Chinese field. Due to the characteristics of Chinese characters, Chinese NER still has many problems. The main difficulties of Chinese NER are as follows: 1) Chinese characters usually have multiple meanings, and the meanings may be very different in different text contexts; 2) Chinese text does not have obvious entity boundary identifiers such as spaces in English text; 3) Chinese NER research started late, there are few related annotated data sets, and there are problems such as a single field.
[0004] There are usually two methods for existing Chinese named entity recognition, word-based sequence labeling methods and character-based sequence labeling methods. In the word-based labeling method, the text is first segmented using a word segmentation tool, and then entity recognition is performed. In this type of method, the word boundary is also the entity boundary. If an error occurs in the word segmentation stage, the subsequent NER model will not be able to correctly recognize the entity. Character-based sequence labeling methods usually have semantic deficiencies, so people mainly consider how to better utilize word information. Some users introduce external vocabulary information based on the character-based sequence labeling method and integrate it into the character vector representation at the input layer, which changes the model itself. At the same time, the introduction of external word vectors also makes the model training efficiency lower, and ultimately reduces the accuracy of named entity recognition; some users build the ElMo model based on the stroke sequence based on the character-based sequence labeling method, which has defects in the effectiveness and accuracy of named entity recognition. Summary of the invention
[0005] In view of the above problems, the present invention provides a Chinese named entity recognition method and system based on stroke convolution and word vector. On the basis of the character-based sequence labeling method in the named entity recognition method, the influence of the stroke sequence of Chinese characters on Chinese characters is taken into account. After combining the stroke feature vector, word feature vector and character feature vector of Chinese characters, named entity recognition is performed to improve the effect of named entity recognition.
[0006] To achieve the above object, the present invention provides a Chinese named entity recognition method based on stroke convolution and word vector, comprising:
[0007] Obtain the stroke sequence corresponding to each Chinese character in the text and the character feature vector of each Chinese character;
[0008] Inputting the stroke sequence into a stroke convolutional neural network to obtain a stroke feature vector;
[0009] Setting a sliding window according to the maximum length of entities in the text, and obtaining a word vector for each word in the sliding window through a self-attention mechanism;
[0010] The stroke feature vectors, word vectors and character feature vectors of each Chinese character in the text are concatenated and input into a BiLSTM network to obtain a score of each Chinese character corresponding to each entity label;
[0011] A CRF model is used to determine an optimal entity tag for each Chinese character in the text.
[0012] As a further improvement of the present invention, a mapping table from Chinese characters to stroke sequences is constructed, and the stroke sequence corresponding to each Chinese character is obtained through the mapping table.
[0013] As a further improvement of the present invention, the stroke convolutional neural network convolves the stroke sequence through convolution kernels with different window sizes to obtain the stroke feature vector.
[0014] As a further improvement of the present invention, the stroke convolution neural network obtains a stroke feature map by convolution kernels of different window sizes, and the feature map is subjected to maximum pooling and full connection to obtain a stroke feature vector, and the formula is:
[0015]
[0016] in:
[0017] w represents the weight in convolutional neural network training;
[0018] M t,t+k-1 Represents the characteristics of the input;
[0019] b represents the bias in convolutional neural network training;
[0020] As a further improvement of the present invention, a classification loss function L(cls) is added during the training process of the stroke convolutional neural network:
[0021] L(cls)=-logP(z|X)=-logsoftmax(w*semb)
[0022] in,
[0023] X represents the input stroke sequence;
[0024] z represents the Chinese label corresponding to the stroke sequence;
[0025] w represents the parameters in the network;
[0026] semb represents the stroke feature vector.
[0027] As a further improvement of the present invention, the step of obtaining the word vector of each word in the sliding window by the self-attention mechanism comprises:
[0028] Calculate the similarity between every two words in the sliding window by the self-attention mechanism;
[0029] The softmax function is used to obtain the word vector of each word in the sliding window according to the similarity.
[0030] As a further improvement of the present invention,
[0031] For each Chinese character in the sliding window, a corresponding Query vector, Key vector and Value vector are generated according to the character feature vector;
[0032] The dot product of the Query vector and the Key vector is calculated to obtain the score of each word, and the score is multiplied by the Value vector of each word to obtain the word vector of the word in the sliding window.
[0033] As a further improvement of the present invention, the method of using the CRF model to determine an optimal entity tag for each Chinese character in the text comprises:
[0034] Define the word sequence of input text as x = (x1, x2, ..., x n ), the predicted label sequence is y = (y1, y2, ..., y n );
[0035] definition The i-th word output by the BiLSTM network model is labeled as label y i The predicted score of
[0036] Define a label transfer matrix in Represents the score of converting from label yi to label yi+1;
[0037] pass Calculate the final score of each predicted label sequence;
[0038] The predicted label sequence with the highest score is used as the final label sequence, and the Chinese named entities are obtained according to the labels.
[0039] As a further improvement of the present invention,
[0040] Calculate the conditional probability of each predicted label sequence
[0041] If the conditional probability of the predicted label sequence with the highest score is also the largest, the predicted label sequence with the highest score is used as the final label sequence.
[0042] The present invention also provides a Chinese named entity recognition system based on stroke convolution and word vector, including a pre-preparation module, a stroke feature acquisition module, a word vector acquisition module, a label prediction module and an optimal label acquisition module;
[0043] The pre-preparation module is used to:
[0044] Obtain the stroke sequence corresponding to each Chinese character in the text and the character feature vector of each Chinese character;
[0045] The stroke feature acquisition module is used to:
[0046] Inputting the stroke sequence into a stroke convolutional neural network to obtain a stroke feature vector;
[0047] The word vector acquisition module is used to:
[0048] Setting a sliding window according to the maximum length of entities in the text, and obtaining a word vector for each word in the sliding window through a self-attention mechanism;
[0049] The label prediction module is used to:
[0050] The stroke feature vectors, word vectors and character feature vectors of each Chinese character in the text are concatenated and input into a BiLSTM network to obtain a score of each Chinese character corresponding to each entity label;
[0051] The optimal label acquisition module is used to:
[0052] A CRF model is used to determine an optimal entity tag for each Chinese character in the text.
[0053] Compared with the prior art, the present invention has the following beneficial effects:
[0054] The present invention is based on the character-based sequence labeling method in the named entity recognition method, takes into account the influence of the stroke sequence of Chinese characters on Chinese characters, combines the stroke feature vectors, word feature vectors and character feature vectors of Chinese characters, and then performs named entity recognition to improve the effect of named entity recognition.
[0055] In the process of obtaining the stroke feature vector, the present invention adopts the convolution method to extract the stroke feature vector of Chinese characters. The convolution method is more suitable for the number range of Chinese character strokes. At the same time, in the convolution process, convolution kernels with multiple window sizes are selected to convolve the stroke sequence to obtain the most effective stroke feature vector.
[0056] In the process of obtaining the word feature vector of Chinese characters in the present application, the word vector information in the sliding window is obtained through the self-attention mechanism to make up for the lack of semantics and avoid the situation in the prior art where the introduction of external vocabulary causes the prediction accuracy to decrease.
[0057] The present invention adds a classification loss function during the stroke convolutional neural network training process to improve the accuracy of the stroke convolutional neural network training. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 A flow chart of a Chinese named entity recognition method based on stroke convolution and word vector disclosed in an embodiment of the present invention;
[0059] Figure 2 A schematic diagram of a Chinese named entity recognition system based on stroke convolution and word vector disclosed in an embodiment of the present invention;
[0060] Figure 3 A schematic diagram of a stroke convolutional neural network disclosed in an embodiment of the present invention;
[0061] Figure 4 A schematic diagram of a self-attention mechanism model disclosed in an embodiment of the present invention;
[0062] Figure 5 A schematic diagram of a bidirectional time series model and a CRF model disclosed in an embodiment of the present invention. DETAILED DESCRIPTION
[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.
[0064] The following further describes the present invention in detail with reference to the accompanying drawings:
[0065] As Figure 1 shown, the Chinese named entity recognition method based on stroke convolution and word vectors provided by the present invention includes:
[0066] S1. Obtain the stroke sequence corresponding to each Chinese character in the text and the character feature vector of each Chinese character;
[0067] Among them,
[0068] The stroke sequence of each Chinese character in the training set during the training process is obtained through the Chinese Dictionary website, a mapping table from Chinese characters to stroke sequences is constructed, and the stroke sequence corresponding to each Chinese character in the text is obtained through the mapping table.
[0069] For example: For Figure 3 the Chinese character "清" shown, the stroke sequence obtained through the mapping table is "丶丶 一一 丨一丨一一".
[0070] S2. Input the stroke sequence into the stroke convolution neural network to obtain the stroke feature vector;
[0071] Among them,
[0072] As Figure 3 shown, the stroke convolution neural network performs convolution on the stroke sequence through convolution kernels with different window sizes. After convolution by the stroke convolution neural network, a stroke feature map is obtained. The feature map is subjected to max pooling and fully connected to obtain the stroke feature vector. The formula is:
[0073]
[0074] Among them:
[0075] w represents the weight in the training of the convolution neural network;
[0076] M t,t+k-1 represents the input feature;
[0077] b represents the bias in the training of the convolution neural network;
[0078] In the present invention, a classification loss function is added during the stroke convolutional neural network training process to improve the accuracy of training. The classification loss function is expressed as follows:
[0079] L(cls)=-logP(z|X)=-logsoftmax(w*semb)
[0080] in,
[0081] X represents the input stroke sequence;
[0082] z represents the Chinese label corresponding to the stroke sequence;
[0083] w represents the parameters in the network;
[0084] semb represents the stroke feature vector.
[0085] S3, set the sliding window according to the maximum length of the entity in the text, and obtain the word vector of each word in the sliding window through the self-attention mechanism;
[0086] in,
[0087] The word-based sequence labeling method usually has the problem of insufficient semantics. In order to better utilize the word vector information, the SA mechanism (self-attention mechanism) is used to obtain the word vector information in the sliding window to solve this problem.
[0088] During the training process, the maximum length of the entity in the training set is obtained, and the maximum length is used as the sliding window. The similarity between every two words in the sliding window is calculated through the self-attention mechanism; then the softmax function is used to obtain the word vector of each word in the sliding window according to the similarity.
[0089] Specifically, for each Chinese character in the sliding window, the corresponding Query vector, Key vector and Value vector are generated according to the character feature vector;
[0090] Calculate the dot product of the Query vector and the Key vector to get the score of each word, multiply the score by the Value vector of each word to get the word vector of the word in the sliding window.
[0091] For example:
[0092] like Figure 4 As shown, the text content is "Beijing City", then e 1 、e 2 、e 3 For each word, a Query vector, a Key vector and a Value vector are generated. These vectors are obtained by passing through the character feature vector e corresponding to each word. 1 、e2 、e 3 The score of each word is calculated by multiplying the three weight matrices created during the training process. The score of each word is calculated by the dot product between the Query vector and the Key vector, and then the score is multiplied by the corresponding Value vector to obtain the word vector corresponding to each word in the sliding window. The formula is:
[0093]
[0094] S4, concatenating the stroke feature vector, word vector and character feature vector of each Chinese character in the text, and inputting them into the BiLSTM network to obtain the score of each Chinese character corresponding to each entity label;
[0095] in,
[0096] Splicing is a direct splicing of vector dimensions. For example, the stroke feature vector of a Chinese character can be represented as 1*20, the word vector can be represented as 1*30, and the character feature vector can be represented as 1*60. After splicing, a spliced feature vector of 1*110 can be obtained.
[0097] BiLSTM (Bi-directional Long Short-Term Memory) is a bidirectional long short-term memory network; LSTM (Long Short-Term Memory) is a long short-term memory network, which is an improved time series network. It solves the gradient message problem and realizes the effective use of long-distance information. LSTM can only obtain unidirectional time series information, but context information has an important impact on the NER (named entity recognition) task. Therefore, this application uses the BiLSTM network to obtain context information;
[0098] like Figure 5 As shown in the figure, taking "Beijing Historical Materials" as an example, the scores of each word corresponding to multiple tags are obtained through forward LSTM calculation and reverse LSTM calculation. The tags here are pre-set and can include: address, time, name, book name, etc.
[0099] S5. Use the CRF model to determine the best entity label for each Chinese character in the text.
[0100] in,
[0101] In NER tasks, there are usually strong constraints between adjacent labels. For example, after the B-LOC label (the start label of the address), only the I-LOC label or the O label can be used, and other labels such as the B-PER label (the start label of a person's name) cannot be used. Therefore, after the sequence modeling is performed through the BiLSTM network, this paper uses the Conditional Random Field (CRF) to predict the label of the entire sequence, specifically:
[0102] Define the word sequence of input text as x = (x1, x2, ..., x n ), the predicted label sequence is y = (y1, y2, ..., y n ); Y(x) represents the set of all possible label sequences of the text;
[0103] definition The i-th word output by the BiLSTM network model is labeled as label y i The predicted score of
[0104] Define a label transfer matrix in Represents the score of converting from label yi to label yi+1;
[0105] pass Calculate the final score for each predicted label sequence;
[0106] The predicted label sequence with the highest score is used as the final label sequence, and the Chinese named entities are obtained based on the labels.
[0107] Furthermore,
[0108] You can set the loss function, such as:
[0109] Calculate the conditional probability of each predicted label sequence
[0110] If the conditional probability of the predicted label sequence with the highest score is also the largest, the predicted label sequence with the highest score is used as the final label sequence.
[0111] Finally, the best label sequence is found through the Viterbi algorithm. The formula is:
[0112]
[0113] like Figure 2 As shown, the present invention also provides a Chinese named entity recognition system based on stroke convolution kernel word vector, including a pre-preparation module, a stroke feature acquisition module, a word vector acquisition module, a label prediction module and an optimal label acquisition module;
[0114] Pre-prepared modules for:
[0115] Obtain the stroke sequence corresponding to each Chinese character in the text and the character feature vector of each Chinese character;
[0116] The stroke feature acquisition module is used to:
[0117] Input the stroke sequence into the stroke convolutional neural network to obtain the stroke feature vector;
[0118] Word vector acquisition module, used for:
[0119] Set a sliding window based on the maximum length of entities in the text, and obtain the word vector of each word in the sliding window through the self-attention mechanism;
[0120] Label prediction module, used to:
[0121] Concatenate the stroke feature vectors, word vectors, and character feature vectors of each Chinese character in the text, and input them into the BiLSTM network to obtain the scores of each Chinese character corresponding to each entity label;
[0122] Best label acquisition module for:
[0123] The CRF model is used to determine the best entity tag for each Chinese character in the text.
[0124] Advantages of the present invention:
[0125] The present invention is based on the character-based sequence labeling method in the named entity recognition method, takes into account the influence of the stroke sequence of Chinese characters on Chinese characters, combines the stroke feature vectors, word feature vectors and character feature vectors of Chinese characters, and then performs named entity recognition to improve the effect of named entity recognition.
[0126] In the process of obtaining the stroke feature vector, the present invention adopts the convolution method to extract the stroke feature vector of Chinese characters. The convolution method is more suitable for the number range of Chinese character strokes. At the same time, in the convolution process, convolution kernels with multiple window sizes are selected to convolve the stroke sequence to obtain the most effective stroke feature vector.
[0127] In the process of obtaining the word feature vector of Chinese characters in the present application, the word vector information in the sliding window is obtained through the self-attention mechanism to make up for the lack of semantics and avoid the situation in the prior art where the introduction of external vocabulary causes the prediction accuracy to decrease.
[0128] The present invention adds a classification loss function during the stroke convolutional neural network training process to improve the accuracy of the stroke convolutional neural network training.
[0129] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A Chinese named entity recognition method based on stroke convolution and word vector, characterized in that: include: Obtain the stroke sequence corresponding to each Chinese character in the text and the character feature vector of each Chinese character; Inputting the stroke sequence into a stroke convolutional neural network to obtain a stroke feature vector; Setting a sliding window according to the maximum length of entities in the text, and obtaining a word vector for each word in the sliding window through a self-attention mechanism; The stroke feature vectors, word vectors, and character feature vectors of each Chinese character in the text are concatenated and input into a BiLSTM network to obtain a score of each Chinese character corresponding to each entity label; Using a CRF model to determine an optimal entity tag for each Chinese character in the text; The stroke convolution neural network convolves the stroke sequence through convolution kernels of different window sizes to obtain the stroke feature vector; The stroke convolution neural network obtains a stroke feature map by convolution kernels of different window sizes, and performs maximum pooling and full connection on the feature map to obtain a stroke feature vector, and the formula is: in: w represents the weight in convolutional neural network training; M t,t+k-1 Represents the characteristics of the input; b represents the bias in convolutional neural network training; The step of obtaining the word vector of each word in the sliding window through the self-attention mechanism includes: Calculate the similarity between every two words in the sliding window by the self-attention mechanism; The softmax function is used to obtain the word vector of each word in the sliding window according to the similarity; specifically including: For each Chinese character in the sliding window, a corresponding Query vector, Key vector and Value vector are generated according to the character feature vector; The dot product of the Query vector and the Key vector is calculated to obtain the score of each word, and the score is multiplied by the Value vector of each word to obtain the word vector of the word in the sliding window.
2. The Chinese named entity recognition method according to claim 1, characterized in that: A mapping table from Chinese characters to stroke sequences is constructed, and the stroke sequence corresponding to each Chinese character is obtained through the mapping table.
3. The Chinese named entity recognition method according to claim 1, characterized in that: The classification loss function L(cls) is added during the stroke convolutional neural network training process: L(cls)=-logP(z|X)=-logsoftmax(w*semb) in, X represents the input stroke sequence; z represents the Chinese label corresponding to the stroke sequence; w represents the parameters in the network; semb represents the stroke feature vector.
4. The Chinese named entity recognition method according to claim 1, characterized in that: The method of using the CRF model to determine an optimal entity tag for each Chinese character in the text comprises: Define the word sequence of input text as x = (x1, x2, ..., x n ), the predicted label sequence is y=(y1,y2,…,y n ); definition The i-th word output by the BiLSTM network model is labeled as label y i The predicted score of Define a label transfer matrix in Indicates that from label y i Convert to label y i+1 score; pass Calculate the final score of each predicted label sequence; The predicted label sequence with the highest score is used as the final label sequence, and the Chinese named entities are obtained according to the labels.
5. A system for implementing the Chinese named entity recognition method according to any one of claims 1 to 4, characterized in that: It includes a pre-preparation module, a stroke feature acquisition module, a word vector acquisition module, a label prediction module and an optimal label acquisition module; The pre-preparation module is used to: Obtain the stroke sequence corresponding to each Chinese character in the text and the character feature vector of each Chinese character; The stroke feature acquisition module is used to: Inputting the stroke sequence into a stroke convolutional neural network to obtain a stroke feature vector; The word vector acquisition module is used to: Setting a sliding window according to the maximum length of entities in the text, and obtaining a word vector for each word in the sliding window through a self-attention mechanism; The label prediction module is used to: The stroke feature vectors, word vectors, and character feature vectors of each Chinese character in the text are concatenated and input into a BiLSTM network to obtain a score of each Chinese character corresponding to each entity label; The optimal label acquisition module is used to: A CRF model is used to determine an optimal entity tag for each Chinese character in the text.
Citation Information
Patent Citations
Text processing method and device and computing equipment
CN111274793A
Chinese named entity recognition method based on strokes
CN111523325A
