A method, device and storage medium for named entity recognition

Abstract

The present invention discloses a method, device and storage medium for named entity recognition, including: using BERT model for pre-training in two ways: using characters and words after word segmentation as tokens; after pre-training, using the last layer of the output part Transformer, the hidden layer tokens are spliced ​​in the last hidden layer to supplement token information; connecting a softmax classification layer in series above the last layer of Transformer; after obtaining the classification probability of each token based on characters and words, converting the classification probability of each token based on words into a label probability based on characters; according to the classification probability based on characters and words, each token takes the highest value as the label value of the token. The present invention can improve the performance of the model; capture the bidirectional context information in the true sense; supplement the entity slots for encoding and make reasonable use of them.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a named entity recognition method, device and storage medium. Background Art

[0002] NER (Named Entity Recognition) refers to the identification of entities with specific meanings in text or character strings, mainly including names of people, places, institutions and proper nouns. Judging whether a named entity is correctly identified includes two aspects: whether the boundary of the entity is correct; whether the type of the entity is correctly labeled. Named entities in English have relatively obvious formal signs (that is, the first letter of each word in the entity must be capitalized), so entity boundary recognition is relatively easy, and the focus of the task is to determine the category of the entity. Compared with English, the Chinese named entity recognition task is more complex, and compared with the entity category labeling subtask, the identification of entity boundaries is more difficult.

[0003] Existing named entity recognition methods mainly include: rule-based methods and statistic-based methods.

[0004] Early named entity recognition was mostly rule-based, using rule templates constructed by linguists, selecting features including statistical information, punctuation, keywords, indicator words and direction words, position words (such as tails), center words, etc., with pattern and string matching as the main means. This method is mainly used in some specific occasions where features are easy to summarize.

[0005] Statistical methods are ultimately classification methods. Given multiple categories of named entities, the model is used to classify the entities in the text. There are two approaches: one is to first identify the boundaries of all named entities in the text and then classify these named entities; the other is the serialization annotation method. For each word in the text, there can be several candidate category labels, which correspond to its position in each type of named entity. At this time, the NER task is to automatically annotate each word in the text in a serialized manner, and then integrate the automatically annotated labels to finally obtain the named entities composed of several words and their categories. Among them, serialization annotation is the most effective and most common NER method. Typical methods include: SVM (Support Vector Machine), ME (Maximum Entropy), HMM (Hidden Markov Model), CRF (Conditional Random Field), Neural Network, etc.

[0006] The deficiency of the prior art is that due to the inherent limitations of various named entity recognition methods in principle, entity boundary problems exist in the existing recognition models. Summary of the invention

[0007] The present invention provides a named entity recognition method, device and storage medium, which are used to solve the entity boundary problem existing in a named entity recognition model based on characters.

[0008] The present invention provides the following technical solutions:

[0009] A named entity recognition method, comprising:

[0010] Use the BERT model for pre-training in two ways: using characters as tokens and words after segmentation;

[0011] After the pre-training is completed, the last layer of the output part Transformer is used to concatenate the tokens of the hidden layer in the last hidden layer to supplement the information;

[0012] Connect a softmax classification layer in series above the last layer of Transformer;

[0013] After obtaining the character-based and word-based classification probabilities of each token, the word-based classification probability of each token is converted into a character-based label probability.

[0014] According to the character-based and word-based classification probabilities, each token takes the highest value as the label value of the token.

[0015] In implementation, the Token supplementary information is the average of the corresponding word vectors of one of the following information that can be collected or a combination thereof: known entity definitions, known entity description information, and structured knowledge graph information corresponding to known entities.

[0016] In implementation, the embedding of the BERT model input is the sum of the following parameter representations: word representation, positional representation, segment representation.

[0017] In the implementation, the vectorized representation of the words or characters in the word representation is as follows: the words or characters in the corpus after word segmentation are used as the dictionary, and the corresponding word or character vectors are trained with the Chinese corpus to obtain the word vectorized representation of the token;

[0018] The position vector representation of positional representation is as follows: after embedding the position information, the vector representation of the position is obtained;

[0019] The sentence vector of segment representation is represented as follows: for the data of sentence pairs, the embedding of sentence A is added to each word of the previous sentence, and the embedding of sentence B is added to each word of the next sentence.

[0020] In implementation, the pre-training is pre-training of a Masked Language Model.

[0021] In the implementation, before using the BERT model for pre-training, it further includes:

[0022] A binary classification model of whether it is the next sentence or not generated from a Chinese corpus is pre-trained.

[0023] During implementation, it further includes:

[0024] The parameters obtained from pre-training are jointly retrained.

[0025] A named entity recognition device, comprising:

[0026] The processor reads the program in the memory and performs the following processes:

[0027] Use the BERT model for pre-training in two ways: using characters as tokens and words after segmentation;

[0028] After the pre-training is completed, the last layer of the output part Transformer is used to concatenate the tokens of the hidden layer in the last hidden layer to supplement the information;

[0029] Connect a softmax classification layer in series above the last layer of Transformer;

[0030] After obtaining the character-based and word-based classification probabilities of each token, the word-based classification probability of each token is converted into a character-based label probability.

[0031] According to the character-based and word-based classification probabilities, each token takes the highest value as the label value of the token;

[0032] A transceiver is used to receive and send data under the control of the processor.

[0033] In implementation, the Token supplementary information is the average of the corresponding word vectors of one of the following information that can be collected or a combination thereof: known entity definitions, known entity description information, and structured knowledge graph information corresponding to known entities.

[0034] In implementation, the embedding of the BERT model input is the sum of the following parameter representations: word representation, positional representation, segment representation.

[0035] In the implementation, the vectorized representation of the words or characters in the word representation is as follows: the words or characters in the corpus after word segmentation are used as the dictionary, and the corresponding word or character vectors are trained with the Chinese corpus to obtain the word vectorized representation of the token;

[0036] The position vector representation of positional representation is as follows: after embedding the position information, the vector representation of the position is obtained;

[0037] The sentence vector of segment representation is represented as follows: for the data of sentence pairs, the embedding of sentence A is added to each word of the previous sentence, and the embedding of sentence B is added to each word of the next sentence.

[0038] In implementation, the pre-training is pre-training of a Masked Language Model.

[0039] In the implementation, before using the BERT model for pre-training, it further includes:

[0040] A binary classification model of whether it is the next sentence or not generated from a Chinese corpus is pre-trained.

[0041] During implementation, it further includes:

[0042] The parameters obtained from pre-training are jointly retrained.

[0043] A named entity recognition device, comprising:

[0044] The pre-training module is used to pre-train the BERT model using two methods: using characters as tokens and using words after segmentation;

[0045] Transformer module, which is used to output the last layer of the transformer after pre-training, and concatenate the tokens of the hidden layer with the tokens of the last hidden layer to supplement the information;

[0046] The softmax module is used to connect a softmax classification layer in series on the last layer of Transformer;

[0047] The probability module is used to obtain the classification probability of each token based on characters and words respectively, and then convert the classification probability of each token based on words into the label probability based on characters;

[0048] The label module is used to take the highest value of each token as the label value of the token based on the character-based and word-based classification probabilities.

[0049] During implementation, the Transformer module is further used to use the average of the corresponding word vectors of one or a combination of the following information that can be collected: known entity definitions, known entity description information, and structured knowledge graph information corresponding to known entities as the token supplementary information.

[0050] In implementation, the pre-training module is further used to input the embedding of the BERT model as the sum of the following parameter representations: word representation, positional representation, segment representation.

[0051] In the implementation, the pre-training module is further used to embed the BERT model input, which is the sum of the following parameter representations:

[0052] The vectorized representation of the word or character in word representation is as follows: the word or character in the corpus after word segmentation is used as the dictionary, and the corresponding word or character vector is trained with Chinese corpus, and the word vector representation of the token is obtained;

[0053] The position vector representation of positional representation is as follows: after embedding the position information, the vector representation of the position is obtained;

[0054] The sentence vector of segment representation is represented as follows: for the data of sentence pairs, the embedding of sentence A is added to each word of the previous sentence, and the embedding of sentence B is added to each word of the next sentence.

[0055] During implementation, the pre-training module is further used for pre-training using the Masked Language Model.

[0056] During implementation, the pre-training module is further used to pre-train a binary classification model of whether it is the next sentence generated in a Chinese corpus before using the BERT model for pre-training.

[0057] During implementation, the pre-training module is further used to jointly retrain the parameters obtained through pre-training.

[0058] A computer-readable storage medium stores a computer program for executing the above-mentioned named entity recognition method.

[0059] The beneficial effects of the present invention are as follows:

[0060] In the technical solution provided in the embodiment of the present invention, since the BERT model is used for pre-training in two ways, namely, using characters and words after word segmentation as tokens, the entity boundary problem existing in the named entity recognition model based on characters is solved by using the BERT model based on characters and words to reduce the problem that errors in word segmentation of unregistered entities will be forwarded, thereby improving the model performance.

[0061] Because it uses Transformer, it is more efficient than RNN and can capture longer-distance dependencies. Compared with the previous pre-trained model, it captures true bidirectional context information.

[0062] For the entity boundary problem existing in the character-based BI-LSTM-CRF model, the model performance can be further improved because the method based on character and word fusion adjustment is used to help determine the entity boundary;

[0063] Since the token supplementary information will be spliced ​​in the last hidden layer of the hidden layer, the external supplementary information such as entity definition, entity description information, and structured knowledge graph information corresponding to the entity can be used to supplement and encode the entity slot and make reasonable use of it;

[0064] Furthermore, since the parameters obtained by pre-training are jointly retrained, the training model can be adjusted by means of the pre-training model, thereby solving the problem that it is difficult to obtain a good training model when the training data of the current entity recognition method is small. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0066] Figure 1A schematic diagram of a named entity recognition method implementation flow in an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of the BERT model structure in an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of the pre-training model structure in an embodiment of the present invention;

[0069] Figure 4 Schematic diagram of a named entity recognition device structure 1 in an embodiment of the present invention;

[0070] Figure 5 Schematic diagram of the structure 2 of the named entity recognition device in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The inventors noticed during the invention process that:

[0072] Existing named entity recognition methods mainly include: rule-based methods and statistic-based methods. Early named entity recognition was mostly rule-based methods, which mostly used rule templates constructed by linguists, selected features including statistical information, punctuation, keywords, indicator words and directional words, positional words (such as tails), central words, etc., and used pattern and string matching as the main means. When the extracted rules can reflect language phenomena more accurately, the performance of rule-based methods is better than that of statistical-based methods. However, these rules often depend on specific languages, fields, and text styles. The compilation process is time-consuming and difficult to cover all language phenomena. They are particularly prone to errors, and the system is not portable. Linguistic experts need to rewrite the rules for different systems. Therefore, this method is mainly used in specific occasions where some features are easy to summarize.

[0073] Statistical methods are ultimately classification methods. Given multiple categories of named entities, the model is used to classify the entities in the text. There are two approaches: one is to first identify the boundaries of all named entities in the text and then classify these named entities; the other is the serialization annotation method. For each word in the text, there can be several candidate category labels, which correspond to its position in each type of named entity. At this time, the NER task is to automatically annotate each word in the text in a serialized manner, and then integrate the automatically annotated labels to finally obtain the named entities composed of several words and their categories. Among them, serialization annotation is the most effective and most common NER method. Typical methods include: SVM (Support Vector Machine), ME (Maximum Entropy), HMM (Hidden Markov Model), CRF (Conditional Random Field), Neural Network, etc.

[0074] HMM is a probabilistic directed graph model. The model mainly makes two assumptions, namely "first-order Markov assumption" and "observation independence assumption". The "first-order Markov assumption" describes that yt at the current moment is generated by yt-1 at the previous moment. The model can only model the previous information, but not the following information, which limits the model's expressive power. CRF is a probabilistic undirected graph model. The model needs to define relevant feature templates, which are used to scan and match the entire sentence. The features of the entire sequence are obtained by the linear weighted combination of local features. It cannot flexibly consider long-term context information, but can flexibly apply local context features. In addition, its model expression can cover the model parameters of HMM, so it has stronger expressive power than HMM. Among neural network methods, BI-LSTM (Bi-directional Long Short-Term Memory) is very powerful in sequence modeling and can capture long-term contextual information. It also has the ability to fit nonlinearity in neural networks, which are beyond the reach of traditional methods. However, this method lacks entity boundary features and is prone to errors in determining entity boundaries. It does not make reasonable use of external supplementary information such as entity definitions. In addition, this method requires more labeled training data. When there is less training data, it is difficult to train the network well.

[0075] The disadvantage of the prior art is that due to the inherent limitations of the principles of various methods, the existing recognition methods are prone to recognition errors.

[0076] As mentioned above, the existing Chinese named entity recognition methods have the following main problems:

[0077] 1. Rule-based methods rely on specific languages, fields, and text styles. The compilation process is time-consuming and difficult to cover all language phenomena. They are particularly prone to errors. The system is not portable and requires linguistic experts to rewrite the rules for different systems.

[0078] 2. The word-based BI-LSTM-CRF model needs to first use a word segmentation tool to segment text or strings. Word segmentation is very easy to make mistakes on unregistered words, and names of people, places, institutions, and proper nouns are unregistered words. Errors in word segmentation when using this model will be passed forward and affect model performance.

[0079] 3. Although the performance of the character-based BI-LSTM-CRF model in Chinese named entity recognition is higher than that of the word-based BI-LSTM-CRF model, the character-based BI-LSTM-CRF model lacks entity boundary information and is prone to errors in entity boundary recognition.

[0080] 4. There is no reasonable use of external supplementary information such as entity definition, entity description information, and knowledge graph information corresponding to the entity. The structured entity information in the knowledge graph is not used.

[0081] 5. The current entity recognition method requires more labeled training data. When there is less training data, it is difficult to train the network well and get the ideal effect.

[0082] Based on this, the technical solution provided in the embodiment of the present invention will solve at least one of the above problems. The specific implementation of the present invention is described below with reference to the accompanying drawings.

[0083] Figure 1 A flowchart for implementing a named entity recognition method, as shown in the figure, may include:

[0084] Step 101, using the BERT model for pre-training in two ways: using characters and words after segmentation as tokens;

[0085] Step 102: After the pre-training is completed, the last layer of the output part Transformer is used to concatenate the tokens of the hidden layer in the last hidden layer to supplement the information;

[0086] Step 103: Connect a softmax classification layer in series above the last layer of Transformer;

[0087] Step 104: After obtaining the classification probability of each token based on characters and words respectively, convert the classification probability of each token based on words into a label probability based on characters;

[0088] Step 105: According to the character-based and word-based classification probabilities, each token takes the highest value as the label value of the token.

[0089] First, the implementation of the BERT model structure is explained.

[0090] Figure 2 This is a schematic diagram of the BERT model structure. As shown in the figure, the model uses the BERT model for pre-training in two ways: using characters and words after segmentation as tokens. After the pre-training is completed, the last layer of the output part Transformer is used to concatenate tokens in the last hidden layer to supplement the information. A softmax (software maximum) classification layer is connected in series above the last layer. After obtaining the classification probability of each token based on characters and words, the classification probability of each token based on words is converted into a label probability based on characters. Finally, the classification probabilities based on characters and words are compared, and the highest value is taken as the label value of the token.

[0091] Among them, BERT is a pre-trained model. Assuming that there is a training set A, the network is pre-trained with A first, and the network parameters are learned on task A, and then saved for later use. When a new task B comes, the same network structure is adopted. When the network parameters are initialized, the parameters learned in A can be loaded, and other high-level parameters are randomly initialized. Then, the training data of task B is used to train the network. When the loaded parameters remain unchanged, it is called "frozen". When the loaded parameters are constantly changed with the training of task B, it is called "fine-tuning", that is, the parameters are adjusted better to make them more suitable for the current task B.

[0092] The core of this model is the focus mechanism. For a sentence, multiple focus points can be enabled at the same time, without being limited to serial processing from front to back or from back to front. Not only must the structure of the model be selected correctly, but the parameters of the model must also be trained correctly to ensure that the model can accurately understand the semantics of the sentence. BERT uses two steps to try to correctly train the parameters of the model.

[0093] The first step is to mask 15% of the words in an article and let the model predict the masked words omnidirectionally based on the context. If there are 10,000 articles, each with an average of 100 words, and 15% of the words are randomly masked, the model's task is to correctly predict these 150,000 masked words. By omnidirectionally predicting the masked words, the parameters of the Transformer model are preliminarily trained.

[0094] Then, the second step is used to continue training the model parameters. For example, from the above 10,000 articles, select 200,000 pairs of sentences, a total of 400,000 sentences. When selecting sentence pairs, 2*100,000 pairs of sentences are two consecutive context sentences, and the other 2*100,000 pairs of sentences are not consecutive sentences. Then let the Transformer model identify which of the 200,000 pairs of sentences are consecutive and which are not.

[0095] These two steps of training are combined into one, which is called pre-training. After the training, the Transformer model, including its parameters, is a general language representation model.

[0096] Tokens are pre-trained using both characters and words after segmentation. After pre-training, the last layer of the output part Transformer is used to concatenate tokens with hidden layer information in the last hidden layer. A softmax classification layer is connected in series above the last layer. After obtaining the classification probability of each token based on characters and words, the classification probability of each token based on words is converted into a label probability based on characters. Finally, the classification probabilities based on characters and words are compared, and the highest value is taken as the label value of the token.

[0097] In implementation, the Token supplementary information is the average of the corresponding word vectors of one of the following information that can be collected or a combination thereof: known entity definitions, known entity description information, and structured knowledge graph information corresponding to known entities.

[0098] Specifically, the token supplementary information is the average of the word vectors corresponding to the known entity definitions, entity description information, and structured knowledge graph information corresponding to the entities that can be collected.

[0099] The following is an explanation of the input implementation of the model.

[0100] In implementation, the embedding of the BERT model input is the sum of the following parameter representations: word representation, positional representation, segment representation.

[0101] In the specific implementation, the vectorized representation of the words or characters in the word representation is as follows: the words or characters in the corpus after word segmentation are used as the dictionary, and the corresponding word or character vectors are trained with the Chinese corpus to obtain the word vectorized representation of the token;

[0102] The position vector representation of positional representation is as follows: after embedding the position information, the vector representation of the position is obtained;

[0103] The sentence vector of segment representation is represented as follows: for the data of sentence pairs, the embedding of sentence A is added to each word of the previous sentence, and the embedding of sentence B is added to each word of the next sentence.

[0104] Token pre-trains models in the form of characters and words. The following pre-training is based on words. All available Chinese corpora are used for pre-training.

[0105] Figure 3 This is a schematic diagram of the pre-trained model structure. As shown in the figure, the embedding of the model input is the sum of three types of representations: word representation, positional representation, and segment representation.

[0106] 1) Word representation word (character) vector representation:

[0107] The words or characters in the corpus after word segmentation are used as the dictionary, and the corresponding word or character vectors are trained with the Chinese corpus, that is, word embedding (converting the word representation represented by the numerical value into a vector with a fixed size) to obtain the word vector representation of the token.

[0108] 2) Positional representation:

[0109] The position information is also embedded to obtain the vector representation of the position. The maximum length of the sequence is 512.

[0110] 3segment representation sentence vector representation:

[0111] For the data of sentence pairs, add the embedding of sentence A to each word in the previous sentence, and add the embedding of sentence B to each word in the latter sentence.

[0112] 4) The beginning of a sentence is indicated by [CLS], the end of a sentence is indicated by [SEP], and the space between two sentences in a sentence pair is also indicated by [SEP].

[0113] The implementation of model pre-training is described below.

[0114] In implementation, the pre-training is pre-training of a Masked Language Model.

[0115] The details can be as follows:

[0116] Task #1: Masked LM

[0117] In order to train the bidirectional features, the Masked Language Model pre-training method is used here to randomly mask some tokens in the sentence (for example, 15%), and then train the model to predict the removed tokens. The vector obtained by the hidden token in the last hidden layer is put into the softmax to calculate the probability of each word in the dictionary.

[0118] The specific operations are:

[0119] Randomly mask 15% of the tokens in the corpus, and then feed the final hidden vectors output at the masked token position into softmax to predict the masked token.

[0120] If [MASK] is used instead of token, it will affect the model, so the following strategy is adopted when random masking:

[0121] 1) 80% of the words are replaced by [MASK] token:

[0122] I watched the movie Avatar yesterday → I watched the movie [MASK] yesterday.

[0123] 2) 10% of the words are replaced with random words:

[0124] I watched the movie Avatar yesterday → I watched the movie Green Book yesterday.

[0125] 3) 10% of the words remain unchanged:

[0126] I watched the movie Avatar yesterday → I watched the movie Avatar yesterday.

[0127] Task 2#: Predict the next sentence

[0128] In implementation, before using the BERT model for pre-training, the following steps may be further performed:

[0129] A binary classification model of whether it is the next sentence or not generated from a Chinese corpus is pre-trained.

[0130] Specifically, in order to train a model that has the ability to understand the connection of long sequence contexts and understand sentence relationships, a binary classification model generated from a Chinese corpus can be pre-trained to determine whether it is the next sentence. This can be done as follows:

[0131] First prepare a training set: (sentence A, sentence B), and in 50% of cases B is the next sentence of A, and in 50% of cases B is another sentence randomly selected from the remaining corpus. Label is the relationship between two sentences (is it the next sentence / is not the next sentence), the beginning of a sentence is represented by [CLS], the end is represented by [SEP], and the space between two sentences in a sentence pair is also represented by [SEP]. For example:

[0132] Input=[CLS] I watched a movie yesterday[MASK][SEP]

[0133] Microsoft releases XiaoIce robot[MASK][SEP]

[0134] Label=NotNext

[0135] Input=[CLS] I watched a movie yesterday[MASK][SEP]

[0136] [MASK] I really like it [SEP]

[0137] Label=IsNext

[0138] The implementation of model fine-tuning is described below.

[0139] Slot information, i.e., token supplementary information, can be the average of the word vectors corresponding to the known entity definition, entity description information, and structured knowledge graph information corresponding to the entity that can be collected. For example, for "movie viewing time", the token supplementary information feature obtains the encoding vector of the token description by averaging the word vectors of all words in the description sentence.

[0140] In the output part, the last layer of the Transformer classifies the corresponding position of each word. For the sequence-level classification problem, add another layer to the original BERT model, W(K*H), where K is the number of categories to be classified and H is the output dimension of the transformer. After a softmax layer, the probability of the predicted category is P(K dimension) = softmax(CW^T).

[0141] During implementation, the method may further include:

[0142] The parameters obtained from pre-training are jointly retrained.

[0143] All parameters, including BERT's original pre-trained parameters and the new W parameters, will be jointly retrained, with the goal of minimizing the distance between the model's predicted probability and the true probability.

[0144] The implementation of label adjustment is described below.

[0145] After obtaining the classification probabilities of each token based on characters and words respectively, the classification probabilities of each token based on words are converted into label probabilities based on characters. For example, Xiao Ming: B-PER, p1 is converted into Xiao: B-PER, p1 and Ming: I-PER, p1,

[0146] Finally, compare the character-based and word-based classification probabilities and take the highest value as the label value of the token.

[0147] Table 1 Comparison table of part of speech categories

[0148] Part of Speech meaning n noun nr Name ns Place name nt Organization Name nz Other proper names

[0149] Based on the same inventive concept, an embodiment of the present invention also provides a named entity recognition device and a computer-readable storage medium. Since the principles of solving the problem by these devices are similar to those of the named entity recognition method, the implementation of these devices can refer to the implementation of the method, and the repeated parts will not be repeated.

[0150] When implementing the technical solution provided by the embodiment of the present invention, it can be implemented as follows.

[0151] Figure 4 The structure of the named entity recognition device 1 is shown in the figure. As shown in the figure, the device includes:

[0152] The processor 400 is used to read the program in the memory 420 and execute the following process:

[0153] Use the BERT model for pre-training in two ways: using characters as tokens and words after segmentation;

[0154] After the pre-training is completed, the last layer of the output part Transformer is used to concatenate the tokens of the hidden layer in the last hidden layer to supplement the information;

[0155] Connect a softmax classification layer in series above the last layer of Transformer;

[0156] After obtaining the character-based and word-based classification probabilities of each token, the word-based classification probability of each token is converted into a character-based label probability.

[0157] According to the character-based and word-based classification probabilities, each token takes the highest value as the label value of the token;

[0158] The transceiver 410 is configured to receive and send data under the control of the processor 400 .

[0159] In implementation, the Token supplementary information is the average of the corresponding word vectors of one of the following information that can be collected or a combination thereof: known entity definitions, known entity description information, and structured knowledge graph information corresponding to known entities.

[0160] In implementation, the embedding of the BERT model input is the sum of the following parameter representations: word representation, positional representation, segment representation.

[0161] In the implementation, the vectorized representation of the words or characters in the word representation is as follows: the words or characters in the corpus after word segmentation are used as the dictionary, and the corresponding word or character vectors are trained with the Chinese corpus to obtain the word vectorized representation of the token;

[0162] The position vector representation of positional representation is as follows: after embedding the position information, the vector representation of the position is obtained;

[0163] The sentence vector of segment representation is represented as follows: for the data of sentence pairs, the embedding of sentence A is added to each word of the previous sentence, and the embedding of sentence B is added to each word of the next sentence.

[0164] In implementation, the pre-training is pre-training of a Masked Language Model.

[0165] In the implementation, before using the BERT model for pre-training, it further includes:

[0166] A binary classification model of whether it is the next sentence or not generated from a Chinese corpus is pre-trained.

[0167] During implementation, it further includes:

[0168] The parameters obtained from pre-training are jointly retrained.

[0169] Among them, Figure 4In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 400 and memory represented by memory 420. The bus architecture may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 410 may be a plurality of components, i.e., including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 400 is responsible for managing the bus architecture and general processing, and the memory 420 may store data used by the processor 400 when performing operations.

[0170] Figure 5 The structure of the named entity recognition device 2 is shown in the figure. As shown in the figure, the device includes:

[0171] A pre-training module 501 is used to perform pre-training using a BERT model in two ways: using characters and words after segmentation as tokens;

[0172] Transformer module 502, used to use the last layer of the output part of the transformer after the pre-training is completed, and the tokens of the hidden layer are concatenated with the tokens in the last hidden layer to supplement the information;

[0173] The softmax module 503 is used to connect a softmax classification layer in series above the last layer of the Transformer;

[0174] A probability module 504 is used to obtain the classification probability of each token based on characters and words respectively, and then convert the classification probability of each token based on words into a label probability based on characters;

[0175] The label module 505 is used to take the highest value of each token as the label value of the token according to the character-based and word-based classification probabilities.

[0176] During implementation, the Transformer module is further used to use the average of the corresponding word vectors of one or a combination of the following information that can be collected: known entity definitions, known entity description information, and structured knowledge graph information corresponding to known entities as the token supplementary information.

[0177] In implementation, the pre-training module is further used to input the embedding of the BERT model as the sum of the following parameter representations: word representation, positional representation, segment representation.

[0178] In the implementation, the pre-training module is further used to embed the BERT model input, which is the sum of the following parameter representations:

[0179] The vectorized representation of the word or character in word representation is as follows: the word or character in the corpus after word segmentation is used as the dictionary, and the corresponding word or character vector is trained with Chinese corpus, and the word vector representation of the token is obtained;

[0180] The position vector representation of positional representation is as follows: after embedding the position information, the vector representation of the position is obtained;

[0181] The sentence vector of segment representation is represented as follows: for the data of sentence pairs, the embedding of sentence A is added to each word of the previous sentence, and the embedding of sentence B is added to each word of the next sentence.

[0182] During implementation, the pre-training module is further used for pre-training using the Masked Language Model.

[0183] During implementation, the pre-training module is further used to pre-train a binary classification model of whether it is the next sentence generated in a Chinese corpus before using the BERT model for pre-training.

[0184] During implementation, the pre-training module is further used to jointly retrain the parameters obtained through pre-training.

[0185] For the convenience of description, the various parts of the above-mentioned device are divided into various modules or units according to their functions and described separately. Of course, when implementing the present invention, the functions of each module or unit can be implemented in the same or multiple software or hardware.

[0186] A computer-readable storage medium is also provided in an embodiment of the present invention, wherein the computer-readable storage medium stores a computer program for executing the above-mentioned named entity recognition method.

[0187] For specific implementation, please refer to the implementation of the named entity recognition method.

[0188] To sum up, in the technical solution provided in the embodiment of the present invention, for the entity boundary problem existing in the character-based BI-LSTM-CRF model, a method based on the fusion adjustment of characters and words is adopted to help optimize the entity boundaries and further improve the model performance.

[0189] Utilize external supplementary information such as entity definition, entity description information, and structured knowledge graph information corresponding to the entity to supplement and encode the entity slots and make reasonable use of them.

[0190] It can be more efficient, capture longer-distance dependencies, and provide truly bidirectional contextual information.

[0191] By using the pre-training model solution and then adjusting the training model, we can solve the problem that it is difficult to obtain a good training model when there is less training data in the current entity recognition method.

[0192] The character and word-based BERT (Bidirectional Encoder Representations from Transformers) model is used to reduce the problem of errors in word segmentation of unregistered entities being passed forward and improve model performance. The Transformer is used, which is more efficient than RNN (Recurrent Neural Network) and can capture longer-distance dependencies.

[0193] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0194] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0195] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0196] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0197] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for named entity recognition, characterized in that: include: The BERT model is pre-trained using a Transformer-based bidirectional encoder representation with tokens using characters and segmented words. After the pre-training is completed, the last layer of the output part Transformer is used to concatenate the tokens of the hidden layer in the last hidden layer to supplement the information; Connect a softmax classification layer in series above the last layer of Transformer; After obtaining the character-based and word-based classification probabilities of each token, the word-based classification probability of each token is converted into a character-based label probability. According to the character-based and word-based classification probabilities, each token takes the highest value as the label value of the token.

2. The method according to claim 1, characterized in that The token supplementary information is the average of the corresponding word vectors of one or a combination of the following information that can be collected: known entity definitions, known entity description information, and structured knowledge graph information corresponding to known entities.

3. The method according to claim 1, characterized in that The embedding of the BERT model input is the sum of the following parameter representations: word representation, positional representation, and segment representation.

4. The method according to claim 3, characterized in that The vectorized representation of the word or character in word representation is as follows: the word or character in the corpus after word segmentation is used as the dictionary, and the corresponding word or character vector is trained with Chinese corpus, and the word vector representation of the token is obtained; The position vector of positional representation is as follows: after embedding the position information, the vector representation of the position is obtained; The sentence vector of segment representation is represented as follows: for the data of sentence pairs, the embedding of sentence A is added to each word of the previous sentence, and the embedding of sentence B is added to each word of the next sentence.

5. The method according to claim 1, characterized in that The pre-training is pre-training of a hidden language model MaskedLanguage Model.

6. The method according to claim 1, characterized in that Before using the BERT model for pre-training, further include: A binary classification model of whether it is the next sentence or not generated from a Chinese corpus is pre-trained.

7. The method according to claim 1, characterized in that Further including: The parameters obtained from pre-training are jointly retrained.

8. A named entity recognition device, characterized in that: include: The processor reads the program in the memory and performs the following processes: Use the BERT model for pre-training in two ways: using characters as tokens and words after segmentation; After the pre-training is completed, the last layer of the output part Transformer is used to concatenate the tokens of the hidden layer in the last hidden layer to supplement the information; Connect a softmax classification layer in series above the last layer of Transformer; After obtaining the character-based and word-based classification probabilities of each token, the word-based classification probability of each token is converted into a character-based label probability. According to the character-based and word-based classification probabilities, each token takes the highest value as the label value of the token; A transceiver is used to receive and send data under the control of the processor.

9. A named entity recognition device, characterized in that: include: The pre-training module is used to pre-train the BERT model using two methods: using characters as tokens and using words after segmentation; Transformer module, which is used to output the last layer of the transformer after pre-training, and concatenate the tokens of the hidden layer with the tokens of the last hidden layer to supplement the information; The softmax module is used to connect a softmax classification layer in series on the last layer of Transformer; The probability module is used to obtain the classification probability of each token based on characters and words respectively, and then convert the classification probability of each token based on words into the label probability based on characters; The label module is used to take the highest value of each token as the label value of the token based on the character-based and word-based classification probabilities.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 7.

Images