A Scalable and Universal End-to-End Named Entity Recognition Approach

Abstract

The present invention discloses an extensible universal end-to-end named entity recognition method, including a text preprocessing process, building a model M, training the model M, and using the model M to perform prediction and entity resolution processes, wherein the model M includes an input layer, a context encoding layer, and a graph modeling layer. The text preprocessing process: receiving a text input and an entity category, generating an input sequence; the training model includes obtaining a data set, converting the data set into a training set, and using the training set to perform multiple rounds of training on the model; after the model training is completed, the input sequence processed by the text preprocessing process is input into the model M, and the graph modeling layer of the model M calculates the connection relationship between words, and finally resolves the entities identified in the graph. The present invention can adapt to the recognition of overlapping entities and discontinuous entities, and can adapt to changes in demand such as the addition of new entity categories without modifying the model structure, so it is easy to expand and conduct domain transfer learning.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an extensible universal end-to-end named entity recognition method. Background Art

[0002] Named Entity Recognition (NER) is an important part of natural language processing. Named entity recognition refers to the process of identifying the names or symbols of things with specific meanings in text. Named entities mainly include names of people, places, institutions, dates, proper nouns, etc. Many downstream NLP tasks or applications rely on NER for information extraction, such as question answering, relationship extraction, event extraction, and entity linking. If the named entities in the text can be identified more accurately, it will help the computer better understand the semantics of the language and perform tasks better, thereby improving the human-computer interaction experience.

[0003] Transformer was proposed by the Google Brain team in the paper Attention is all you need in 2017. It uses the attention mechanism and full connection to build a network. Unlike the serial computing structure of recurrent neural networks, it can make full use of parallel computing.

[0004] Named entity recognition methods based on deep neural networks usually regard named entity recognition as a multi-classification task or a sequence labeling task, which can be divided into three processes: input representation layer, context encoding layer and label decoding. The input representation can be divided into three types according to the encoding object: character level, word level and mixed level, and the vector representation of each word can be obtained; semantic encoding usually uses deep neural networks, such as bidirectional long short-term memory neural networks, bidirectional encoder representation from Transformers (BERT) and transfer learning networks, so that the word vector of each word in the text contains context information; label decoding is completed by the classifier, and the classifier often uses a fully connected neural network + Softmax layer or a conditional random field + Viterbi algorithm to obtain the label of each word.

[0005] Most of the sequence-labeled named entity recognition models use CRF as the label decoding layer, and perform global optimization by adding label transfer score matrix parameters and defining the prediction score of the sequence. However, when the number of labels is large, the performance of CRF will be significantly reduced and the time complexity is very high. Later, a span-based recognition method appeared, which calculates the start and end positions of the entity, which can solve the entity recognition problem in the case of entity overlap, but cannot recognize non-continuous entities.

[0006] In reality, most named entity recognition scenarios are accompanied by a lack of corpus and changes in requirements, such as the need to add an additional entity category. A feasible method is to use the knowledge learned from a model trained in other similar scenario tasks, and use the weights of the model trained to initialize the model weights. For most named entity recognition models, the final classification layer structure is specially designed for an application scenario, and different numbers of entity categories will result in different output dimensions of the classification layer. Due to the differences in task scenarios, the dimensions of the last layer output of the two models will most likely be different. It is necessary to discard the last layer of the model and use the weights learned in the previous layer as the starting point for model training. Summary of the invention

[0007] In view of the above-mentioned problems, the purpose of the present invention is to provide a scalable universal end-to-end named entity recognition method, which can be applied to entity recognition in different task scenarios with different or changing numbers of entity categories without modifying the model, so it is very easy to expand and migrate to other fields. After the model in the present invention is trained in other task scenarios, it is not necessary to discard the last classification layer that depends on the task scenario and directly carry out the next scenario task training, thereby utilizing the knowledge learned by the model from other tasks. In the case of changing requirements, such as adding several new categories of entities, there is no need to modify the model and retrain, and only training data needs to be provided for the newly added categories of entities.

[0008] The present invention is applicable to entity recognition in a non-continuous situation, and is also applicable to entity recognition in a situation where entities overlap.

[0009] The specific technical solution for achieving the purpose of the present invention is:

[0010] A scalable general end-to-end named entity recognition method, the method comprising the following specific steps:

[0011] Step 1: Text preprocessing process to generate input sequence, specifically:

[0012] Receive a text input and an entity category, add a symbol at the beginning and end of the text, and add the entity category to the end of the text;

[0013] Segment the input text with symbols and entity categories added at the beginning and end to obtain word sequences;

[0014] Map the word sequence to numbers, with numbers and words mapped one to one, satisfying the bijective relationship, and taking the mapped number sequence as the input sequence, the output is the sequence;

[0015] Step 2: Build model M, including:

[0016] Use the context encoding layer to receive the input sequence output by the text preprocessing process, use the self-attention mechanism to generate a word vector group, and discard the word vector corresponding to the entity category name;

[0017] Use a directed connection graph to model the directed connection relationship between words. Use the word vector group to calculate the directed connection graph between each word. The directed connection graph is represented by a matrix, and the output is a graph represented by a matrix.

[0018] Step 3: Train model M;

[0019] Step 4: Use model M to make predictions;

[0020] Step 5: Perform entity resolution on the output of the model in step 4, specifically:

[0021] Receive the graph output by model M, start from the beginning, traverse the entire graph, except for the path that directly connects the first symbol to the last tail symbol, each path starting with the first symbol and ending with the last tail symbol corresponds to a word combination in the order of the path, which is an entity belonging to the category, and the output is the parsed entity set.

[0022] In step 2, a directed connection graph is used to model the directed connection relationship between entity words, and a word vector group is used to calculate the directed connection graph between each word, specifically:

[0023] If a word is the beginning of an entity, a directed edge is established connecting the sentence start symbol to the word;

[0024] If a word B follows word A in an entity, a directed edge is created from word A to word B;

[0025] If a word is the end of an entity, a directed edge connecting the word to the end symbol is created;

[0026] The words other than the first and last symbols are called middle words, and the corresponding word vectors are middle word vectors;

[0027] The connection between the first symbol and the middle word is calculated using the first word vector and the middle word vector, indicating the probability of the middle word starting the entity.

[0028] Use the word vectors of any two intermediate words to calculate the connection between the two words;

[0029] Use the end word vector and the middle word vector to calculate the connection between the middle word and the end symbol, indicating the probability of ending an entity with the middle word.

[0030] After the calculation is completed, a directed connection graph between words represented by a matrix is ​​obtained.

[0031] The training model M is specifically:

[0032] Get a labeled dataset. Each piece of data in the dataset includes text t and label y. Label y records all entity category names and corresponding entity sets contained in the text. If text t does not contain any type of entity, label y is empty.

[0033] Convert the dataset to a training set:

[0034] Define all entity category names appearing in the data set as set S, and assume that set S contains n entity category names; for each data (t, y) in the data set, t is the text, y is the label, and for each category s in set S, if the label y contains the entity corresponding to category s, that is, the text t contains a non-empty set e of entities belonging to category s, take category s and the corresponding entity set e as label y′, and take text t and label y′ as a piece of data in the training set; if the text t does not contain an entity belonging to category s, take category s and the corresponding empty set e′ of entities as label y′, and take text t and label y′ as a piece of data in the training set;

[0035] The model M is trained for multiple rounds using the training set. Each round of training includes:

[0036] The data of the training set is divided into multiple batches. Each time a batch of data is extracted from the training set, for each piece of data extracted, the entity set in the label is used to generate the true value of the adjacency matrix of the batch of data;

[0037] For each piece of data in a batch, use the text preprocessing process in step 1 to process the text in the data and the entity category in the label to generate an input sequence;

[0038] The input sequence is fed into the model, which calculates the connection between all words including symbols and outputs an adjacency matrix.

[0039] Finally, the loss is calculated using the matrix predicted by the model and the true value matrix generated by the labels, and the weights of the model are updated based on the loss.

[0040] The prediction using model M is specifically as follows:

[0041] Enter a text to extract entities from, without tags or other information;

[0042] Select an entity category to determine the entity category to search for in the text;

[0043] Input the text and entity categories into the text preprocessing process in step 1 to obtain an input sequence;

[0044] The input sequence is fed into the model M, and the model outputs a graph represented by an adjacency matrix.

[0045] Compared with the prior art, the present invention provides a scalable universal end-to-end named entity recognition method, which can be applied to different task scenarios without modifying the model, and can therefore be easily migrated to other fields. After the model in this method is trained in other task scenarios, there is no need to discard the last classification layer that depends on the task scenario, and there is no need to modify the model. It can be directly trained as a model for the next scenario task, so that the knowledge learned by the model from other tasks can be utilized. In the case of changes in demand, such as adding several types of entities, the conventional model needs to modify the output of the last layer of classification and retrain, while the model of this method does not need to modify the model and retrain, and only needs to provide training data for the newly added types of entities. This method uses a directed graph to model the connection relationship between entity words, which is suitable for entity recognition in non-continuous situations, and is also suitable for entity recognition in overlapping entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a structural diagram of a named entity recognition model according to an embodiment of the present invention;

[0047] Figure 2 is a flowchart of a text preprocessing process according to an embodiment of the present invention;

[0048] Figure 3 is a flow chart of converting a data set into a training set according to an embodiment of the present invention;

[0049] Figure 4 is a training flow chart of a method for named entity recognition according to an embodiment of the present invention;

[0050] Figure 5 The figure is a flowchart of a method for named entity recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] Example

[0052] The following example illustrates the method of the present invention. For simplicity, the word segmenter regards each Chinese character as a word, and uses [S] and [E] as the first and last characters:

[0053] This embodiment provides a scalable universal end-to-end named entity recognition method, the steps are as follows:

[0054] (1) The text preprocessor receives two inputs, including input text and entity categories, such as Figure 2As shown, for the input text, a symbol [S] and [E] are added to the beginning and end of the text respectively. The entity category is used as a hint to the model to extract the entity category from the input text. The text and entity category with symbols added to the beginning and end are used to convert the text input into a word sequence through word segmentation and dictionary mapping. The word sequence is mapped to numbers. The numbers and words are mapped one by one to satisfy the bijective relationship. The mapped number sequence is used as the input sequence.

[0055] (2) Construct a model, such as Figure 1 As shown, the model is divided into an input layer, a context encoding layer and a graph modeling layer. In the embodiment, the model uses BERT as the context encoding layer, and the input sequence processed by the text preprocessor is input into the context encoding layer. Each word vector in the word vector group output by the context encoding layer BERT layer is 768-dimensional. The word vector is reduced to 64 through a fully connected layer. Finally, the connection relationship between words is calculated using the word vector group. The Sigmoid function is used as the activation function of the last layer to model a directed connection graph represented by an adjacency matrix. The main diagonal and the part below the main diagonal are set to 0, and only the part above the main diagonal is used.

[0056] (3) Obtain annotated datasets. Each piece of data in most public datasets currently includes text t and label y. Label y records all entity categories and corresponding entity sets contained in the text. If text t does not contain any type of entity, label y is empty.

[0057] (4) Figure 3 As shown, the data set is converted into a training set: define all entity category names appearing in the data set as a set S, and assume that the set S contains n entity category names; for each data (t, y) in the data set, t is a text, y is a label, and for each category s in the set S, if the label y contains an entity corresponding to category s, that is, the text t contains a non-empty set e of entities belonging to category s, take category s and the corresponding entity set e as label y′, and take text t and label y′ as a piece of data in the training set; if the text t does not contain an entity belonging to category s, take category s and the corresponding empty set e′ of entities as label y′, and take text t and label y′ as a piece of data in the training set;

[0058] (5) Figure 4 As shown, the built model is trained using the training set for multiple rounds. Each round of training includes:

[0059] (51) Divide the data of the training set into multiple batches, extract a batch of data from the training set each time, and for each piece of data extracted, use the entity set in the label to generate the true value of the adjacency matrix;

[0060] (52) Use a text preprocessing program to process the data, add an [S] symbol to the beginning of the text of the data, add an [E] symbol to the end of the text, extract the category name in the label of the data, concatenate the category name after [E], perform word segmentation on the text with special symbols and category names added, and generate an input sequence;

[0061] (53) The input sequence is input into the model, and the model calculates the connection relationship between all words including special characters and outputs an adjacency matrix;

[0062] (54) Finally, the binary cross entropy loss is calculated using the model prediction matrix and the true value matrix generated by the labels and the model weights are updated based on the loss.

[0063] (6) After the model training is completed, the model is used for prediction. The specific process is as follows:

[0064] (61) selecting a text to extract entities from;

[0065] (62) selecting an entity category to determine the entity category to be searched in the text, which may be an entity category that has not appeared in the training set;

[0066] (63) Inputting the text and entity categories into a text preprocessing process to obtain an input sequence;

[0067] (64) Input the input sequence into the model M;

[0068] (65) Use the entity resolution process to resolve the entities identified in the graph.

[0069] After steps (1) to (5), the model is trained. When performing entity recognition on a text, each entity type in the entity type set needs to be recognized once in turn. The model will only give entities belonging to that category each time. By parsing the path of the graph output by the model, we can get the entities recognized by the model and contained in the text.

[0070] The present invention also takes the category of named entities as part of the input, with the purpose of extracting entities belonging to the category in the text. Taking the input text "Xiao Ming works at A software, Xiao Hong works at B song." as an example, after adding the first and last special characters [S] and [E], "company" is spliced ​​after the text as the category name, so that the text becomes "[S] Xiao Ming works at A software, Xiao Hong works at B song. [E] company", "A software" and "B song" are entities belonging to the company category contained in the text, and the goal of the model is to output "A software" and "B song". Starting with 0 as the index, in "[S] Xiao Ming works at A software, Xiao Hong works at B song. [E]", "A" and "soft" are at positions 4 and 5 respectively, "B" and "song" are at positions 12 and 13 respectively, and the two special characters "[S]" and "[E]" are at positions 0 and 17 respectively. Then the adjacency matrix E is a 17*17 dimensional matrix, E 0,4 、E 4,5 、E 5,17 The value of E is 1. 0,12 、E 12,13 、E 13,17 Corresponding to 1, E 0,4 、E 0,12 Respectively represent the start words with "A" and "B" as entities, E 5,17 and E 13,17 Indicates that "soft" and "song" are the end words of the entity, while E 4,5 and E 12,13 The next word of "A" is "soft", and the next word of "B" is "song". The first row of the matrix records the start word of the entity, and the last column records the end word of the entity. Although "Xiao Ming" and "Xiao Hong" belong to the category of names, when the category of "company" is input, these two words should not appear in the output graph.

[0071] From the above examples, we can see that the method of modeling the connection between words can easily deal with the situation of entity overlap and discontinuity. The entities with overlapping parts will contain a common path in the graph. If the entities are discontinuous, there will be a connection relationship even if the two words are separated by a distance. The entity in the graph output by the model is a path starting with "[S]" and ending with "[E]".

[0072] In this case where the category information is used as part of the input, the categories that have not appeared in the training set can also be spliced ​​at the end of the text as category information and input into the model. When the ClUENER2020 dataset is used to train the model, the dataset contains 10 categories, namely: name, address, organization, company, government, book, game, movie, position, and scenic spot. With "[S]o many animals on the prairie, lions, tigers, antelopes. [E] Animals" as input, the model correctly outputs "lion", "tiger", and "antelope", which shows that this model can use the knowledge learned from training on other tasks to predict entity categories that have not been seen.

[0073] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A scalable general end-to-end named entity recognition method, characterized in that: The method comprises the following specific steps: Step 1: Text preprocessing process to generate input sequence, specifically: Receive a text input and an entity category, add a symbol at the beginning and end of the text, and add the entity category to the end of the text; Segment the input text with symbols and entity categories added at the beginning and end to obtain word sequences; Map the word sequence to numbers, with numbers and words mapped one to one, satisfying the bijective relationship, and taking the mapped number sequence as the input sequence, the output is the sequence; Step 2: Build model M, including: Use the context encoding layer to receive the input sequence output by the text preprocessing process, use the self-attention mechanism to generate a word vector group, and discard the word vector corresponding to the entity category name; Use a directed connection graph to model the directed connection relationship between words. Use the word vector group to calculate the directed connection graph between each word. The directed connection graph is represented by a matrix, and the output is a graph represented by a matrix. Step 3: Train model M; Step 4: Use model M to make predictions; Step 5: Perform entity resolution on the output of the model in step 4, specifically: Receive the graph output by model M, start from the beginning, traverse the entire graph, except for the path that directly connects the first symbol to the last symbol, each path starting with the first symbol and ending with the last symbol corresponds to an entity belonging to the category according to the combination of words in the order of the path, and the output is the parsed entity set; where: In step 2, a directed connection graph is used to model the directed connection relationship between words, and a word vector group is used to calculate the directed connection graph between each word, specifically: If a word is the beginning of an entity, a directed edge is established connecting the sentence start symbol to the word; If a word B follows word A in an entity, a directed edge is created from word A to word B; If a word is the end of an entity, a directed edge connecting the word to the end symbol is created; The words other than the first and last symbols are called middle words, and the corresponding word vectors are middle word vectors; The connection between the first symbol and the middle word is calculated using the first word vector and the middle word vector, indicating the probability of the middle word starting the entity. Use the word vectors of any two intermediate words to calculate the connection between the two words; Use the end word vector and the middle word vector to calculate the connection between the middle word and the end symbol, indicating the probability of ending an entity with the middle word. After the calculation is completed, a directed connection graph between words represented by a matrix is ​​obtained; The training model M is specifically: Get a labeled dataset. Each piece of data in the dataset includes text t and label y. Label y records all entity categories and corresponding entity sets contained in the text. If text t does not contain any type of entity, label y is empty. Convert the dataset to a training set: Define all entity category names appearing in the data set as set S, and assume that set S contains n entity category names; for each data (t, y) in the data set, t is the text, y is the label, and for each category s in set S, if the label y contains the entity corresponding to category s, that is, the text t contains a non-empty set e of entities belonging to category s, take category s and the corresponding entity set e as label y′, and take text t and label y′ as a piece of data in the training set; if the text t does not contain an entity belonging to category s, take category s and the corresponding empty set e′ of entities as label y′, and take text t and label y′ as a piece of data in the training set; The model M is trained for multiple rounds using the training set. Each round of training includes: The data of the training set is divided into multiple batches. Each time a batch of data is extracted from the training set, for each piece of data extracted, the entity set in the label is used to generate the true value of the adjacency matrix of the batch of data; For each piece of data in a batch, use the text preprocessing process in step 1 to process the text in the data and the entity category in the label to generate an input sequence; The input sequence is fed into the model, which calculates the connection between all words including symbols and outputs an adjacency matrix. Finally, the loss is calculated using the matrix predicted by the model and the real value matrix generated by the label, and the weight of the model is updated according to the loss; the prediction using the model M is specifically as follows: Enter a text to extract entities from, without tags or other information; Select an entity category to determine the entity category to search for in the text; Input the text and entity categories into the text preprocessing process in step 1 to obtain an input sequence; The input sequence is fed into the model M, and the model outputs a graph represented by an adjacency matrix.

Images