A nested named entity recognition method based on generative adversarial training

Abstract

The invention discloses a nested named entity recognition method based on generative adversarial training, which includes obtaining an input sequence; designing an entity tag for each entity type; using an extractor to extract all entity text segments specific to a certain entity tag to form an entity text segment candidate set; using an evaluator to evaluate the extracted entity text segment candidate set and give a score; the extractor and the discriminator are iteratively trained through generative adversarial training so that the extractor extracts an accurate entity text segment candidate set; after iterative training, the extractor selects all entity text segments of a certain entity type; each entity type is extracted and selected until all nested and non-nested entities in the input sequence are extracted. The invention adopts a text segment selection framework to solve the problem of nested named entity recognition, and a hybrid selection strategy uses text segment boundary information and text segment content information for extraction, and utilizes entity category information to design entity tags for each entity category.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a nested named entity recognition method based on generative adversarial training. Background Art

[0002] Named entity recognition (NER) is a basic task in natural language processing. Its purpose is to detect entity fragments and semantic categories from text. Named entity recognition is of great significance in the entire field of natural language processing and is widely used in knowledge graphs, machine question answering systems, and recommendation systems. NER is a very important part in many downstream tasks. For example, the e-commerce industry may need to extract brands or products from user queries to return content to users in a targeted manner.

[0003] Previous work usually regards the named entity recognition task as a sequence labeling problem, assigning a label (such as B / I / E tag) to each word in the text. However, when performing named entity recognition through sequence labeling methods, each word can only be in one entity reference at most. Therefore, it is impossible to handle the nested named entity recognition problem where "one word belongs to multiple entity references". Figure 1 In the example in Figure 1, a LOC (Location) entity "western Canadian" is nested in another GPE (Geo-Political Entity) entity "the western Canadian province of British Columbia". Nested entities are very common: in the widely used named entity recognition dataset GENIA, 17% of entities overlap with other entities; in the ACE2005 dataset, 30% of sentences contain nested named entities. Therefore, developing an effective model to solve the problem of nested named entity recognition is very necessary for subsequent tasks and applications.

[0004] Some existing nested named entity recognition works usually adjust the traditional sequence labeling model, modify the model's classification strategy or decoding layer to make it suitable for nested named entity recognition tasks. However, due to its characteristics of being suitable for assigning single labels, the sequence labeling model is naturally not suitable for the nested named entity recognition task that requires assigning multiple labels to a word. In view of this, some studies have turned to a two-stage framework, including transition-based methods, hypergraph-based methods, and paragraph-based methods. Transition-based methods usually construct a parse tree through state transition to extract nested entities. Each entity is organized into subtrees according to the structure of the entire binary tree. Hypergraph-based methods construct a hypergraph according to the structure of the nested entities, and then decode the nested entities on the hypergraph. Paragraph-based methods enumerate all possible paragraphs and then classify these paragraphs. Although paragraph-based methods have the natural ability to solve the problem of nested named entity recognition, these methods still have a series of shortcomings. First, this two-stage method inevitably faces the problem of error propagation; second, due to the need to enumerate a large number of low-quality candidate paragraphs, these methods have very high computational overhead; third, in these methods, the boundary information of entities is not fully utilized, but entity boundary information is very important for accurately identifying entity paragraphs; fourth, in paragraph-based methods, the length of paragraphs is limited, so this method cannot recognize long entities. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a novel nested named entity recognition method, named Extract-Select. The present invention first proposes a text selection framework, the purpose of which is to train an extractor for each entity type to extract nested entities. Figure 1 In the example in Figure 2, the nested entities "PEBP2" (entity type is PROTEIN) and "PEBP2 site" (entity type is DNA) will be extracted by two extractors, which take different entity types as prior input information. Therefore, the nested named entity recognition problem can be solved naturally: the two nested entities will be extracted by the extractor according to the different entity types. At the same time, due to the characteristics of the paragraph selection framework, the error propagation problem and the high computational overhead problem existing in the previous methods can be naturally solved.

[0006] The present invention then combines the proposed text selection framework with generative adversarial training (GAT) to design a model called Extract-Select. Generative adversarial training can greatly reduce the dependence of the Extract-Select model of the present invention on training data, so that the model can achieve relatively good results even when there is little training data. Previous methods usually rely heavily on training data to train the model, and have relatively high requirements on the quality of the data set.

[0007] The Extract-Select model proposed by the present invention is as follows: First, an entity tag is designed for each entity type, and the entity tag contains rich prior information about the entity type. Then, the present invention designs an extractor, the function of which is to extract (Extract) all entity segments specific to a certain entity tag to form an entity segment candidate set. After that, a discriminator is introduced to evaluate the entity segment candidate set extracted by the extractor and score the candidate set. The extractor and the discriminator are iteratively trained by generative adversarial training, so that the extractor can extract a more accurate entity segment candidate set. Finally, in the inference stage, the iteratively trained extractor will select (Select) all entity segments of a certain entity type. Extraction-selection is performed for each entity type, and all nested and non-nested entities in the input text can be extracted. Initially, the entity tag is constructed by the information of the entity category, encoding rich prior information of the entity category. With the entity tag as an additional input, the extractor can be more clear about what to extract. Compared with the method of the present invention, the previous method only uses the entity category as the category index for entity classification, lacks the guidance of semantic information during entity recognition, and the effect of named entity recognition is worse.

[0008] In the extractor, the present invention needs to design a paragraph extraction strategy. Traditional paragraph extraction methods include extraction based on paragraph boundaries and extraction based on paragraph content, and neither of these two methods is suitable for the extractor of the present invention. Because the entity paragraphs predicted by the above two methods are discrete, they cannot be used in the gradient back propagation of subsequent generative adversarial training. In order to allow the gradient to be transmitted from the discriminator to the extractor, the present invention proposes a new hybrid selection strategy. This strategy takes into account both the boundaries of the paragraph and the content of the paragraph, and the candidate entity paragraphs generated by them can be used as the input of the subsequent discriminator. Considering the boundary information and content information of the paragraph at the same time, the hybrid selection strategy can represent the candidate set of the entity paragraph as a continuous hidden variable. Therefore, gradient back propagation can be implemented in the process of generative adversarial training. In addition, this novel hybrid selection strategy can make full use of the boundary information of the paragraph to achieve more accurate paragraph extraction. This strategy is also beneficial to long entity recognition because it does not limit the length of the entity paragraph in entity paragraph recognition.

[0009] Specifically, the nested named entity recognition method based on generative adversarial training disclosed in the present invention includes the following steps:

[0010] Get the input sequence;

[0011] Design an entity tag for each entity type, wherein the entity tag contains prior information about the entity type;

[0012] Use the extractor to extract all entity texts specific to a certain entity tag to form an entity text candidate set;

[0013] Evaluate the extracted entity paragraph candidate set through an evaluator, and give a score to the entity paragraph candidate set;

[0014] The extractor and the discriminator are iteratively trained by generative adversarial training, so that the extractor extracts an accurate set of entity paragraph candidates;

[0015] After iterative training, the extractor selects all entity segments of a certain entity type;

[0016] Extraction-selection is performed for each entity type until all nested and non-nested entities in the input sequence are extracted, and the fragments and semantic categories of all nested and non-nested entities are output.

[0017] Furthermore, the extractor is used to extract all entity text segments specific to a certain entity tag, including: input sequence representation; entity tag representation and candidate entity text segment extraction.

[0018] Further, the input sequence representation includes:

[0019] Represent all word-grams of the input sequence S as an embedding sequence, where each embedding representation in the embedding sequence is a concatenation of a character vector, a word vector, a context word vector, and a part-of-speech vector, wherein the character vector is generated by inputting the word-gram representation into the first Bi-LSTM network; the context word vector is obtained by generating a context-dependent context embedding for each word-gram;

[0020] The connection of the character vector, word vector, context word vector and part-of-speech vector is input into the second Bi-LSTM network to obtain the final word unit representation.

[0021] Furthermore, the entity tag is a fine-grained representation of the entity type, each entity type corresponds to an entity tag, and the entity tag representation includes:

[0022] Use the connection between the keyword and synonym of the entity type as the entity tag, the keyword is a word describing the entity type, and the synonym refers to a word or phrase extracted from the Oxford Dictionary with the same meaning as the keyword;

[0023] Connect the word vectors of keywords and synonyms to get the entity tag y * The representation vector is then input into the second Bi-LSTM network to obtain the final representation of the entity tag:

[0024]

[0025] Among them, h i is the hidden state vector, encoding the word x i context information;

[0026] The final representation of the entity tags is integrated using a self-attention mechanism:

[0027]

[0028]

[0029] Among them, α j is u j The attention weight, W a is a learnable parameter and m is the final representation of the entity tag.

[0030] Furthermore, the candidate entity paragraph extraction uses a hybrid selection strategy, which first predicts the probability that each word is the start and end index of the entity paragraph, and then uses the probability to generate an entity paragraph candidate set and calculates the representation of the candidate set.

[0031] Furthermore, the hybrid selection strategy specifically includes:

[0032] Given the representation vectors of the sequence and entity tokens, the extractor first predicts the probability that token i is the start index and end index of the entity context:

[0033]

[0034]

[0035] Among them, W s and W e is the parameter to be learned, h i is the hidden state vector of the i-th word, h k is the hidden state vector of the kth word, and m is the representation of the entity tag;

[0036] Get possible start or end index:

[0037] I s ={i|argmax(p s (i|y,S))=1,i=1,...,n}

[0038] I e ={j|argmax(p e (j|y,S))=1,j=1,...,n}

[0039] Then, for each given starting index i s ∈I s and end index j e ∈I e , and i s ≤j e , using a binary classifier to predict entity segment candidates Probability:

[0040]

[0041] Where W C are the parameters to be learned, is the hidden state vector of the starting word, is the hidden state vector of the ending word.

[0042] Entity paragraph candidates are extracted through a probability threshold, and all possible entity paragraph candidates constitute the entity paragraph candidate set C;

[0043] Calculate p C (i|y, S) is the probability that the i-th word appears in the candidate set C. In this way, the continuous vector p C To represent C.

[0044] Furthermore, the evaluation of the discriminator includes the candidate text set representation and candidate set scoring:

[0045] The candidate text set represents encoding the entity type y and the sequence S into the entity text candidate set representation p C , specifically including the following steps: Use Match-LSTM network to build entity-aware sequence representation:

[0046]

[0047] Among them, h i is the hidden state vector, encoding the word x i Rich contextual information, is the final representation of the entity tag, |y * | is the length of the entity tag, and |S| represents the length of the sequence.

[0048] The representation p of the entity segment candidate set C By weighted summation with r i and probability p C (i|y, S) calculation:

[0049]

[0050]

[0051] where r C is the representation of the candidate set of entity paragraphs, β i is the weight coefficient, r k is the representation of the kth word in the sequence S, p C (i|y, S) is the probability that the i-th word appears in the candidate set C,

[0052] The candidate set scoring includes: the scores of the entity segment candidate set extracted by the extractor are calculated by the sigmoid function:

[0053] score = sigmoid(W b r C )

[0054] Among them, W b are learnable parameters and the scores are used to iteratively train the extractor and discriminator during generative adversarial training.

[0055] Furthermore, the extractor and the discriminator are iteratively trained through generative adversarial training, including three tasks, wherein the first task is to train the extractor to correctly predict the start and end indexes of the entity segment; the second task is to train the extractor to better match the start-end indexes; the third task is to jointly train the extractor and the discriminator through generative adversarial training, wherein the extractor is regarded as a generator and is trained to obtain higher scores from the discriminator; after iterative training, the extractor is used to select the final result of the entity segment.

[0056] Furthermore, the first task is specifically to minimize the negative logarithmic probability of the true start and end indexes of the golden entity text segment, and the loss function is as follows:

[0057] l′ E =-logp s (s|y,S)-logp e (e|y,S)

[0058] Where s and e represent the start and end indexes of the golden entity text in sequence S, p s (s|y, S) is the probability of the true start index of the golden entity segment, p e (s|y, S) is the probability of the true end index of the golden entity segment;

[0059] The second task is to minimize the negative logarithmic probability of the true start-end index match of the golden entity segment, and the loss function is as follows:

[0060] l″ E =-logp s，e

[0061] where p s，e is the probability of a true start-end index match of the golden entity segment;

[0062] The third task is to train the extractor to obtain a higher score from the discriminator, and the loss function is as follows:

[0063] l″′ E =log(1-f D (y,S)

[0064] where f D (y, S) is the score of the entity paragraph candidate set extracted by the extractor;

[0065] The overall training objective function of the extractor is:

[0066] l E =γ1l′ E +γ2l″ E +(1-γ1-γ2)l″′E

[0067] Where γ1, γ2∈[0, 1] are hyperparameters that balance the contribution of each part to the overall objective function. The three loss functions are jointly trained in an end-to-end manner;

[0068] The final result of using the extractor to select entity paragraphs includes: in the reasoning stage, given the sequence and entity tag y * , the trained extractors are based on I s and I e The start and end indexes are selected; then the start and end indexes are matched to obtain entity text segments of a specific entity category.

[0069] The beneficial effects of the present invention are as follows:

[0070] The present invention proposes a novel Extract-Select model to solve the problem of nested named entity recognition. The Extract-Select model adopts a text selection framework, which can use an extractor to extract all possible entity texts for each entity category, and use a discriminator to evaluate and score the extracted texts, which can naturally solve the problem of nested named entity recognition. The extractor and the discriminator are trained together through generative adversarial training, so that the extractor can get higher scores from the discriminator. In the reasoning stage, the trained extractor is used to select entity texts specific to each entity type, that is, named entities. The Extract-Select model of the present invention reduces the dependence on training data, so that the model can achieve relatively good results even when there is little training data.

[0071] The present invention proposes a hybrid selection strategy to extract entity paragraph candidates and represent them as a continuous latent variable. This hybrid selection strategy uses both paragraph boundary information and paragraph content information for extraction. Through this strategy, the representation of the entity paragraph candidate set can be propagated to the discriminator and scored. The extractor and the discriminator can be trained and optimized together in the generative adversarial training process.

[0072] The present invention makes full use of entity category information and designs entity tags for each entity category. By considering the named entity recognition task as a text selection task for each entity, the Extract-Select model of the present invention uses entity tags as additional inputs of the extractor to guide the extractor to extract more accurate results. In addition, when the discriminator evaluates the results extracted by the extractor, the entity tags are also used as additional inputs to enhance the representation of the entity text candidate set.

[0073] The present invention conducts experiments on four widely used nested named entity recognition datasets and compares the Extract-Select model with a series of state-of-the-art models. The experimental results show that the Extract-Select model consistently outperforms the state-of-the-art models on all four datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 Schematic diagram of an example of a nested named entity recognition problem;

[0075] Figure 2 The Extract-Select model and training process framework diagram of the present invention. DETAILED DESCRIPTION

[0076] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0077] The input of the passage selection framework is a sequence S = {x1, x2, ..., x |S|}, where |S| represents the length of the sequence. Possible entity text s s，e ={x s , x s+1 , …, x e-1 , x e} is a continuous subset of the sequence S and satisfies s≤e. Let Y represent the set of all predefined entity categories, y∈Y is one of the entity categories, y * is an entity tag specific to entity type y, * is designed to represent the prior information of entity type y. The task of the text selection framework is to train an extractor to find matching y in sequence S for each entity type y. * Through the text selection framework, the problems of error propagation and high computational overhead in traditional text-based methods naturally disappear.

[0078] Generative adversarial training is a novel way to learn generative models of arbitrarily complex data distributions. Generative adversarial training consists of a generator G that generates a random latent distribution p. z (z) samples are mapped to the data space G(z); it also includes an adversarial discriminator D to distinguish between real samples and generated samples as accurately as possible. The purpose of the generator is to learn a generated distribution p that is as close to the real data distribution as possible. G(x)To deceive the discriminator. The generator and the discriminator are trained through the min-max game (min-max is a prior art in this field and will not be described in detail). In other words, to minimize log(1-D(G(z))) and maximize log D(x):

[0079]

[0080] Generative adversarial training uses the discriminator to optimize the generator, avoiding the design of additional loss functions.

[0081] The present invention proposes a model called Extract-Select, which includes an extractor that follows a text selection framework to extract, and a discriminator that aims to evaluate and score the extraction results. The present invention then uses generative adversarial training to train the extractor and the discriminator together. The use of generative adversarial training is suspected to greatly reduce the model's requirements for the size of training data.

[0082] This section introduces the Extract-Select model proposed in this invention in detail. Figure 2 , Extract-Select consists of two main parts: an extractor and a discriminator. Specifically, the extractor adopts a mixed selection strategy of text segments to extract the entity text segment candidate set C based on the input sequence S and entity type y, and represents it as p C , that is (C, p C ) = f E (y, S). Then, P C It is input into the discriminator to evaluate the correctness of the candidate set C and give a score, that is, score = f D (y,S,p C ). After the extractor and discriminator have been iteratively trained (that is, the process of learning how to select the correct entity paragraph), the extractor will select the final result (a set of entity paragraphs), that is

[0083] For the training process, the extractor is trained through multi-task learning. Specifically, the first task is to train the extractor to correctly predict the start and end indexes of the entity paragraph; the second task is to train the extractor to better match the start-end index; the third task is to jointly train the extractor and the discriminator through generative adversarial training, where the extractor is regarded as a generator and is trained to obtain higher scores from the discriminator. After iterative training, the present invention uses the extractor to select the final result of the entity paragraph.

[0084] Given an entity type y and an input sequence The goal of the extractor is to extract entity-specific tags y from S* The entity paragraph candidate set C = {C1, C2, ..., C i}. At the same time, the extractor needs to calculate a continuous latent variable p C To represent C.

[0085] (1) Input sequence representation: The present invention firstly transforms all word elements of the input sequence S into Represented as an embedded sequence Each embedding in this sequence represents w i It is the concatenation of the character vector, word vector, context word vector, and part of speech (POS) vector. The character vector is generated by inputting the word unit representation into a bidirectional long short-term memory network (Bi-LSTM); for the context word vector, the present invention generates a context-dependent context embedding for each word unit. Afterwards, the concatenation of these four vectors is input into another Bi-LSTM network to obtain the final word unit representation:

[0086]

[0087] Among them, the output h i is the hidden state vector, encoding the word x i Rich contextual information.

[0088] (2) Entity tag representation: In the present invention, entity type is an important information. As the input of the extractor, it can provide the extractor with prior knowledge and guide the extractor's extraction. Therefore, the present invention proposes to design an entity tag for each entity type, where the entity tag is a more fine-grained representation of the entity type. The entity tag will serve as an additional input feature of the extractor. The present invention experimented with several different types of entity tags, and finally chose to use the connection between the keywords and synonyms of the entity type as the final entity tag. Specifically, the keyword is a word that describes the entity type. For example, the entity tag of the type "ORG" is "organization"; a synonym refers to an entity tag that is a word or phrase extracted from the Oxford Dictionary that is exactly or approximately the same as the keyword. For example, the entity tag of the entity type "ORG" is "institutionbody group company firm business corporation".

[0089] The present invention connects the word vectors of keywords and synonyms to obtain the entity tag y * The representation vector This vector is then input into a bidirectional long short-term memory network (Bi-LSTM) to obtain the final representation of the entity tag. where |y * | is the length of the entity tag. Then,

[0090] The present invention uses a self-attention mechanism to integrate the information of entity tags:

[0091]

[0092]

[0093] where α j is u j The attention weight, W a are learnable parameters.

[0094] (3) Extraction of candidate entity paragraphs: There are currently two methods that can be used to extract paragraphs from text. The first method focuses more on the boundary information of the paragraph, and uses two multi-class classifiers to predict the start and end indexes of the paragraph respectively; the second method considers the content of the paragraph, and uses n binary classifiers to predict whether each word is in the paragraph. However, although these two methods can obtain candidate paragraphs, they are still not suitable for the extractor of the present invention. The reasons are as follows: the first method uses a SoftMax multi-classifier in the entire context, and can only predict a single paragraph, which is not suitable for finding nested entities; these two methods regard the identified paragraphs as discrete variables, which cannot be used as the input of the subsequent discriminator and further optimized by the discriminator.

[0095] In view of this, the present invention proposes a hybrid selection strategy, which is a combination of these two methods, and considers both the boundary information of the paragraph and the content of the paragraph to generate entity paragraph candidates. The hybrid selection strategy first predicts the probability that each word is the start and end index of the entity paragraph, and then uses the probability to generate the entity paragraph candidate set and calculates the representation of the candidate set (this representation is a continuous latent variable). Therefore, in the generative adversarial training process, the extractor can be trained by back propagation.

[0096] The hybrid selection strategy proceeds as follows. Given the representation vectors of the sequence and entity tags, the extractor first predicts the probability that token i is the start index and end index of the entity context:

[0097]

[0098]

[0099] Among them, W s and W e are the parameters to be learned.

[0100] In the input sequence S, there may be multiple entities of the same category. This means that multiple start and end indexes may be predicted. Therefore, the traditional method of simply matching the start index with its nearest end index is not suitable here. In order to match the start index with its corresponding end index, the present invention first obtains the index that may be at the start or end:

[0101] I s ={i|argmax(p s (i|y,S))=1,i=1,...,n},

[0102] I e ={j|argmax(p e (j|y, S))=1, j=1,...,n}. (6)

[0103] Then, for each given starting index i s ∈I s and end index j e ∈I e (i s ≤j e ), the present invention uses a binary classifier to predict entity segment candidates Probability:

[0104]

[0105] Where W C is a parameter to be learned. Entity paragraph candidates are extracted by a probability threshold, and all possible entity paragraph candidates constitute the entity paragraph candidate set C. Obviously, this strategy does not set a maximum paragraph length when extracting entity paragraphs. Therefore, long entities can also be identified by the method of the present invention.

[0106] Then, in order to represent the entity text candidate set C, the present invention first calculates p C (i|y, S) is the probability that the i-th word appears in the set C. This probability can also be regarded as the probability that the i-th word is within the boundary of the entity paragraph of type y. Specifically, p c (i|y, S) can be used with p s and p e The calculation results are:

[0107]

[0108] Among them, p C (i|y, S) also means the frequency of the i-th word in the candidate set C. In other words, the more frequently the i-th word appears in C, the more likely p C (i|y, S) will be larger.

[0109] In this way, the present invention can use the continuous vector p C To represent C, and input it to the subsequent discriminator. Thus, the gradient can be back-propagated from the discriminator to the extractor during the generative adversarial training process.

[0110] In the extractor, based on the input sequence S and entity tags y * After obtaining the entity text candidate set C, the discriminator aims to evaluate C and give a score to C. In order to better evaluate, the present invention first uses p C To represent the entity text candidate set C.

[0111] (1) Candidate text set representation: The present invention proposes encoding entity type y and sequence S into p C Specifically, the present invention uses Match-LSTM network to construct entity-aware sequence representation. Match-LSTM was originally proposed to enhance text representation in machine reading comprehension. Here, the present invention uses entity tagging to enhance text representation:

[0112]

[0113] Then the entity paragraph candidate set is represented by weighted summation using r i and probability p C (i|y, S) calculation:

[0114]

[0115]

[0116] (2) Candidate set scoring: The scores of the entity segment candidate sets extracted by the extractor can be calculated using the sigmoid function:

[0117] score=f D (p C |y, S) = sigmoid(W b r C ), (11)

[0118] Among them, W b is a learnable parameter. The score f D It will be used to iteratively train the extractor and discriminator in the generative adversarial training process.

[0119] In the training process, the present invention trains the extractor and the discriminator through multi-task learning, and trains the extractor and the discriminator together through generative adversarial training. The algorithm of the training process is shown in the algorithm in Table 1.

[0120] Table 1 Extract-Select model training algorithm

[0121]

[0122] The present invention introduces in detail the training process of the extractor and the discriminator as follows:

[0123] The first task is to minimize the negative log probability of the true start and end index of the golden entity segment:

[0124] l′ E =-logp s (s|y,S)-logp e (e|y,S), (12)

[0125] Where s and e represent the start and end indexes of the golden entity segment in sequence S.

[0126] The second task is to minimize the negative log probability of the true start-end index match of the golden entity segment:

[0127] l″ E =-logp s，e . (13)

[0128] The third task is to train the extractor and the discriminator together using generative adversarial training. Specifically, the present invention trains the extractor to get a higher score from the discriminator:

[0129] l″′ E =log(1f D (y,S)). (14)

[0130] At the same time, the training discriminator of the present invention is as follows: The present invention minimizes log(1-f D (y, S)) and maximize log f D (y, S):

[0131] l D =zlogf D (y, S)+(1-z)log(1-f D (y, S)), (15)

[0132] Where z∈{1,0} indicates whether the golden entity appears in the candidate set of entity paragraphs.

[0133] The overall training objective function of the extractor is:

[0134] l E =γ1l′ E +γ2l″ E +(1-γl -γ2)l″′ E , (16)

[0135] Among them, γ1, γ2∈[0, 1] are hyperparameters that balance the contribution of each part to the overall objective function. The three losses are jointly trained in an end-to-end manner.

[0136] In the reasoning phase, given a sequence S and an entity tag y * , the trained extractors are based on I s and I e The start and end indexes are selected, and then the start and end indexes are matched to obtain entity text segments of a specific entity category.

[0137] The present invention experiments the Extract-Select model on standard nested named entity recognition datasets, including ACE04, ACE05, KBP17 and GENIA. The experimental results show that the Extract-Select model of the present invention can effectively detect nested named entities, and achieves the best results on all four datasets. In addition, the present invention also selects two non-nested named entity recognition datasets for experiments, including CoNLL2003 and Weibo datasets, and achieves the best results on both datasets.

[0138] The beneficial effects of the present invention are as follows:

[0139] The present invention proposes a novel Extract-Select model to solve the problem of nested named entity recognition. The Extract-Select model adopts a text selection framework, which can use an extractor to extract all possible entity texts for each entity category, and use a discriminator to evaluate and score the extracted texts, which can naturally solve the problem of nested named entity recognition. The extractor and the discriminator are trained together through generative adversarial training, so that the extractor can get higher scores from the discriminator. In the reasoning stage, the trained extractor is used to select entity texts specific to each entity type, that is, named entities. The Extract-Select model of the present invention reduces the dependence on training data, so that the model can achieve relatively good results even when there is little training data.

[0140] The present invention proposes a hybrid selection strategy to extract entity paragraph candidates and represent them as a continuous latent variable. This hybrid selection strategy uses both paragraph boundary information and paragraph content information for extraction. Through this strategy, the representation of the entity paragraph candidate set can be propagated to the discriminator and scored. The extractor and the discriminator can be trained and optimized together in the generative adversarial training process.

[0141] The present invention makes full use of entity category information and designs entity tags for each entity category. By considering the named entity recognition task as a text selection task for each entity, the Extract-Select model of the present invention uses entity tags as additional inputs of the extractor to guide the extractor to extract more accurate results. In addition, when the discriminator evaluates the results extracted by the extractor, the entity tags are also used as additional inputs to enhance the representation of the entity text candidate set.

[0142] The present invention conducts experiments on four widely used nested named entity recognition datasets and compares the Extract-Select model with a series of state-of-the-art models. The experimental results show that the Extract-Select model consistently outperforms the state-of-the-art models on all four datasets.

[0143] As used herein, the word "preferred" is intended to be used as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0144] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0145] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

[0146] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A nested named entity recognition method based on generative adversarial training, characterized in that: The following steps are involved: Get the input sequence; Design an entity tag for each entity type, wherein the entity tag contains prior information about the entity type; Use the extractor to extract all entity texts specific to a certain entity tag to form an entity text candidate set; Evaluate the extracted entity paragraph candidate set through an evaluator, and give a score to the entity paragraph candidate set; The extractor and the discriminator are iteratively trained by generative adversarial training, so that the extractor extracts an accurate set of entity paragraph candidates; After iterative training, the extractor selects all entity segments of a certain entity type; Extract and select each entity type until all nested and non-nested entities in the input sequence are extracted, and the fragments and semantic categories of all nested and non-nested entities are output; The extractor is used to extract all entity text segments specific to a certain entity tag, including: input sequence representation; entity tag representation and candidate entity text segment extraction; The candidate entity paragraph extraction uses a hybrid selection strategy, which first predicts the probability that each word is the start and end index of the entity paragraph, then uses the probability to generate an entity paragraph candidate set and calculates the representation of the candidate set; The hybrid selection strategy specifically includes: Given the representation vectors of the sequence and entity tokens, the extractor first predicts the probability that token i is the start index and end index of the entity context: Among them, W s and W e is the parameter to be learned, h i is the hidden state vector of the ith word, h k is the hidden state vector of the kth word, and m is the representation of the entity tag; Get possible start or end index: I s =(i|argmax(p s (i|y,S))=1,i=1,...,n} I e ={j|argmax(p e (j|y,S))=1,j=1,…,n} Then, for each given starting index i s ∈I s and end index j e ∈I e , and i s ≤j e , using a binary classifier to predict entity segment candidates Probability: Where W C are the parameters to be learned, is the hidden state vector of the starting word, is the hidden state vector of the ending word; Entity paragraph candidates are extracted through a probability threshold, and all possible entity paragraph candidates constitute the entity paragraph candidate set C; Calculate p c (i|y, S) is the probability that the i-th word appears in the candidate set C. In this way, the continuous vector p c To represent C.

2. The nested named entity recognition method based on generative adversarial training according to claim 1 is characterized in that: The input sequence representation includes: Represent all word-grams of the input sequence S as an embedding sequence, where each embedding representation in the embedding sequence is a concatenation of a character vector, a word vector, a context word vector, and a part-of-speech vector, wherein the character vector is generated by inputting the word-gram representation into the first Bi-LSTM network; the context word vector is obtained by generating a context-dependent context embedding for each word-gram; The connection of the character vector, word vector, context word vector and part-of-speech vector is input into the second Bi-LSTM network to obtain the final word unit representation.

3. The nested named entity recognition method based on generative adversarial training according to claim 1, characterized in that: The entity tag is a fine-grained representation of an entity type. Each entity type corresponds to an entity tag. The entity tag representation includes: Use the connection between the keyword and synonym of the entity type as the entity tag, the keyword is a word describing the entity type, and the synonym refers to a word or phrase extracted from the Oxford Dictionary with the same meaning as the keyword; Connect the word vectors of keywords and synonyms to get the entity tag y * The representation vector is then input into the second Bi-LSTM network to obtain the final representation of the entity tag: Among them, h i is the hidden state vector, encoding the word x i context information; The final representation of the entity tags is integrated using a self-attention mechanism: Among them, α j is u j The attention weight, W a is a learnable parameter and m is the final representation of the entity tag.

4. The nested named entity recognition method based on generative adversarial training according to claim 1, characterized in that: The evaluation of the discriminator includes the candidate text set representation and candidate set scoring: The candidate text set represents encoding the entity type y and the sequence S into the entity text candidate set representation p c , specifically including the following steps: Use Match-LSTM network to build entity-aware sequence representation: Among them, h i is the hidden state vector, encoding the word x i Rich contextual information, is the final representation of the entity tag, |y * | is the length of the entity tag, |S| represents the length of the sequence; The representation r of the candidate set of entity paragraphs C By weighted summation with r i and probability p C (i|y, S) calculation: where r C is the representation of the candidate set of entity paragraphs, β i is the weight coefficient, r k is the representation of the kth word in the sequence S, p C (i|y, S) is the probability that the i-th word appears in the candidate set C, The candidate set scoring includes: the scores of the entity segment candidate set extracted by the extractor are calculated by the sigmoid function: score=sigmoid(W b r C ) Among them, W b is a learnable parameter that is used to iteratively train the extractor and discriminator during generative adversarial training.

5. The nested named entity recognition method based on generative adversarial training according to claim 1, characterized in that: The extractor and the discriminator are iteratively trained by generative adversarial training, including three tasks, wherein the first task is to train the extractor to correctly predict the start and end indexes of the entity paragraph; the second task is to train the extractor to better match the start-end index; the third task is to jointly train the extractor and the discriminator by generative adversarial training, wherein the extractor is regarded as a generator and is trained to obtain a higher score from the discriminator; After iterative training, the extractor is used to select the final result of the entity segment.

6. The nested named entity recognition method based on generative adversarial training according to claim 5, characterized in that: The first task is to minimize the negative logarithmic probability of the true start and end indexes of the golden entity text segment, and the loss function is as follows: l′ E =-logp s (s|y,S)-logp e (e|y,S) Where s and e represent the start and end indexes of the golden entity text in sequence S, p s (s|y, S) is the probability of the true start index of the golden entity segment, p e (s|y, S) is the probability of the true end index of the golden entity segment; The second task is to minimize the negative logarithmic probability of the true start-end index match of the golden entity segment, and the loss function is as follows: l″ E =-logp s，e where p s，e is the probability of a true start-end index match of the golden entity segment; The third task is to train the extractor to obtain a higher score from the discriminator, and the loss function is as follows: l″′ E =log(1-f D (y,S)) where f D (y, S) is the score of the entity paragraph candidate set extracted by the extractor; The overall training objective function of the extractor is: l E =γ1l′ E +γ2l″ E +(1-γ1-γ2)l″′ E Where γ1, γ2∈[0, 1] are hyperparameters that balance the contribution of each part to the overall objective function. The three loss functions are jointly trained in an end-to-end manner; Where γ1, γ2∈[0, 1] are hyperparameters that balance the contribution of each part to the overall objective function. The three loss functions are jointly trained in an end-to-end manner; The final result of using the extractor to select entity text segments includes: in the reasoning stage, given the sequence S and the entity tag y * , the trained extractors are based on I s and I e The start and end indexes are selected; then the start and end indexes are matched to obtain entity text segments of a specific entity category.

Images