A sample named entity recognition method based on a pointer network and a prototype network

Abstract

This invention discloses a sample named entity recognition method based on pointer networks and prototype networks. The steps include: S1 obtaining a publicly available named entity recognition dataset; S2 dividing the dataset into support sets and query sets, with each sample in the divided dataset consisting of both support and query sets; S3 constructing a decompositional model combining pointer networks and prototype networks, and saving the parameters of the model with the best performance; S4 manually annotating samples of each type of entity in the target domain, training the model using the annotated data, and using the trained model to perform named entity recognition on text in the target domain. This invention divides traditional pattern named entity recognition into two parts: entity range detection and entity classification. It uses pointer networks to recall entities in the text and prototype networks to perform entity classification, thus solving the limitation that entity annotation can only be based on characters and avoiding interference from non-entity characters during entity classification.

Description

Technical Field

[0001] This invention relates to the field of natural language processing information extraction technology, specifically to a sample named entity recognition method based on pointer networks and prototype networks. Background Technology

[0002] Named entity recognition (NER), a fundamental task in natural language processing, aims to identify entities in text and classify them into predefined types. In recent years, various deep learning-based methods have been applied to NER tasks, achieving remarkable results. However, deep learning-based methods often rely on large-scale labeled data, requiring substantial amounts of manually labeled training sets. Furthermore, traditional deep neural network models use CRFs or classifiers as decoders, resulting in a high degree of coupling between entity categories and the model. When applying the model to new domains, it requires retraining the classifier with a large amount of data.

[0003] The shortcomings of existing technology:

[0004] Currently, metric learning can reduce the coupling between the model and entity categories. When the model is applied to a new domain, it can be trained with a small amount of data. However, traditional metric learning methods are based on character sequence labeling of text, which cannot explicitly utilize the semantic information of words and are subject to a large amount of interference from non-entity categories. Summary of the Invention

[0005] The purpose of this invention is to provide a sample named entity recognition method based on pointer networks and prototype networks to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a sample named entity recognition method based on pointer networks and prototype networks, which specifically includes the following steps:

[0007] S1. Obtain a publicly available named entity recognition dataset;

[0008] S2. Divide the dataset into a support set and a query set. Each sample in the divided dataset consists of the support set and the query set.

[0009] S3. Construct a decomposition model that combines pointer network and prototype network. Train the decomposition model using the obtained dataset. After each training round, use the validation set to verify the model's performance and save the parameters of the model with the best performance.

[0010] S4. Select samples of each type of entity in the target domain and manually label them. Use the labeled data to train the model and use the trained model to perform named entity recognition on the text in the target domain.

[0011] Preferably, in step S1, the entity categories are constructed, entities in the domain text are labeled using a labeling tool, and labeling results in JavaScript object representation format or comma-separated value file format are obtained.

[0012] Preferably, the annotation tool is a quick annotation tool or a tag studio.

[0013] Preferably, step S2 specifically includes the following steps:

[0014] a1. Dividing the support set and query set: For the named entity recognition dataset D, ∈=(S,Q)∈D is a sample in the dataset, S and Q represent the support set and query set respectively, and ∈ follows the N way K~2K shot setting. N way means that the support set and query set each contain N types of entities, and K~2K shot means that there are K~2K instances of each type of entity.

[0015] a2. Construct each sample in the dataset in sequence, and build the support set and query set for each sample;

[0016] a3. For support sets where the number of entity categories does not reach N, add text containing new entity types to the support set where the number of entity categories does not reach N, and the newly added text will not violate the K~2K shot constraint. The query set does not need to satisfy the K~2K shot constraint.

[0017] Preferably, step S3 specifically includes the following steps:

[0018] b1. Using a model-independent meta-learning method, a training session is divided into two parts: an inner loop and an outer loop. An outer loop contains multiple inner loops, and each inner loop corresponds to a sample in the dataset.

[0019] b2. At the beginning of each inner loop, obtain the mirror image of the model, train the mirror image of the model using the support set data, use the trained mirror image to perform entity recognition on the query set, and obtain the loss through the cross-entropy function. The gradient of the loss is obtained by differentiating the parameters.

[0020] b3. In the outer loop, average all gradients obtained in the inner loop to update the initial model parameters.

[0021] Preferably, the model in step S3 includes an entity range detection module and an entity classification module. For the input text, a pointer network is used to generate the position of the entity in the text, the hidden vector of the feature vector of the corresponding character is extracted, the feature vector of the entity is obtained by averaging the feature vectors of each character of the entity, and the prototype network is used for entity classification.

[0022] Preferably, the main steps for entity range detection using pointer networks include:

[0023] c1. Given an input text X = {x1, x2, ..., x...} n The encoder Encoder(·) encodes X into a representation vector H in the hidden space. en :

[0024] H en =Encoder(X)

[0025] in, d is the dimension of the hidden vector; the decoder (Decoder(·)) uses the encoder's encoded result H. en The entity positions i1, i2, ..., i output by the decoder t-1 As input, decode i t , Indicates the position of the named entity in the sentence, 0≤i m ≤n;

[0026] c2, through To retrieve the character at the corresponding position, Map(·) is a mapping from character position to character. It is position i m characters, This represents the decoded entity sequence, and i is calculated using the following formula. t The probability at:

[0027] h t =Decoder(H en ;x)

[0028] q t =W q h t

[0029] K = W k H en

[0030]

[0031] in, W is the hidden vector obtained from decoding. q , This is the mapping matrix for the attention mechanism, and Softmax(·) is the activation function. Let represent the attention of decoder position t to the encoded sequence X, which can be viewed as the probability distribution of each character position in the encoded sequence. During the training phase, the cross-entropy function CrossEntropy(·) is used to measure the probability distribution p of the position. t With the correct position i t Losses between:

[0032]

[0033] c3. In the prediction phase, the trained model is used to sequentially generate the positions of entities in the sentence, and the beam search algorithm is used to decode the generated entity position sequence.

[0034] Preferably, the main steps for entity classification using a prototype network include:

[0035] d1. Given an input text X = {x1, x2, ..., x...} n The encoder Encoder(·) is used to compute the contextual semantic representation H = {h1, h2, ..., h} of the input text. n}:

[0036] H = Encoder(X)

[0037] Given data ∈ = (S, Q), for entity type t ∈ T, Represents a set of entity types, and constructs a prototype using the semantic representations of entities of that type in the support set S. This indicates that the training set contains L instances, X u Let the character sequence of the u-th instance be... The semantic representation of entity type t in the u-th instance can be calculated using the following formula:

[0038]

[0039] d2. Construct a prototype for each type of entity in the query set. Based on the type prototype, classify the entities in the query set, and let w [j,k] This represents the entity obtained during the entity range detection phase, with its starting position being i. j The ending position is i k The semantic representation of this entity is achieved through... The encoding result is obtained as follows:

[0040]

[0041] The probability that the entity is of type t is:

[0042] p(t|e [j,k] ) = Softmax(-d(c t ,e [j,k] ))

[0043] Where d(·,·) denotes the distance function, let t [j,k] ∈T represents e [j,k] The actual type, and the cross-entropy loss function are as follows:

[0044]

[0045] Preferably, in step S3, saving the model parameters involves: inserting a low-rank adapter into the model and adjusting the parameters of the inserted part relative to the projection matrix in the model. Decomposed into the form of the product of two low-rank matrices:

[0046] W+ΔW=W+BA

[0047] in, And r << min(d,k), W does not update its parameters during training, while A and B contain trainable parameters. For input vector x, the output vector h = Wx is redefined as:

[0048] h=(W+ΔW)x=Wx+BAx

[0049] At the beginning of training, matrix A is initialized with random Gaussian initialization, and the values ​​of matrix B are all zero. Therefore, ΔW = BA is zero. During training, the updates to the parameters of matrices A and B can be viewed as updates to the projection matrix W. The LoRA structure is inserted into the feedforward neural network or attention layer. In the attention layer, this includes querying the mapping matrix W. q Key mapping matrix W k Value mapping matrix W v and output mapping matrix W o In the feedforward neural network layer, this includes the input mapping matrix W. i and output mapping matrix W o .

[0050] Preferably, the beam search is performed as follows: during the generation of the target sequence, the beam search algorithm outputs the current position value for each candidate sequence and calculates the overall probability of the candidate sequence. The k candidate sequences are sorted according to their probabilities. Some sequences output termination symbols in advance, while others terminate after reaching the pre-set maximum output sequence length. After the termination condition is met, the candidate sequence with the highest probability is selected as the final solution.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] This invention discloses a sample named entity recognition method based on pointer networks and prototype networks. The steps include: S1, obtaining a publicly available named entity recognition dataset; S2, dividing the dataset into support and query sets, with each sample in the divided dataset consisting of both support and query sets; S3, constructing a decompositional model combining pointer and prototype networks, training the decompositional model using the obtained dataset, validating the model's performance using a validation set after each training round, and saving the parameters of the model with the best performance; S4, manually labeling samples of each entity class in the target domain, training the model using the labeled data, and using the trained model to perform named entity recognition on text in the target domain. This invention divides traditional pattern named entity recognition into two parts: entity range detection and entity classification. It uses a pointer network to recall entities in the text and a prototype network for entity classification, explicitly utilizing the semantic information of words. This solves the limitation that entity labeling can only be based on characters and avoids interference from non-entity characters during entity classification. Attached Figure Description

[0053] Figure 1 The method flowchart provided by the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] In the description of this invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0056] In the description of this patent, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "setting" should be interpreted broadly. For example, they can refer to a fixed connection or setting, a detachable connection or setting, or an integrated connection or setting. Those skilled in the art can understand the specific meaning of the above terms in this patent according to the specific circumstances.

[0057] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a number" means two or more, unless otherwise explicitly specified.

[0058] Example

[0059] Please see Figure 1 As shown, this invention provides a sample named entity recognition method based on pointer networks and prototype networks. The method specifically includes the following steps:

[0060] S1. Obtain a publicly available named entity recognition dataset, construct entity categories, use annotation tools to annotate entities in the domain text, and obtain annotation results in JavaScript object representation format or comma-separated value file format.

[0061] S2. Divide the dataset into a support set and a query set. Each sample in the divided dataset consists of the support set and the query set.

[0062] a1. Dividing the support set and query set: For the named entity recognition dataset D, ∈=(S,Q)∈D is a sample in the dataset, S and Q represent the support set and query set respectively, and ∈ follows the N way K~2K shot setting. N way means that the support set and query set each contain N types of entities, and K~2K shot means that there are K~2K instances of each type of entity.

[0063] a2. Construct each sample in the dataset in sequence, and build the support set and query set for each sample;

[0064] a3. For support sets where the number of entity categories does not reach N, add text containing new entity types.

[0065] For support sets where the number of entity categories does not reach N, and newly added text does not violate the K~2K shot constraint, the query set does not need to satisfy the K~2K shot constraint;

[0066] S3. Construct a decomposition model that combines pointer network and prototype network. Train the decomposition model using the obtained dataset. After each training round, use the validation set to verify the model's performance and save the parameters of the model with the best performance.

[0067] b1. Using a model-independent meta-learning method, a training session is divided into two parts: an inner loop and an outer loop. An outer loop contains multiple inner loops, and each inner loop corresponds to a sample in the dataset.

[0068] b2. At the beginning of each inner loop, obtain the mirror image of the model, train the mirror image of the model using the support set data, use the trained mirror image to perform entity recognition on the query set, and obtain the loss through the cross-entropy function. The gradient of the loss is obtained by differentiating the parameters.

[0069] b3. In the outer loop, average all gradients obtained in the inner loop to update the initial model parameters;

[0070] S4. Select samples of each type of entity in the target domain and manually label them. Use the labeled data to train the model and use the trained model to perform named entity recognition on the text in the target domain.

[0071] Furthermore, the model includes an entity range detection module and an entity classification module. For the input text, a pointer network is used to generate the position of the entity in the text, and the hidden vectors of the corresponding characters are extracted. By averaging the feature vectors of each character of the entity, the entity's feature vector is obtained, and a prototype network is used for entity classification. The main steps of entity range detection using a pointer network include:

[0072] c1. Given an input text X = {x1, x2, ..., x...} n The encoder Encoder(·) encodes X into a representation vector H in the hidden space. en :

[0073] H en =Encoder(X)

[0074] in, d is the dimension of the hidden vector; the decoder (Decoder(·)) uses the encoder's encoded result H. en The entity positions i1, i2, ..., i output by the decoder t-1 As input, decode i t , Indicates the position of the named entity in the sentence, 0≤i m ≤n;

[0075] c2, through To retrieve the character at the corresponding position, Map(·) is a mapping from character position to character. It is position i m characters, This represents the decoded entity sequence, and i is calculated using the following formula. t The probability at:

[0076] h t =Decoder(H en ;x)

[0077] q t =W q h t

[0078] K = W k H en

[0079]

[0080] in, W is the hidden vector obtained from decoding. q , This is the mapping matrix for the attention mechanism, and Softmax(·) is the activation function. Let represent the attention of decoder position t to the encoded sequence X, which can be viewed as the probability distribution of each character position in the encoded sequence. During the training phase, the cross-entropy function CrossEntropy(·) is used to measure the probability distribution p of the position. t With the correct position i t Losses between:

[0081]

[0082] On the one hand, the number of entities generated by the pointer network is only related to the number of entities in the training data, and the imbalance between the number of entities and non-entities does not affect its prediction results; on the other hand, the pointer network does not pay attention to boundary information, thus avoiding the problem of overlapping entity ranges caused by the intersection of the start and end boundaries of different entities.

[0083] c3. In the prediction phase, the trained model is used to generate the positions of entities in the sentence sequentially. The beam search algorithm is used for decoding to improve the overall effect of the generated entity position sequence. The beam search algorithm outputs the current position value for each candidate sequence and calculates the overall probability of the candidate sequence during the generation of the target sequence. The k candidate sequences are sorted according to their probability. Some sequences output the termination symbol in advance, and the other part terminates after reaching the pre-set maximum output sequence length. After the termination condition is met, the candidate sequence with the highest probability is selected as the final solution.

[0084] Furthermore, the main steps for entity classification using a prototype network include:

[0085] d1. Given an input text X = {x1, x2, ..., x...} nThe encoder Encoder(·) is used to compute the contextual semantic representation H = {h1, h2, ..., h} of the input text. n}:

[0086] H = Encoder(X)

[0087] Given data ∈ = (S, Q), for entity type t ∈ T, Represents a set of entity types, and constructs a prototype using the semantic representations of entities of that type in the support set S. This indicates that the training set contains L instances, X u Let the character sequence of the u-th instance be... The semantic representation of entity type t in the u-th instance can be calculated using the following formula:

[0088]

[0089] d2. Construct a prototype for each type of entity in the query set. Based on the type prototype, classify the entities in the query set, and let e [j,k] This represents the entity obtained during the entity range detection phase, with its starting position being i. j The ending position is i k The semantic representation of this entity is achieved through... The encoding result is obtained as follows:

[0090]

[0091] The probability that the entity is of type t is:

[0092] p(t|e [j,k] ) = Softmax(-d(c t ,e [j,k] ))

[0093] Where d(·,·) denotes the distance function, let t [j,k] ∈T represents e [j,k] The actual type, and the cross-entropy loss function are as follows:

[0094]

[0095] Furthermore, in step S3, when saving the model parameters: the LoRA component is inserted into the model, and the parameters of the inserted part are adjusted relative to the projection matrix in the model. Decomposed into the form of the product of two low-rank matrices:

[0096] W+ΔW=W+BA

[0097] in, And r << min(d,k), W does not update its parameters during training, while A and B contain trainable parameters. For input vector x, the output vector h = Wx is redefined as:

[0098] h=(W+ΔW)x=Wx+BAx

[0099] At the beginning of training, matrix A is initialized with random Gaussian initialization, and the values ​​of matrix B are all zero. Therefore, ΔW = BA is zero. During training, the updates to the parameters of matrices A and B can be viewed as updates to the projection matrix W. The LoRA structure is inserted into the feedforward neural network or attention layer. In the attention layer, this includes querying the mapping matrix W. q Key mapping matrix W k Value mapping matrix W v and output mapping matrix W o In the feedforward neural network layer, this includes the input mapping matrix W. i and output mapping matrix W o Using the LoRA method, model parameters can be fixed, avoiding the catastrophic forgetting problem; and only the parameters of the LoRA component need to be stored for each target domain, while the parameters of the main model only need to be stored once, saving storage space.

[0100] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for sample named entity recognition based on pointer network and prototype network, characterized in that: This method specifically includes the following steps: S1. Obtain a publicly available named entity recognition dataset; S2. Divide the dataset into a support set and a query set. Each sample in the divided dataset consists of the support set and the query set. S3. Construct a decomposition model that combines pointer networks and prototype networks. Train the decomposition model using the obtained dataset. After each training round, use the validation set to verify the model's performance and save the parameters of the model with the best performance. S4. Select samples of each type of entity in the target domain and manually label them. Use the labeled data to train the model and use the trained model to perform named entity recognition on the text in the target domain. In step S3, the model includes an entity range detection module and an entity classification module. For the input text, a pointer network is used to generate the position of the entity in the text, and the hidden vector of the feature vector of the corresponding character is extracted. By averaging the feature vectors of each character of the entity, the feature vector of the entity is obtained, and the prototype network is used for entity classification. The main steps for entity range detection using pointer networks include: c1. Given an input text Using an encoder Will Encoded as a representation vector in the hidden space : in, , It is the dimension of the hidden vector; decoder The encoding result of the encoder and the entity location output by the decoder As input, decode , Indicates the position of the named entity in the sentence. ; c2, through Get the character at the corresponding position. It is a mapping from character position to character. The position is characters, The decoded entity sequence is calculated using the following formula. The probability at: in, It is the hidden vector obtained from decoding. It is a mapping matrix for the attention mechanism. For activation function, Indicates decoder position For the encoded sequence The attention is treated as the probability distribution of each character position in the encoded sequence. During the training phase, the cross-entropy function is used. Probability distribution for measuring location With the correct position Losses between: c3. In the prediction stage, the trained model is used to sequentially generate the positions of entities in the sentence, and the beam search algorithm is used to decode the generated entity position sequence. The main steps for entity classification using prototype networks include: d1. Given an input text Using an encoder Calculate the contextual semantic representation of the input text : Given data For entity types , Represents a collection of entity types, utilizing support sets. The semantic representation of this type of entity is used to construct a prototype. This indicates that the training set contains One example, Indicates the first The character sequence of each instance, let Indicates the first In each instance, the entity type is The semantic representation of can be calculated using the following formula: d2. Construct a prototype for each type of entity in the query set. Based on the type prototype, classify the entities in the query set, and let... This represents the entity obtained during the entity range detection phase, with its starting position being... The end position is The semantic representation of this entity is achieved through... The encoding result is obtained as follows: The type of this entity is The probability is: in, Let the distance function be denoted by . express The actual type, and the cross-entropy loss function are as follows: 。 2. The sample named entity recognition method based on pointer networks and prototype networks according to claim 1, characterized in that: In step S1, entity categories are constructed, entities in the domain text are labeled using annotation tools, and annotation results in JavaScript object representation format or comma-separated value file format are obtained.

3. The sample named entity recognition method based on pointer networks and prototype networks according to claim 2, characterized in that: The annotation tool is either a quick annotation tool or a tag studio.

4. The sample named entity recognition method based on pointer networks and prototype networks according to claim 1, characterized in that: Step S2 specifically includes the following steps: a1. Dividing the support set and query set based on the named entity recognition dataset. , It is a sample in the dataset. and These represent the support set and the query set, respectively. follow set up, This indicates that the support set and query set respectively contain Entity class This indicates that each type of entity exists. One instance; a2. Construct the support set for each sample in the dataset sequentially. Query set; a3. For entities where the number of categories has not reached the required threshold The support set will include new types of entities. The text addition did not meet the requirement that the number of entity categories was not sufficient. The support set, and newly added text will not break. The constraints do not need to be satisfied by the query set. constraint.

5. The sample named entity recognition method based on pointer networks and prototype networks according to claim 1, characterized in that: Step S3 specifically includes the following steps: b1. Using a model-independent meta-learning method, a training session is divided into two parts: an inner loop and an outer loop. An outer loop contains multiple inner loops, and each inner loop corresponds to a sample in the dataset. b2. At the beginning of each inner loop, obtain the mirror image of the model, train the mirror image of the model using the support set data, use the trained mirror image to perform entity recognition on the query set, and obtain the loss through the cross-entropy function. The gradient of the loss is obtained by differentiating the parameters. b3. In the outer loop, average all gradients obtained in the inner loop to update the initial model parameters.

6. The sample named entity recognition method based on pointer networks and prototype networks according to claim 1, characterized in that: In step S3, which saves the model parameters, the low-rank adapter is inserted into the model, and the parameters of the inserted part are adjusted relative to the projection matrix in the model. It can be decomposed into the form of the product of two low-rank matrices: in, , ,and During the training process Without updating the parameters, and It contains trainable parameters for the input vector. Output vector Redefining: At the beginning of training, the matrix Initialize the matrix using random Gaussian initialization. If all values ​​are zero, then The value is zero during the training process. and The parameter update can be viewed as an adjustment to the projection matrix. Updates; LoRA structures are inserted into feedforward neural networks or attention layers, where the query mapping matrix is ​​included in the attention layer. Key mapping matrix Value mapping matrix and output mapping matrix In the feedforward neural network layer, this includes the input mapping matrix. and output mapping matrix .

7. The sample named entity recognition method based on pointer networks and prototype networks according to claim 1, characterized in that: The beam search is as follows: During the generation of the target sequence, the beam search algorithm outputs the current position value for each candidate sequence and calculates the overall probability of the candidate sequence. The k candidate sequences are sorted according to their probability. Some sequences output termination symbols in advance, while others terminate after reaching the pre-set maximum output sequence length. After the termination condition is met, the candidate sequence with the highest probability is selected as the final solution.

Images