A named entity recognition method based on global pointer and adversarial training

Abstract

The application discloses a named entity recognition method based on a global pointer and adversarial training, which is composed of four parts: a BERT module, an attention mechanism module, a global pointer module and an adversarial training module. The method adopts BERT as a sentence encoder, generates adversarial samples by adding disturbance through adversarial training, and integrates relative position information when calculating attention scores, so that the position relationship between entities is enhanced, and finally decoding is carried out through the global pointer. The method can process the entity recognition problem of a data set with a large label scale, can recognize nested entities, can cover more entity recognition scenes, has globality, can pay attention to global information of entities, improves the robustness and generalization of the network model through adversarial training, and the performance on multiple public data sets is improved, and the method can utilize potential information in original text data and improve information richness.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to a named entity recognition method based on global pointers and adversarial training. Background Technology

[0002] Early research on Named Entity Recognition (NER) primarily relied on rule-based and dictionary-based methods. Rule-based NER methods depend on manually constructed rules, designed for specific domains or grammars. Therefore, their performance is closely related to the size of the domain-specific dictionary; they perform better when the domain-specific vocabulary is rich. Existing rule-based NER systems include SRA, FASTUS, and LTG. These systems mainly identify entities through manually designed rules. Because rules and dictionaries often differ across domains, systems built using this method can only be applied to a single domain and cannot be used across multiple domains.

[0003] With the development of machine learning and deep learning, a large number of named entity recognition methods have been proposed, including shallow machine learning methods and deep learning methods. Shallow machine learning methods, also known as statistical learning-based methods, mainly include Hidden Markov Model (HMM), Support Vector Machine (SVM), and Conditional Random Fields (CRF). However, these methods require manual annotation to extract features and use feature templates to interpret the features of the entity context so that the machine can understand the meaning of the entity context. Therefore, the selection of feature templates is very important, requires manual participation, and is highly dependent on the corpus. Deep learning methods use neural network models for named entity recognition, such as Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN) for entity recognition. In deep learning, named entity recognition tasks generally have two annotation methods: (1) Sequence labeling, which uses a tag to label each token in a sentence. The advantage of this method is that it is fast when there are few entity categories, but the computational cost is high when the number of entity categories is large. The disadvantage is that a token is labeled with only one tag. In reality, a token may belong to multiple entities or different entity types. Therefore, this method cannot solve the problem of nested entities. (2) Pointer labeling: This method only focuses on the head and tail positions of the entity. The advantage is that it can solve the problem of nested entities. The disadvantage is that each pointer network uses two modules to identify the head and tail of the entity respectively. A single module can only focus on the information of the head or tail of the entity and cannot pay attention to the global information of the entity.

[0004] The above entity recognition methods have the following disadvantages: (1) Sequence labeling methods use conditional random fields as label decoders, which have high computational costs when the number of labels is too large; and cannot solve the problem of nested entity recognition. (2) Pointer labeling methods use multiple pointer networks for multiple entity relationship categories. Each pointer network uses two modules to identify the beginning and end of the entity respectively. When training and evaluating a single module, it can only pay attention to the information of the head or tail of the entity and cannot pay attention to the global information of the entity. However, when predicting, it predicts according to the entire entity, which leads to inconsistency between training and prediction. (3) Neural network models have local instability, and the robustness and generalization of the model are poor. Summary of the Invention

[0005] The purpose of this invention is to provide a named entity recognition method based on global pointers and adversarial training. A global pointer is used as a label decoder to handle nested entity problems, and adversarial training is used to generate adversarial examples to enhance the robustness and generalization of the model. Based on the above conditions, this invention provides a named entity recognition method based on global pointers and adversarial training. This method consists of four parts: a ChineseBERT module, an attention mechanism module, a global pointer module, and an adversarial training module. This method uses ChineseBERT as a sentence encoder, uses adversarial training to add perturbations to generate adversarial examples, and incorporates relative positional information when calculating the attention score to enhance the positional relationship between entities. Finally, decoding is performed using global pointers.

[0006] This invention includes the following steps:

[0007] S1: The BERT module is used to process the input sequence S = {c1, c2, ..., cn} of length n. n The vector X is obtained by encoding the vector X, which is represented as X = {x1, x2, ..., x}. n};

[0008] S2: Adversarial sample X is obtained by adding a perturbation Δx to vector X through the adversarial training module. adv ;

[0009] S3: Combine vector X and adversarial example X adv After encoding by the linear layer module, the resulting vector H is represented as: H = {h1, h2, ..., h...} n};

[0010] S4: Input vector H into the attention mechanism module to obtain vectors Q and K. Vectors Q and K represent the query vector and keyword vector, respectively. The inner product of vectors Q and K yields the weighted attention score. In the attention module, relative position encoding is used to add relative position information to vectors Q and K.

[0011] S5: Decode the output of the attention module into multiple Span matrices using the global pointer module.

[0012] The beneficial effects of this invention are:

[0013] This invention is a named entity recognition method based on global pointers and adversarial training. Compared with existing technologies, this invention has the following technical advantages:

[0014] 1. Capable of handling entity recognition problems with large-scale datasets; can recognize nested entities, covering more entity recognition scenarios;

[0015] 2. It has a global perspective, allowing you to focus on the overall information of an entity;

[0016] 3. Adversarial training improved the robustness and generalization of the network model, resulting in improved performance on multiple public datasets;

[0017] 4. It can utilize the potential information in the original text data, such as pinyin information, to greatly enhance the richness of information. Attached Figure Description

[0018] Figure 1 This is the overall model architecture diagram of the present invention;

[0019] Figure 2 This is the global pointer module of this patent. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0021] The overall architecture of the Named Entity Recognition Model Based on Global Pointers and Adversarial Training (AT-CBGP) proposed in this invention is as follows: Figure 1 As shown, the model mainly consists of the following modules: ChineseBERT-base, adversarial training, attention mechanism, and global pointer.

[0022] The initial input sequence of the model is S = {c1, c2, ..., c...} n First, the ChineseBERT-base module takes an input sequence of length n, S = {c1, c2, ..., c...}. n The encoding process yields a vector representation X = {x1, x2, ..., x}. n Secondly, the adversarial training module adds a perturbation Δx to the vector X to obtain the adversarial sample X. adv Then combine the original vector X and the adversarial sample X adv After encoding by the linear layer module, the vector H = {h1, h2, ..., h} is obtained.n Next, vector H is input into the attention mechanism module to obtain vectors Q and K. Vectors Q and K represent the query vector and keyword vector, respectively. The inner product of vectors Q and K yields the weighted attention score. In the attention module, relative position encoding is used to add relative position information to vectors Q and K. Finally, the output of the attention module is decoded into multiple Span matrices through the global pointer module.

[0023] The model consists of the following modules:

[0024] 1. ChineseBERT

[0025] This invention employs ChineseBERT, combining word embedding and pinyin embedding of the text. The ChineseBERT-base contains 12 Transformer layers, each with 12-head self-attention and 768 hidden layer units. Assume the input sequence of length n is S = {c1, c2, ..., c...}. n}, c t This represents the t-th character in the sentence.

[0026] 2. Attention Mechanism

[0027] Positional encoding in attention mechanisms can be divided into absolute positional encoding and relative positional encoding. Although absolute positional encoding can add positional information to word vectors, this positional information is related to a fixed position and cannot represent the contextual information of that fixed position. Therefore, relative positional encoding was proposed to represent contextual information. Relative positional encoding does not depend on a fixed position but only on the relative position.

[0028] This invention uses rotational relative position encoding to inject relative position information into the model, thereby making the model more sensitive to the relative positions between entities and better able to identify entities.

[0029] 3. Global pointers

[0030] Global pointers can solve the problem of nested entities; compared with ordinary pointer networks, global pointers treat the beginning and end of an entity as a whole for discrimination, which is more global and avoids inconsistencies during training and prediction.

[0031] The global pointer module performs entity recognition based on entities as the basic unit. For a sentence of length seq, it constructs C seq×seq Span matrices. Each entity class in the sentence corresponds to one matrix, where C represents the entity class. Rows marked with 1 indicate the entity header, and columns marked with 1 indicate the entity footer. For example... Figure 2As shown in the figure, the matrices corresponding to two types of entities in the sentence are listed, where Span{"妈"}{"爱"} = 1, representing that "妈咪爱" is a drug entity, and Span{"助"}{"化"} = 1, representing that "助消化" is a drug category entity.

[0032] The global pointer module regards the entity recognition problem as a multi-label classification problem. However, there are usually only a small number of partial category entities in a sentence, resulting in a serious class imbalance problem. To solve this problem, the present invention adopts a unified loss function form:

[0033]

[0034] where i belongs to the negative sample set Ω neg , j belongs to the positive sample set Ω pos , γ is a scaling factor, s i is the non-target class score, s j is the target class score, and m is the margin.

[0035] This loss function is suitable for multi-label classification problems with a large total number of classes and a small number of target classes. It transforms the multi-label classification problem into pairwise differences between the target class scores and the non-target class scores for comparison, thereby restricting the target class scores to be greater than the non-target class scores. In addition, since the entity classes and quantities in each sentence are not fixed, a threshold s0 = 0 is set to determine which classes to output, such that each target class score s j is greater than 0, and each non-target class score s i is less than 0.

[0036] In the present invention, the specific loss function is formula (2). Among them, the head and tail sets (i, j) of all entities of type α form the positive sample set P α , and the head and tail sets (i, j) of all entities of non-α type or non-entities form the negative sample set N α . Among them, s α (i, j) represents the score that the subsequence formed by the i-th element to the j-th element in the input sequence S is an entity of the α-th class.

[0037]

[0038] 4. Adversarial training

[0039] Adversarial training can improve the robustness of the model when dealing with malicious samples. Secondly, as a way of regularization, it can slow down the overfitting of the model and improve the generalization ability. As Figure 1 shown, in the AT-CBGP model, the embedding vector obtained after encoding by ChineseBERT-base is X = {x1, x2,..., x nThe adversarial embedding vector X is generated by adding a perturbation Δx to the original embedding X. adv The calculation formula is as follows:

[0040]

[0041] g=▽ X L(X,y;θ) (4)

[0042] X adv =X + Δx (5)

[0043] Wherein, ∈ is a hyperparameter representing the degree of perturbation, which needs to be manually adjusted according to the specific experimental conditions. In this invention, it is set to 1; g is the gradient of the loss function, L(X,y;θ) is the loss function, and θ is the model parameter.

[0044] The comparison case with the existing technology is shown in the figure. The comparison is carried out on four public datasets. The experimental results show that the present invention can achieve better results.

[0045]

[0046]

[0047] The technical solutions of the present invention are not limited to the specific embodiments described above. Any technical modifications made in accordance with the technical solutions of the present invention fall within the protection scope of the present invention.

Claims

1. A named entity recognition method based on global pointers and adversarial training, characterized in that, Includes the following steps: S1: Use the BERT module to convert the length to... input sequence Encode to obtain vector , represented as ; S2: Vectors are trained via the adversarial training module. Add perturbation Obtain adversarial examples The calculation formula is as follows: in, This is a hyperparameter representing the degree of perturbation. This parameter needs to be manually adjusted according to the specific experimental conditions. In this invention, it is set to 1. It is the gradient of the loss function. It is a loss function. These are model parameters; S3: Transfer vector and adversarial examples After encoding by the linear layer module, a vector is obtained. , is represented as: ; S4: Transfer vector The vector is obtained by inputting into the attention mechanism module. and ;vector sum vector These represent the query vector and the keyword vector, respectively. and The weighted attention score is obtained by performing an inner product; within the attention module, it is encoded into a vector using relative positions. and Add relative position information; S5: Decode the output of the attention module into multiple Span matrices using the global pointer module; The global pointer module adopts a loss function form: Among them, all types are The first and last set of entities The positive sample set was formed. All types are non- The first and last set of entities or non-entities The negative sample set was formed ; Represents the input sequence From the middle The element to the first The subsequence consisting of the nth element is the nth... The score for entity classes.

2. The named entity recognition method based on global pointers and adversarial training according to claim 1, characterized in that: The BERT uses ChineseBERT-base, which contains 12 Transformer layers, each containing 12-head self-attention and 768 hidden layer units.

Images