A named entity recognition model based on adversarial transfer learning

Abstract

The present invention belongs to the field of natural language processing, and relates to a named entity recognition model based on adversarial transfer learning. It includes: S1. Embedding layer; S2. BiLSTM feature extraction layer; S3. Attention mechanism layer; S4. Adversarial transfer layer; S5. CRF decoding layer. Beneficial effect: In order to more accurately determine the entity boundary information, the present invention transfers the knowledge learned in the word segmentation task to the entity recognition task, that is, the information in the word segmentation task that is helpful for entity boundary detection is used to improve the effect of the entity recognition task; the present invention uses a large number of trained or annotated data sets for transfer learning, and transfers the knowledge learned by the model on a large amount of corpus to a specific field or a specific language NER task, thereby improving the NER effect. A unified framework for solving cross-task, cross-language and cross-domain NER is proposed.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing and relates to a named entity recognition model based on adversarial transfer learning. Background Art

[0002] In the era of rapid development of the Internet, massive amounts of text data are characterized by oral and networked features, which poses a challenge to extracting valuable information from a large amount of unstructured text. Named Entity Recognition (NER) is one of the basic tasks of information extraction and a key task in natural language processing. In the NER task, the Chinese NER task is more difficult because there are no boundaries between words in Chinese text; languages ​​such as English and Chinese have a large amount of annotated data, but for small languages, there are fewer annotated labeled data resources; in addition, there is a lack of sufficient annotated data in a certain field, which leads to the problem that the NER model works well in a certain language or field, but the effect drops sharply in another language or field, which poses a challenge to the domain adaptability of the model. A model that solves the problems of cross-task NER, cross-language NER and cross-domain NER is urgently needed. Summary of the invention

[0003] In order to solve the problems of cross-task NER, cross-language NER and cross-domain NER, the present invention proposes the following technical solution: a named entity recognition model based on adversarial transfer learning, comprising:

[0004] S1. Embedding layer: used to extract word-level features, character-level features and feature fusion;

[0005] S2.BiLSTM feature extraction layer: Use BiLSTM to obtain the context information of the input sequence;

[0006] S3. Attention mechanism layer: Use multi-head attention mechanism to capture multiple features of semantically interrelated information and learn relevant information in different representation subspaces;

[0007] S4. Adversarial transfer layer: weights are applied between transfer learning datasets to balance the datasets through adversarial neural networks;

[0008] S5.CRF decoding layer: calculate the optimal label sequence;

[0009] The model transforms, extracts, pools, and fuses the text X sequence through the embedding layer to obtain the output v of the fully connected network. i ; The BiLSTM feature extraction layer uses the output v of the fully connected network i As input, the word-level features of the input sequence context are extracted through the gating mechanism, and the output is h shared 、hsource 、h Target ;

[0010] h source 、h Target They are used as inputs of the attention mechanism layer respectively. The multi-head attention mechanism captures rich information of the text, improves the text performance, and the output is A.

[0011] h shared As the input of the adversarial transfer layer, the weight α is used to balance the impact of the large difference in training scale between high and low resources, making the feature representations extracted from the resource domain and the target domain more compatible;

[0012] A is used as the input of the CRF layer. The CRF filters the score of each label output by the neural network model. The label with the highest score is the best result of training.

[0013] Further, the embedding layer comprises the following steps:

[0014] Glove extracts word-level features: converts text sequences into pre-trained word vectors;

[0015] CNN extracts word character-level features: The word vector is sent to CNN to extract word character-level features, that is, the word vector matrix is ​​made into a uniform size through placeholders, and then the character-level features of the current word are extracted from the matrix composed of word vectors of each word through convolution, and then the key information in the features is extracted through pooling to obtain the word character-level features.

[0016] Feature fusion: The word-level vector and the character-level vector are concatenated and then input into a fully connected neural network to integrate the character-level features and the word-level features.

[0017] Furthermore, the BiLSTM feature extraction layer uses a bidirectional LSTM neural network to extract word-level features of the input sequence context.

[0018] Furthermore, the BiLSTM feature extraction layer is divided into a resource BiLSTM for extracting resource collection data, a target BiLSTM for extracting target data, and a shared BiLSTM for extracting shared features.

[0019] Furthermore, the attention mechanism layer is connected to the resource BiLSTM and the target BiLSTM as inputs respectively.

[0020] Furthermore, the attention mechanism layer is a multi-head attention layer; the output of the attention mechanism layer is A.

[0021] Beneficial effects: (1) The entity boundary detection effect in the NER task is effectively improved; named entity recognition can be considered as two processing tasks, namely, detecting entity boundaries and determining entity types. In order to more accurately determine entity boundary information, the present invention transfers the knowledge learned in the word segmentation task to the entity recognition task, that is, using the information in the word segmentation task that is helpful for entity boundary detection to improve the effect of the entity recognition task.

[0022] (2) Effectively solves the entity recognition problem in specific fields and minority languages. In order to solve the problem of lack of corpus in specific fields and minority languages, the present invention uses a large number of trained or annotated data sets for transfer learning, and transfers the knowledge learned by the model on a large amount of corpus to the NER task in a specific field or specific language, thereby improving the NER effect.

[0023] (3) A unified framework for solving cross-task, cross-language, and cross-domain NER is proposed. First, better feature representation is obtained by fusing word-level and character-level features, and relevant information is learned in different representation subspaces of the sentence through a multi-head attention mechanism. Then, in order to make the model more universal, the proposed model uses a shared BiLSTM component to extract shared feature representations. Finally, adversarial transfer learning is introduced to make the extracted shared feature representations more compatible.

[0024] The results show that the MFAT-NER model proposed in this paper can solve the NER problem in different scenarios under a unified framework and achieves good results, which provides a new idea for solving the domain adaptability problem of transfer learning in named entity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The overall structure diagram of the MFAT-NER model;

[0026] Figure 2 CNN extracts word features based on characters;

[0027] Figure 3 Schematic diagram of character features and word features splicing;

[0028] Figure 4 Schematic diagram of LSTM unit structure;

[0029] Figure 5 BiLSTM model diagram;

[0030] Figure 6 Multi-head attention structure diagram;

[0031] Figure 7 The impact of Dropout on experimental results;

[0032] Figure 8The impact of the number of multi-head attention heads on experimental results. DETAILED DESCRIPTION

[0033] In order to handle cross-task, cross-language and cross-domain NER tasks on a unified model, this paper proposes a named entity recognition model based on adversarial transfer learning, named MFAT-NER model. For cross-task NER, the knowledge learned from the CWS task is transferred to the Chinese NER task. For cross-language NER, the knowledge learned from English NER is transferred to Spanish NER and Dutch NER tasks. For cross-domain NER, the knowledge learned from the news domain is transferred to the Twitter comment domain.

[0034] The MFAT-NER model mainly consists of five parts: embedding layer, BiLSTM feature extraction layer, attention mechanism layer, adversarial transfer layer and CRF decoding layer. The overall structure of the model is as follows: Figure 1 As shown in the figure. For the feature fusion representation layer, the character-level feature representation learned by CNN and the word-level feature representation learned by Glove are connected; the BiLSTM feature extraction layer uses BiLSTM to obtain the context information of the input sequence; then the attention mechanism layer uses a multi-head attention mechanism to capture multiple features of semantically related information and learn related information in different representation subspaces; the adversarial transfer layer uses adversarial neural networks to apply weights between the transfer learning data sets to balance the impact of the large size difference of the data sets; finally, the CRF decoder is used to calculate the optimal label sequence to improve the effect of the model.

[0035] 1.1 Embedding Layer

[0036] Since character-level features have the ability to capture morphological and semantic information, the performance of the sequence labeling model can be improved. Character-level CNN or BiLSTM can encode these character-level features. In practical applications, CNN is more efficient and has fewer parameters. Therefore, the present invention uses character-level CNN to capture character-level feature representations. At the same time, in order to obtain rich and more informative word-level feature representations, the present invention connects the character-based feature representation of words learned by CNN and the word-level feature representation learned by Glove. The specific feature extraction process is described in detail in the following subsections.

[0037] 1.1.1 Glove extracts word-level features

[0038] Word embedding is a low-dimensional distributed word representation. This paper uses word-level vector representation to learn the semantics of words and their relationships, and uses word-level dynamic features extracted by Glove to improve the performance of the model. Specifically, for a text sequence X = (x1, x2, ... x i), where i is the length of the text. The text sequence is converted into a pre-trained word vector word = (word1, word2, ..., word i ).

[0039] 1.1.2 CNN Extracts Word Character Level Features

[0040] Compared with word-level feature representation, character-level feature representation can solve the problem of non-vocabulary. Character-level feature representation can effectively extract the morphological information of words, especially in certain specific fields, some words exist in irregular forms, and character-level feature representation can effectively solve the problem of irregular words. The specific process of CNN extracting word-based character features is as follows: Figure 2 As shown in the figure, character-level encoders include RNN and CNN, but character-level CNN is more efficient and has fewer parameters, so CNN is used to extract word-based character-level features.

[0041] Specifically, for a text sequence X = (x1, x2, ... x i ), where i is the length of the text. First, convert the text sequence into a pre-trained word vector char=(char1, char2,..., char i ), and then send the word vector to CNN to extract word character level features. Specifically, all word vector matrices are padded to the same size with the longest word length as the standard through placeholders (padding), and then the current word character level features are extracted from the matrix composed of each word vector through convolution, and then the key information in the features is further extracted through pooling to obtain word character level features. Where i is the length of the text.

[0042] 1.1.3 Feature Fusion

[0043] The advantage of word-level embedding is that it can learn the semantic relationship between words in a sentence, but its disadvantage is that it cannot well capture the dependencies between words. The advantage of character-level embedding is that it can capture the relevant information between characters within a word by learning characters, and it can solve the problem of unknown words very well. Therefore, in order to better utilize the advantages of word-level features and character-level features, word-level features and character-level features are fused. Specifically, the word-level vector and the character-level vector are concatenated in series, and then the concatenated vector is input into a fully connected neural network to integrate character-level features and word-level features. The advantage of this method is that the model can generate word-level feature representations that contain more information.

[0044] As shown in formula (1), the three embedding vectors are concatenated into a single word vector v i, the implementation process is as follows Figure 3 As shown. Single word vector and weight matrix W i Together with the bias vector b, it is fed into the fully connected network. The word embedding is the output of the fully connected network, and the vector size is 300. The output of the fully connected network is used as the final feature representation, and its dimension size is 300. As shown in formula (2).

[0045]

[0046] v i =f(W i vec i +b) (2)

[0047] in, Represents the character-level features of words, word i represents word-level features, and i is the length of the text.

[0048] 1.2.1 BiLSTM feature extraction layer

[0049] BiLSTM is an improved RNN model. It introduces a gating mechanism. Through the three gate structures of input gate, forget gate and output gate, the model can selectively save the previous information and effectively utilize long-distance information, thus overcoming the gradient vanishing problem caused by the long sequence of traditional RNN. The bidirectional structure of BiLSTM can effectively utilize contextual information and will not cause the gradient explosion problem. A feature of the sequence labeling task is that the forward input information and reverse input information of a given time step are helpful for label inference. In order to take advantage of this feature, a bidirectional LSTM neural network is used to extract word-level features of the input sequence context. BiLSTM can effectively utilize contextual information for each sentence without causing the gradient explosion problem. BiLSTM has gradually become a standard method for solving NER tasks. The structure of the LSTM unit is as follows Figure 4 shown.

[0050] The calculation method of LSTM unit is shown in formula (3)-(8). There is not much difference between LSTM and RNN externally, but its internal structure is different from RNN. LSTM can selectively retain information and selectively extract the latest information. The advantage of this processing is that more and longer time series data information can be stored in a limited storage space. For this reason, compared with RNN, LSTM can capture longer time series data, so as to better obtain the long-distance dependency of the input sequence and effectively solve the gradient disappearance problem caused by too long text sequence.

[0051] f i =σ(W f ·[hi-1 ,v i ]+b f ) (3)

[0052] i i =σ(W i ·[h i-1 ,v i ]+b i ) (4)

[0053]

[0054]

[0055] o i =σ(W o ·[h i-1 ,v i ]+b o ) (7)

[0056] h i =o i *tanh(C i ) (8)

[0057] Among them, v i is the input information of LSTM, W f , W i , W c , W o , b f , b i , b c and b o are all trainable parameters, f i The purpose is to deal with forgotten information. i Represents the current input information. Represents all current status information, C i Represents the current real state information, o i Represents the current output information, h i represents the current hidden state. σ represents the sigmoid activation function, tanh represents the tanh activation function, and h i-1 Represented as the hidden state of the previous moment, C i-1 Represents the real state information of the previous moment.

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067] in, and is the output of the forward LSTM, and is the output of the reverse LSTM, represents vector connection, h sh ,h so and h ta They are the final outputs of shared LSTM, resource LSTM and target LSTM respectively.

[0068] BiLSTM model model Figure 5 As shown in the figure, BiLSTM is composed of bidirectional LSTM. Because of this bidirectional structure, the current node is not only determined by the previous information, but also by the following information. Therefore, the information of the current node contains the context information of the current node.

[0069] As shown in formulas (18)-(20), and It is the output in two directions. represents vector connection, h i is the final output.

[0070]

[0071]

[0072]

[0073] In order to consider the differences in word-level features of different datasets, MFTA-NER decomposes BiLSTM into source BiLSTM, shared BiLSTM and target BiLSTM. Source BiLSTM and Target BiLSTM extract features of source data and target data respectively, and Shared BiLSTM extracts features that can be shared. is the output of Shared BiLSTM, is the output of Source BiLSTM, is the output of Target BiLSTM.

[0074] 1.3.1 Attention Mechanism Layer

[0075] The attention mechanism can selectively focus on the important information of the text. For the input text sequence, the information in a sentence may be determined by only a few words, while other components may only play the role of modifiers. According to the previous processing method, all words in a sentence are considered equally important, but this is obviously unreasonable. It is usually hoped that the model can focus on the important information in the text sequence. The advantage of this processing is that the model can quickly capture the semantic representation information of a sentence, and at the same time, it also reduces the amount of calculation to a certain extent and speeds up the calculation speed of the model. In addition, for the input text sequence, the information in a sentence may contain multiple levels and multiple aspects. In addition, in order to better capture multiple aspects or different angles of information in a sentence, this paper adopts a multi-head attention mechanism. The multi-head attention mechanism has two main advantages. One is that it can expand the ability of the model to pay attention to different positions; the other is that it provides multiple different representation subspaces for the attention mechanism layer, so that the model can learn the feature information of different aspects of the text from different representation subspaces. Therefore, the output of the resource BiLSTM and the target BiLSTM is sent to the multi-head attention layer.

[0076] Although BiLSTM can take all context information into account during calculation, it cannot highlight the importance of key information in the context. Therefore, after using BiLSTM to obtain context dependencies, the Source BiLSTM and TargetBiLSTM are connected to the attention layer. The inputs of the attention layer are and by As an example of the input of the attention layer, a linear transformation is first performed to obtain the query, key, and value. The specific calculation is shown in formulas (21)-(23). In fact, parallel calculation is achieved through matrices during calculation. The specific calculation is shown in formula (24). At the same time, when parsing complex text content, the multi-head attention mechanism can fully capture the rich information of the text, thereby improving the performance of the model. This paper adopts an 8-head attention mechanism, and its specific structure is as follows Figure 5 As shown, its essence is to connect the values ​​of self-attention. The specific calculation is shown in formulas (25)-(26).

[0077]

[0078]

[0079]

[0080]

[0081]

[0082] A=Concat(a1,a2,...,a8)W o (26)

[0083] Among them, Q, K, and V are q i , k i and v i The matrix is ​​composed of . d is the vector dimension, W q , W k , W v , W o are all trainable parameters. A is the output of the attention layer. is the output of Source BiLSTM, and Concat represents the concatenated vector.

[0084] 1.4.1 Anti-migration layer

[0085] This paper designs a unified model architecture for solving cross-task, cross-language and cross-domain NER. Its purpose is to use existing easily accessible corpus data for transfer learning to improve the effect of NER on scarce and difficult to obtain corpus data. In addition, due to the imbalance in the size of the resource dataset and the target dataset, if this imbalance is not taken into account, the feature representations extracted from them are incompatible. In order to capture shared features, this paper introduces a shared BiLSTM and an adversarial discriminator.

[0086] Specifically, there are few annotated training data in the target dataset. In contrast, the resource dataset contains sufficient annotated training data. Since the feature representations of different corpus data are different when performing cross-language NER, cross-domain NER, and cross-task NER, the model in this paper uses Shared BiLSTM to extract the same and sharable feature representations between different data to achieve NER migration on corpus data of different languages, different domains, and different tasks. However, there is a problem that must be considered. For example, when performing cross-domain named entity recognition, the dataset in the resource domain is easier to obtain and larger in scale than the dataset in the target domain. This results in the resource domain data dominating when Shared BiLSTM extracts data sharing feature representations. If this imbalance is not considered in training, stochastic gradient descent optimization will bias the model towards datasets with more annotated data. To solve this problem, a weight α is set to balance the impact of the large difference in training scale between high and low resources.

[0087] In order to apply the shared feature representation to the target language NER, target domain NER and target task NER, the feature representation of the corpus data is migrated. Only by making the feature representation more compatible and domain-independent can the feature representation be better migrated and the model more transferable. Specifically, for the output of Shared BiLSTM The gradient is reversed through the Gradient Reversal layer to change its gradient to the opposite sign, and then through this adversarial mechanism, the shared layer is encouraged to learn shared feature representations. For the problem of too large a difference in the size of the dataset, the weight α is used to balance the impact of the large difference in the training scale of high and low resources, so that the feature representations extracted from the resource domain and the target domain are more compatible. The output of the shared BiLSTM is independent of the domain, and an adaptive weight is provided for each sample, so that the focus of model training is placed on difficult samples. Only in this way can the shared feature representation be more compatible and better migrated to other languages, other domains, and other tasks. Then, through the attention mechanism, the output of the attention layer in the adversarial migration layer is a i , and then projected to the scalar a' through linear transformation i , the specific calculation is as shown in formula (27). At this time, the calculation of the adversarial discriminator is as shown in formula (28).

[0088] a' i =W s ·a i +b s (27)

[0089]

[0090] Where W s , b sThese are all trainable parameters. and is an identification function, indicating whether the feature comes from the resource domain or the target domain; the parameter γ measures the loss contribution of difficult and simple samples, (1-a' i ) γ (or a' i γ ) controls the loss contribution of each sample by measuring the difference between the predicted value and the true label. The weights α and (1-a' i ) γ (or (a' i ) γ ) reduce the loss contribution of high-resource samples and simple samples respectively.

[0091] 1.5.1 CRF Decoding Layer

[0092] CRF is a feature-flexible global optimal labeling framework that learns constraints from training data to ensure the effectiveness of predicted labels. CRF filters the scores of each label output by the neural network model, and the label with the highest score is the best result of training. The input of the CRF layer is A = (a1, a2, ..., a i ), for sentence X = {x1, x2, ..., x i}, the predicted label is Y = {y1,y2,…,y i}, the specific formulas for CRF label prediction are as shown in formulas (29)-(31). The specific calculation of the probability formula is as shown in formula (32). The specific calculation of the loss function is as shown in formula (33).

[0093] O i =W c ·a i +b c (29)

[0094]

[0095]

[0096]

[0097]

[0098] Among them, W c and b c are all trainable variables. is x i Prediction for the yth i The score of the labels, n is the sequence length, T is the transfer score matrix, Y x For all candidate tag sequences of sentence X, use the Viterbi algorithm to predict the tag sequence is the correct label sequence, X i Represents the i-th label of sentence X.

[0099] 2.1 Experiment and results analysis

[0100] In this section, we first introduce the details of the dataset, the experimental comparison model, the experimental evaluation indicators, experimental parameters, and experimental environment settings. Then, we report and analyze the results of cross-task, cross-language, and cross-domain named entity recognition in detail, and conduct an ablation experiment on the MFAT-NER model.

[0101] 2.1.1 Dataset

[0102] In order to verify the effectiveness of the model, experiments were conducted on different datasets. The NER task and the word segmentation task have many similar word boundaries. The character features learned from the word segmentation task may provide key information for tagging. Therefore, feature information is shared between the word segmentation task and the NER task to enhance the feature representation of the NER task. The word segmentation task uses the MSR dataset from SIGHAN2005 (this corpus is provided by Microsoft Yanzhou Research Institute, and the corpus is a simplified Chinese corpus), and the NER task uses the NER dataset (MSRANER dataset) from SIGHAN2006, the Weibo NER dataset, and the Chinese literature dataset. The statistical information of these datasets is shown in Table 1. In addition, experiments were also conducted on the CoNLL-2003 English NER dataset, the CoNLL-2002 Spanish and Dutch NER dataset, and the WNUT2016 and WNUT2017 English Twitter NER datasets. In the cross-language and cross-domain NER tasks, the CoNLL-2003 English NER dataset is used as the source domain and source language, and the CoNLL-2002 and WNUT datasets are used as the target domain and target language, respectively. The statistical information of these datasets is shown in Table 2 and Table 3.

[0103] Table 1 Statistics of cross-task NER datasets

[0104] Tab.1 Cross-task NER dataset statistics

[0105]

[0106] Table 2 Statistics of cross-language NER datasets

[0107] Tab.2 Cross-language NER dataset statistics

[0108]

[0109] Table 3 Statistics of cross-domain NER datasets

[0110] Tab.3 Cross-domain NER dataset statistics

[0111]

[0112] 2.2.1 Comparison Model

[0113] BiLSTM: It is the baseline model for the NER task.

[0114] CRF: is a statistical modeling model commonly used in pattern recognition and machine learning. Their feature templates include unigram and bigram features.

[0115] Pipeline Seg.Repr.+NER: This is a NER system through multi-task learning and Chinese word segmentation.

[0116] Lattice: It is a lattice-structured LSTM model for Chinese NER that encodes the input character sequence along with all potential words matched with the dictionary.

[0117] BCAT-NER: It is a novel adversarial transfer learning framework that fully exploits the task-shared boundary information and prevents the task-specific characteristics of CWS.

[0118] WC-LSTM: It is a word and character LSTM model by adding word information to the beginning or end character of the word. It mitigates the impact of word segmentation errors.

[0119] CAN-NER: It uses character-based CNN and gated recurrent units to capture character-level and sentence-level information.

[0120] DoubADV: It is a dual AD-Versarial model that uses two adversarial networks and considers the shared and private features in Chinese word segmentation (CWS) task and Chinese named entity recognition task.

[0121] LM-LSTM-CRF: It is a sequence labeling framework that uses both word-level and character-level knowledge.

[0122] BERT-base: It is a pre-trained BERT model for NER, which can be applied to many downstream tasks, such as named entity recognition tasks.

[0123] Flair Embeddings: It uses a language model to generate a new type of word embeddings called contextual string embeddings.

[0124] DATNet: It aims to solve the low-resource NER problem.

[0125] 2.3.1 Experimental environment

[0126] The MFAT-NER model proposed in this invention is implemented in the tensorflow framework; the Adam optimizer is used for small batch training; the learning rate is set to 0.001, the neural unit deactivation rate is set to 0.5; the size of the hidden layer is 300; the weight parameter α is set to 0.25, and γ is set to 2.0; all experiments are carried out on the same machine, using an Intel (R) Xeon (R) CPU E5-2640 v4 @ 2.40GHz and a GeForce PTX2080Ti GPU. The specific model parameters are shown in Table 4, and the experimental environment is shown in Table 5.

[0127] Table 4 Main parameter settings in the experiment

[0128] Tab.4 Main parameter settings in the experiment

[0129]

[0130] Table 5 Experimental environment settings

[0131] Tab.5 Experimental environment settings

[0132]

[0133] 2.4.1 Evaluation indicators

[0134] The detection standard for named entity recognition tasks is divided into two steps: identifying entity boundaries and labeling entity types.

[0135] Therefore, accurate evaluation of named entity recognition also includes two aspects:

[0136] 1. Correctly identify entity boundaries;

[0137] 2. Correctly label entity types.

[0138] In order to evaluate the performance of the model, the experiment uses three evaluation indicators: precision (P), recall (R), and F1 value to evaluate the accuracy of the model. The specific calculation is as shown in formulas (1)-(3).

[0139]

[0140]

[0141]

[0142] Simply considering the accuracy and recall rate may lead to contradictions, such as the case where the accuracy is very high and the recall rate is very low, or the case where the recall rate is very high and the accuracy rate is very low. Therefore, the F1 value is introduced to comprehensively consider the impact of the two indicators.

[0143] 2.5.1 Experimental analysis

[0144] (1) Comparison of experimental results

[0145] To verify the effectiveness of the model, the experimental results are shown in Tables 6 to 8.

[0146] Table 6 Cross-task named entity recognition results

[0147] Tab.6 Cross-task named entity recognition results

[0148]

[0149] Table 7 Cross-language named entity recognition results

[0150] Tab.7 Cross-lingual named entity recognition results

[0151]

[0152] Table 8 Cross-domain named entity recognition results

[0153] Tab.8 Cross-domain named entity recognition results

[0154]

[0155] As shown in Tables 6-8, for the cross-task NER experimental results, the F1 values ​​of the three datasets of MSRA, Chinese Weibo and Chinese literature reached 91.97%, 59.79% and 81.09% respectively. Among them, due to the flexible and changeable text of Chinese Weibo, the language logic is not standardized, there are few public corpora and no word boundary features, which brings certain difficulties to the NER model, resulting in the lowest NER effect of Chinese Weibo among the three Chinese corpora. However, compared with the previous NER models for processing Weibo text, the model proposed in this paper still achieved excellent results, which shows that the model in this paper is effective. For the cross-language NER experimental results, the F1 value on the CoNLLl-2002 Spanish dataset reached 88.74%, and the F1 value on the CoNLL-2002 Dutch dataset reached 92.06%. For the cross-domain NER experimental results, the F1 value on the WNUT-2016 dataset reached 54.36%, and the F1 value on the WNUT-2017 dataset reached 51.30%. Also, due to the small scale of WNUT-2016 and WNUT-2017 datasets, there are few datasets available for training NER models, resulting in relatively low cross-domain NER effects. However, our model still achieves good results on these two datasets, which further proves the effectiveness of our model.

[0156] Table 9 Comparison of F1 values ​​of cross-task NER model with other models

[0157] Tab.9 Comparison of F1 values ​​between cross-task NER model and other models

[0158]

[0159] As can be seen from Table 9, in order to verify the effectiveness of the MFAT-MER model for cross-task NER, this paper compares it with the previous SOTA model on two datasets: Chinese Weibo and Chinese literature.

[0160] For the Chinese literature dataset, the BiLSTM

[63] and CRF

[64] models have a simple structure compared to complex network models, resulting in poor learning ability, so the model performance is not outstanding. For the Weibo dataset, compared with the ensemble model

[65] , the MFTA-NER model uses a multi-head attention mechanism to adjust different weights on the BiLSTM model output to capture multiple features of semantically related information in different subspaces. It makes the correlation of word information between individual characters and the correlation of connections between individual words more prominent. Compared with the models proposed by Zhang et al.

[66] and Liu et al.

[47] , the MFTA-NER model shares word boundary information and can make full use of the rich word boundary information of the word segmentation task to improve the performance of the NER task. Compared with the BCAT-NER model

[67] , the MFTA-NER model uses a multi-head attention mechanism instead of a self-attention mechanism to capture feature information in different subspaces. Zhu et al.

[48] explored the CAN-NER model that uses the word segmentation task to improve the performance of NER in the Chinese Weibo NER dataset. Compared with the CAN-NER model, the MFTA-NER model uses the boundary information of the word segmentation task to improve the performance of the NER task. In addition, the MFTA-NER model introduces adversarial transfer learning, which can make the boundary information extracted in the word segmentation task and the boundary feature information extracted in the NER task more compatible. Compared with the DoubADV model designed by Hu et al.

[68] , the MFTA-NER model shares character-level features, the source word-level embedding is the input of the source BiLSTM, and the target word-level representation is the input of the target BiLSTM. The final feature representation is more compatible through adversarial transfer learning.

[0161] Table 10 Comparison of F1 values ​​between cross-domain / cross-language NER model and other models

[0162] Tab.10 Comparison of F1 values ​​between cross-domain / cross-languageNER model and other models

[0163]

[0164]

[0165] To evaluate the performance of the proposed model on cross-lingual NER and cross-domain NER tasks, the proposed model is compared with previous state-of-the-art models on the CoNLL-2002 (Dutch) and WNUT-2017 datasets. The results are described in Table 10.

[0166] The LM-LSTM-CRF model achieved excellent performance on multiple tasks. The model has a Highway layer that maps the character-level representation generated by LSTM to another representation space. Compared with the LM-LSTM-CRF model, the MFTA-NER model concatenates the character-based word-level feature representation learned by CNN and the word-level feature representation learned by Glove, and the MFTA-NER model outperforms the models that use only character-level features or only word-level features. For the pre-trained BERT-base model, the MFTA-NER model introduces LSTM neural networks, multi-head attention mechanisms, and adversarial transfer learning, which enables the model to learn more about the internal features of the sequence. Compared with the Flair embedding model proposed by Akbik et al.

[71] and the DATNet model proposed by Zhou et al., the MFTA-NER model introduces a multi-head attention mechanism after the BiLSTM layer to capture multiple features of semantically relevant information and related information in different subspaces.

[0167] (2) Ablation experiment

[0168] In order to verify the impact of different modules in the MFAT-NER model on the overall performance of the model, an ablation experiment of the model was conducted on the MSRA, Weibo datasets, CoNLL-2002 (Spanish) datasets, and WNUT-2016 datasets. The ablation experiment results are shown in Table 11:

[0169] Table 11 Ablation experimental model results

[0170] Tab.11 Comparison of results of ablation experimental models

[0171]

[0172] Table 11 shows that in previous NER tasks, the BiLSTM-CRF model is one of the mainstream models, so the BiLSTM-CRF model is used as the baseline model. The multi-head self-attention mechanism is combined with the baseline model to accurately capture various features, while also eliminating the interference of redundant information to a large extent. The F1 values ​​on the four data sets are increased by 0.58%, 1.24%, 1.31% and 1.17% respectively.

[0173] On this basis, adversarial learning is added. Since a sub-part of the model is a shared feature module, considering that the shared features contain noise problems, and also to make the features of the source domain and the target domain more compatible, adversarial learning is introduced to further purify the shared features. In addition, a pooling layer is added to extract features, a gradient reversal layer is used to optimize the discriminator parameters, and an attention mechanism is used to assign different weights to different features. The F1 values ​​on the four data sets are increased by 0.93%, 0.83%, 0.81 and 1.12% respectively.

[0174] CNN is used to extract word character-level features, which can not only retain the advantages of word vectors, but also make full use of the semantic information of the word corresponding to the current word. The F1 values ​​on the four data sets are improved by 0.33%, 4.94%, 4.13% and 3.78% respectively. The results are remarkable in Chinese Weibo dataset, CoNLL-2002 (Spanish) dataset and WNUT-2016 dataset. Firstly, the three datasets have a lot of room for improvement in NER tasks. Secondly, the flexible and changeable text of Chinese Weibo and the irregular language logic make NER tasks difficult. Because the word character level feature representation contains deeper semantic information, the model can learn more knowledge, so the improvement effect becomes significant. For CoNLL-2002 (Spanish) dataset and WNUT-2016 dataset, the scale of both datasets is very small. In order to obtain higher quality feature representation for these two datasets, the character feature representation learned from the relatively large CoNLL-2003 dataset may provide key information for annotation, especially for rare words and words with insufficient vocabulary. By obtaining word character level features, the advantages of word vectors can be retained and the semantic information of the current word corresponding to the word can be fully utilized. At the same time, a multi-head attention mechanism is introduced to capture multiple features. As a result, the final effect is significantly improved.

[0175] 2.6.1 Hyperparameter Settings

[0176] (1) Dataset running time analysis

[0177] Due to the different sizes of data sets, the model learning time will also vary. The running time of the model for different data sets is shown in Table 12:

[0178] Table 12 Dataset running time

[0179] Tab.12 Datasets running time

[0180]

[0181] (2) Analysis of neural unit loss rate

[0182] In order to better verify the performance of the model and prevent overfitting, the initial dropout rate (dropout) of the neural network unit is 0.3. In order to further explore the impact of dropout on the experimental results, the dropout values ​​are set to 0.1 to 0.9, and the experimental curve is as follows: Figure 7 shown.

[0183] Dropout is to stop some neurons from working with a certain probability value during forward propagation, which can make the model more generalizable. Because in this way, the model will not be too dependent on certain local features. The model performance is optimal when Dropout is 0.5. At this time, there is a 50% probability that the neurons will be retained and a 50% probability that they will be inactivated, and the randomly generated network structure is the most. It can learn feature information well while preventing overfitting.

[0184] (3) Analysis of the number of multi-head attention heads

[0185] Integrating the attention mechanism into the deep network can improve the processing ability of the input information subset. The attention mechanism gives more important weights to features with greater correlation, which can make up for the shortcomings of deep learning to a certain extent. On this basis, in order to enable the model to learn relevant information in different representation subspaces, capture multiple features of semantically related information, and learn the internal structure features of sentences, this paper adopts a multi-head attention mechanism. Figure 8 The effect of the number of attention heads on the experiment was verified. The model effect is optimal when the number of attention heads is 8. When the number of heads is 1, the relevant information of different representation subspaces cannot be fully considered. When the number of heads increases to 12, the attention mechanism will focus on more feature information, which will bring some redundant information, so the model effect will decrease instead of increase.

Claims

1. A named entity recognition model based on adversarial transfer learning, characterized in that: include: S1. Embedding layer: used to extract word-level features, character-level features and feature fusion; S2.BiLSTM feature extraction layer: Use BiLSTM to obtain the context information of the input sequence; S3. Attention mechanism layer: Use multi-head attention mechanism to capture multiple features of semantically interrelated information and learn relevant information in different representation subspaces; S4. Adversarial transfer layer: weights are applied between transfer learning datasets to balance the datasets through adversarial neural networks; S5.CRF decoding layer: calculate the optimal label sequence; The model transforms, extracts, pools, and fuses the text X sequence through the embedding layer to obtain the output v of the fully connected network. i ; The BiLSTM feature extraction layer uses the output v of the fully connected network i As input, the word-level features of the input sequence context are extracted through the gating mechanism, and the output is h shared 、h source 、h Target ; h source 、h Target They are used as inputs of the attention mechanism layer respectively. The multi-head attention mechanism captures rich information of the text, improves the text performance, and the output is A. h shared As the input of the adversarial transfer layer, the weight α is used to balance the impact of the large difference in training scale between high and low resources, making the feature representations extracted from the resource domain and the target domain more compatible; A is used as the input of the CRF layer. The CRF filters the scores of each label output by the neural network model. The label with the highest score is the best result of the training. The embedding layer comprises the following steps: Glove extracts word-level features: converts text sequences into pre-trained word vectors; CNN extracts word character-level features: The word vector is sent to CNN to extract word character-level features, that is, the word vector matrix is ​​made into a uniform size through placeholders, and then the character-level features of the current word are extracted from the matrix composed of word vectors of each word through convolution, and then the key information in the features is extracted through pooling to obtain the word character-level features. Feature fusion: The word-level vector and the character-level vector are concatenated and then input into a fully connected neural network to integrate the character-level features and the word-level features. The BiLSTM feature extraction layer is divided into a resource BiLSTM for extracting resource collection data, a target BiLSTM for extracting target data, and a shared BiLSTM for extracting shared features.

2. According to claim 1, a named entity recognition model based on adversarial transfer learning is characterized in that: The BiLSTM feature extraction layer uses a bidirectional LSTM neural network to extract word-level features of the input sequence context.

3. According to claim 1, a named entity recognition model based on adversarial transfer learning is characterized in that: The attention mechanism layer is connected to the resource BiLSTM and the target BiLSTM as input respectively.

4. According to claim 1, a named entity recognition model based on adversarial transfer learning is characterized in that: The attention mechanism layer is a multi-head attention layer; The output of the attention mechanism layer is A.

Images