An autoregressive model-based cross-domain named entity recognition method

Abstract

The application discloses a cross-domain named entity recognition method based on an autoregressive model, and comprises the following steps: S1, encoding an input sequence; S2, encoding a label through a label encoder; S3, obtaining label background information; S4, obtaining label context information; S5, connecting the label background information to the input sequence and connecting the label context information to the predicted named entity label as final label perception information z i , and finally obtaining a final sequence representation u; the application provides a cross-domain named entity recognition method based on an autoregressive model, which improves the relationship between a source text and its named entity label, improves the portability of label information, and helps the model to promote domain adaptation.

Description

Technical Field

[0001] This invention relates to the field of computer natural language processing technology, and more specifically to a cross-domain named entity recognition method based on an autoregressive model. Background Technology

[0002] Named entity recognition (NER) is a fundamental task in Natural Language Processing (NLP), aiming to identify salient information such as people and locations from raw text, and is considered a specific sequence labeling problem. However, due to differences in text types and limitations of labeled data, most traditional methods trained for a specific domain (source domain) are difficult to generalize to new domains (target domain). Therefore, to alleviate this problem, cross-domain NER has been proposed, aiming to learn information from the source domain to enhance the target domain NER, borrowing entity information from the source domain to help entity recognition in the target domain with limited labeled data. As an important research direction in Natural Language Processing, cross-domain named entity recognition has been extensively studied, and a large number of methods have emerged, among which models based on pre-trained language models have recently achieved significant improvements. Although existing methods have good performance, most methods focus on reducing the differences in labeled representations between the source and target domains, while the transmission of valuable label information is often not explicitly considered or even ignored. Therefore, a new autoregressive cross-domain named entity recognition framework has been proposed to enhance the transmission of label information and help the model adapt to the domain. However, using this model to solve this cross-domain named entity recognition problem has the following three problems:

[0003] 1) The semantic information of the labels in the training samples is not utilized, that is, the sequence information of the token-token itself is lost;

[0004] 2) The label information is not fully mined, and the original sequence information after LSTM will be lost after attention combination;

[0005] 3) The information between token and label has semantic differences. The token embedding and label embedding themselves do not have a direct semantic relationship. Directly using attention for QK query does not conform to semantics and hinders the attention matching between label and token.

[0006] Therefore, how to provide a cross-domain named entity recognition method based on an autoregressive model that does not lose label information and can capture bidirectional information of token-label and label-token is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a cross-domain named entity recognition method based on an autoregressive model, which improves the portability of tag information by enhancing the relationship between the source text and its named entity tags, and helps the model to promote domain adaptation.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A cross-domain named entity recognition method based on an autoregressive model includes the following steps:

[0010] S1. Given an input sequence X = {x1, x2, ..., xn} consisting of N characters. N Encode the input characters to obtain the sequence [h1, h2, ..., h...]. N ], where the character represents h i For each character x i A d1-dimensional vector;

[0011] S2. The training samples are labeled y1,...,y using a label encoder. i-1 Encode the sequence to obtain the label representation sequence [e1, e2, ..., e i-1 ];

[0012] S3. e i-1 Projected as the query vector onto h i The same dimension yields e' i-1 , through e' i-1 Calculate the attention weights of the query vector pass The label background information is obtained by performing a weighted summation on the character representations.

[0013] S4. Represent the input character as h i and marking background information As a comprehensive intermediate state, it is further mapped to a 2d2-dimensional vector h' i , through h' i Get the character representation h i Attention weights on the label representation sequence pass The tag context information is obtained by performing a weighted summation on the tag representations.

[0014] S5. Add label background information With tag context information Connect the links to obtain the final label-aware information z. i , will h i and corresponding label-aware information z iAnd the label represents the sequence e i Connecting them together, we get the final sequence representation u:

[0015]

[0016] S6. Map the final sequence representation u onto the actual label Y through a linear layer as the predicted label, and then compare the predicted label with the training sample labels y1,...,y in S2. i-1 Create a new tag by making a connection;

[0017] S7. Apply the new labels to the training sample labels y1,...,y i-1 Perform the update, repeating steps S2-S7.

[0018] Preferably, in S1, for the input sequence X = {x1, x2, ..., x...} N The specific content to be encoded includes:

[0019] The input sequence is encoded using a pre-trained input sequence encoder and the encoder layer of a transformer model, denoted as f. det (·):

[0020] [h1,h2,...,h N ] = f det (x1,x2,...,x N )

[0021] Where h i It is each character x i A d1-dimensional vector is used to capture the contextual information of the corresponding tag.

[0022] Preferably, the specific content of S2 includes:

[0023] A randomly initialized tag lookup table is constructed using the tag encoder. Where K represents the number of unique labels in the source or target domain, d2 is the size of the label embedding, and R is a randomly initialized embedding matrix;

[0024] For a label y k For k∈{1:K}, use the label lookup table U to find the label y k Represented as The tag sequence is encoded by the tag encoder and represented as follows:

[0025] [e1,e2,...,e i-1 ] = f re (s1,s2,...,s i-1 )

[0026] in It is the output of the label encoder for k∈{1:i-1}, used to capture the context information of the label.

[0027] Preferably, the specific content of S3 includes:

[0028] e i-1 Projected onto h i Same dimensions:

[0029] e' i-1 =W2·e i-1 +b2

[0030] Where e' i-1 It is a d1-dimensional vector;

[0031] Calculate the attention weights of the query vector

[0032]

[0033] Here, It is considered as a probability distribution and used to generate a weighted sum of the input character representations:

[0034]

[0035] Tag background information With e i-1 This serves as a guide, indicating the relationship between the label of the current character and the entire input sequence.

[0036] Preferably, the specific content of S4 includes:

[0037] The input character is represented as h. i and the marked background information As a comprehensive intermediate state, it is further mapped to a 2d2-dimensional vector:

[0038]

[0039] pass The tag context information is obtained by performing a weighted summation on the tag representations:

[0040]

[0041] in The character represents h i In e1, e2, ..., e i-1 A weighted vector on.

[0042] Preferably, the specific content of S5 includes:

[0043] The label background information By connecting the context information to the input sequence, the predicted named entity labels are linked together, serving as the final label-aware information z. i :

[0044]

[0045] Where W2, W3, b2, and b3 are learnable parameters;

[0046] h i and corresponding label-aware information z i and tag context information e i Connect them:

[0047]

[0048] Where u is the final sequence representation.

[0049] S5 also includes: through a trainable matrix W o and b o Bias will u i Mapped to output space o i =W o ·u i +b o The distribution of all named entity labels is obtained by using a normalization function.

[0050] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a cross-domain named entity recognition method based on an autoregressive model, which has the following beneficial effects:

[0051] 1. This method can incorporate the original label information into the final classification without losing the label information;

[0052] 2. This method can capture bidirectional information between token-label and label-token;

[0053] 3. This method has a wider range of applications than existing methods. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1This is a flowchart illustrating a cross-domain named entity recognition method based on an autoregressive model, provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] This invention discloses a cross-domain named entity recognition method based on an autoregressive model, comprising the following steps:

[0058] S1. Given an input sequence X = {x1, x2, ..., xn} consisting of N characters. N Encode the input characters to obtain the sequence [h1, h2, ..., h...]. N ], where the character represents h i For each character x i A d1-dimensional vector;

[0059] S2. The training samples are labeled y1,...,y using a label encoder. i-1 Encode the sequence to obtain the label representation sequence [e1, e2, ..., e i-1 ];

[0060] S3. e i-1 Projected as the query vector onto h i The same dimension yields e' i-1 , through e' i-1 Calculate the attention weights of the query vector pass The label background information is obtained by performing a weighted summation on the character representations.

[0061] S4. Represent the input character as h i and marking background information As a comprehensive intermediate state, it is further mapped to a 2d2-dimensional vector h' i , through h' i Get the character representation h i Attention weights on the label representation sequence pass The tag context information is obtained by performing a weighted summation on the tag representations.

[0062] S5. Add label background information With tag context information Connect the links to obtain the final label-aware information z. i , will h i and corresponding label-aware information z i And the label represents the sequence e i Connecting them together, we get the final sequence representation u:

[0063]

[0064] S6. Map the final sequence representation u to the actual label Y through a linear layer as the predicted label. Then, map the predicted label to the training sample labels y1,...,y in S2. i-1 Create a new tag by making a connection;

[0065] S7. Label the training samples y1,...,y using the new labels. i-1 Perform the update, repeating steps S2-S7.

[0066] In this embodiment:

[0067] Autoregressive models refer to:

[0068]

[0069] That is, given an input sequence X = {x1, x2, ..., xn} with N characters. N The goal of the named entity recognition task is to output a sequence of corresponding labels Y = {y1, y2, ..., y} of the same length. N That is, modeling P(Y|X);

[0070] To effectively transfer information to the target domain, the model is trained in two stages: pre-training and fine-tuning. The first stage, pre-training, involves Domain Adaptive Pre-training (DAPT) to enhance text feature extraction from the target domain. This involves pre-training the input sequence encoder on a domain-related corpus to reduce the differences in domain context and text distribution between the source and target domains, and to further capture more effective features from the target domain. This process allows for the learning of valuable tag embeddings before accessing the target domain, particularly for shared named entity tags; such as... Figure 1 As shown, this embodiment uses the BERT model as the input sequence encoder; the encoder layer of the transformer is used to capture the labeled semantic information of the training samples.

[0071] The second stage of fine-tuning refers to: fine-tuning the model in the target domain to make it suitable for D. tgtWith the help of the shared label embeddings pre-trained in the first stage, a Bi-LSTM is used to encode the label sequences. The model further learns the relationship between shared named entity labels and target domain-specific named entity labels (i.e., labels that only exist in the target domain) and the inherent label dependency information. This further helps the model utilize knowledge from the source domain to better understand the labels that are not visible in the target domain.

[0072] To model the relationship between the tag sequence and the label sequence, the tag encoder is based on the commonly used current character representation (i.e., h). i ) and label-aware information extracted from previous labels (i.e., y 1：i-1 To predict the labels of named entities;

[0073] The label predictor uses contextual information from the input sequence and previous label sequences to predict NER labels. To combine these two types of information, a simple and effective dual-attention module is used; the dual-attention module separately obtains the label background information. and tag context information

[0074] To further implement the above technical solution, in S1, the input sequence X = {x1, x2, ..., x...} is... N The specific content to be encoded includes:

[0075] The input sequence is encoded using a pre-trained input sequence encoder and the encoder layer of a transformer model, denoted as f. det (·):

[0076] [h1,h2,...,h N ] = f det (x1,x2,...,x N )

[0077] Where h i It is each character x i A d1-dimensional vector is used to capture the contextual information of the corresponding tag.

[0078] The transformer's encoder layer is used to capture the sequence semantic information of the labeled training samples so that the subsequent sequence representation u contains the sequence semantic information of the input sequence itself.

[0079] Because an encoder layer is added on top of the token embedding, this invention is able to capture the semantic information of the labeled training samples themselves.

[0080] To further implement the above technical solution, the specific content of S2 includes:

[0081] A randomly initialized tag lookup table is constructed using a tag encoder. Where K represents the number of unique labels in the source or target domain, d2 is the size of the label embedding, and R is a randomly initialized embedding matrix;

[0082] For a label y k For k∈{1:K}, use the label lookup table U to find the label y k Represented as The tag sequence is encoded using a tag encoder and represented as follows:

[0083] [e1,e2,...,e i-1 ] = f re (s1,s2,...,s i-1 )

[0084] in It is the output of the tag encoder for k∈{1:i-1}, used to capture the contextual information of the tags.

[0085] To further implement the above technical solution, the specific content of S3 includes:

[0086] The dual attention module refers to the last hidden state (i.e., e) in the Bi-LSTM of the label encoder. i-1 The input sequence encoder will treat the character representation as a label sequence, which will be used as the query vector, while all characters in the input sequence encoder will be represented as characters (i.e., h). 1：N () is considered as a key-value matrix. A fully connected layer is used to... i-1 Projected onto h i Same dimensions:

[0087] e' i-1 =W2·e i-1 +b2

[0088] Where e' i-1 It is a d1-dimensional vector;

[0089] Calculate the attention weights of the query vector using the softmax function.

[0090]

[0091] Here, It is considered as a probability distribution and used to generate a weighted sum of the input character representations:

[0092]

[0093] Tag background information With e i-1This serves as a guide, indicating the relationship between the label of the current character and the entire input sequence.

[0094] To further implement the above technical solution, the specific content of S4 includes:

[0095] The dual attention module needs to capture the current character x. i The relationship with previously predicted labels (i.e., y) 1：i-1 ), to improve x i Sensitivity to previously named entity labels. Represent the input character as h. i and marking background information As a comprehensive intermediate state, it is further mapped to a 2d2-dimensional vector:

[0096]

[0097] pass We obtain the label context information by performing a weighted summation on the label representations:

[0098]

[0099] in The character represents h i In e1, e2, ..., e i-1 A weighted vector on.

[0100] To further implement the above technical solution, the specific content of S5 includes:

[0101] Label background information By connecting the context information to the input sequence, the predicted named entity labels are linked together, serving as the final label-aware information z. i :

[0102]

[0103] Where W2, W3, b2, and b3 are learnable parameters;

[0104] To further integrate tag-related knowledge into the character x i In the middle, h i and corresponding label-aware information z i and tag context information e i Connect them:

[0105]

[0106] Where u is the final sequence representation.

[0107] S5 also includes: through a trainable matrix W o and b oBias will u i Mapped to output space o i =W o ·u i +b o And use a softmax function to obtain the distribution of all named entity labels.

[0108] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0109] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cross-domain named entity recognition method based on an autoregressive model, characterized in that, Includes the following steps: S1. For an input sequence consisting of N characters Encode the input character representation sequence. Specifically, it includes: The input sequence is encoded using a pre-trained input sequence encoder and the encoder layer of a transformer model, denoted as . : The characters represent For each character of A dimensional vector used to capture contextual information for the corresponding tag; S2. The training sample label sequence is processed using a label encoder. Encode the sequence to obtain the label representation. ; S3. Will Projected as a query vector onto Obtain the same dimension ,pass Calculate the attention weights of the query vector ,pass The label background information is obtained by performing a weighted summation on the character representations. ; S4. Represent the input characters and marking background information As a comprehensive intermediate state, and further mapped to dimensional vector ,pass Get character representation Attention weights on the label representation sequence ,pass The tag context information is obtained by performing a weighted summation on the tag representations. ; S5. Add label background information With tag context information Connect them to obtain the final label-aware information. ,Will and corresponding label perception information and label representation Connecting them together yields the final sequence representation. : ； S6. Represent the final sequence. u The predicted label is obtained by mapping a linear layer onto the actual label Y, and then comparing the predicted label with the training sample label sequence in S2. The new label sequence is obtained by performing a connection; S7. Label the training sample sequence using the new label. Perform the update, repeating steps S2-S7.

2. The method for cross-domain named entity recognition based on an autoregressive model according to claim 1, characterized in that, The specific content of S2 includes: A randomly initialized tag lookup table is constructed using the tag encoder. Where K represents the number of unique tags in the source or target domain. R is the size of the tag embedding, and R is a randomly initialized embedding matrix; For a label , Using a label lookup table label Represented as The tag sequence is encoded by the tag encoder and represented as follows: in yes The output of the label encoder is used to capture the context information of the label.

3. The method for cross-domain named entity recognition based on an autoregressive model according to claim 2, characterized in that, The specific content of S3 includes: Will Projected onto Same dimensions: in yes dimensional vector; Calculate the attention weights of the query vector : Here, It is considered as a probability distribution and used to generate a weighted sum of the input character representations: Tag background information by This serves as a guide, indicating the relationship between the label of the current character and the entire input sequence.

4. The cross-domain named entity recognition method based on an autoregressive model according to claim 3, characterized in that, The specific content of S4 includes: Represent the input characters and the marked background information As a comprehensive intermediate state, and further mapped to Dimensional vector: pass The tag context information is obtained by performing a weighted summation on the tag representations: in Character representation exist A weighted vector on.

5. The cross-domain named entity recognition method based on an autoregressive model according to claim 4, characterized in that, The specific content of S5 includes: The label background information By connecting the context information to the input sequence, the predicted named entity labels are obtained, serving as the final label-aware information. : in , , , These are learnable parameters; Will and corresponding label perception information and label representation Connect them: in u It is the final sequence representation.

6. The cross-domain named entity recognition method based on an autoregressive model according to claim 5, characterized in that, S5 also includes: through trainable matrices and Bias will Mapped to output space The distribution of all named entity labels is obtained by using a normalization function.

Images