Span-based contrastive learning for cross-domain named entity recognition method, device, storage medium and product

Abstract

The span contrast learning-based cross-domain named entity recognition method, device, storage medium and product belong to the technical field of named entity recognition, and solve the problems of field deviation and low span boundary information learning performance. The method comprises the following steps: constructing a span contrast learning-based cross-domain named entity recognition model by using pre-training language model, adversarial training, contrast learning, KL divergence and other technologies; converting the sequential label into a global boundary matrix by using entity boundary information; completing span-level contrast learning and KL divergence calculation; and constantly changing the related parameters of the model during the training process, fully considering the entity boundary information, learning more field invariance information, and greatly improving the performance of cross-domain named entity recognition. The present application is suitable for cross-domain named entity recognition.

Description

Technical Field

[0001] This application relates to the field of named entity recognition technology, and in particular to cross-domain named entity recognition. Background Technology

[0002] Named Entity Recognition (NER) is a task that automatically identifies named entities in text, such as names of people, places, and organizations, and classifies them into different types. Deep learning models have surpassed traditional machine learning methods in terms of feature extraction depth and model performance, but they require a large amount of labeled data. In certain specific domains or for languages ​​with limited data resources, NER tasks are difficult to solve. Domain adaptation is an important part of transfer learning, and the domain shift problem is a common issue in domain adaptation. Domain shift refers to the phenomenon where a model's performance degrades when transferring from one domain to another because the training and test sets do not share the same basic distribution.

[0003] Extensive algorithms have been proposed to mitigate the domain offset problem, such as Domain Adversarial Neural Networks (DANNs) and distribution matching. However, these algorithms all suffer from several drawbacks. For DANNs, the instability of the joint optimization training process necessitates extensive hyperparameter tuning. Furthermore, distribution matching algorithms struggle to maintain the model's discriminative ability towards the target task while attempting instance-level alignment. Therefore, a stable and efficient solution is needed to learn both domain invariance and instance matching capabilities through unsupervised domain adaptation.

[0004] In recent research on self-supervised learning (SSL), contrastive learning (CL) has proven to be an effective method, leveraging data agent tasks from the original learning process to define representations at the instance level that facilitate learning. From the perspective of domain adaptation, constructing cross-domain positive samples and aligning domain-cognition has received relatively little attention in the relevant literature. Previous work has focused on designing label-preserving text transformations, such as backtranslation, synonyms, omissions, and their combinations. Summary of the Invention

[0005] The purpose of this invention is to address the problem of low learning performance of domain offset and span boundary information in cross-domain named entity recognition, and to provide a method, device, storage medium and product for cross-domain named entity recognition based on span comparison learning.

[0006] This invention is achieved through the following technical solution. In one aspect, this invention provides a cross-domain named entity recognition method based on span-contrast learning, the method comprising:

[0007] Step 1: Obtain the source domain and target domain datasets, preprocess the datasets, and divide them into training and test sets;

[0008] Step 2: Construct a cross-domain named entity recognition model based on span contrastive learning, specifically including:

[0009] Step 2.1: Obtain the embedded representations of the source domain data and the target domain data, and assign corresponding domain labels to the source domain and the target domain;

[0010] Step 2.2: Construct domain confusion enhancement samples. Embed the source domain and target domain data obtained in Step 2.1 into the pre-trained language model BERT, and use the projective gradient descent (PGD) method to generate adversarial samples. Perform domain classification through adversarial attacks.

[0011] Step 2.3: Generate the global boundary prediction matrix, which specifically includes:

[0012] The source domain embedding is input into BERT, and the resulting output is used to construct a global boundary prediction matrix using Global Pointer; the source domain embedding and the domain confusion enhancement sample embedding generated in step 2.2 are concatenated and input into BERT, and the resulting output is used to construct a global boundary prediction matrix with adversarial perturbation using Global Pointer;

[0013] Step 3: Train the cross-domain named entity recognition model based on span contrast learning from Step 2, which specifically includes the following steps:

[0014] Step 3.1: Using the global boundary prediction matrix obtained from source domain embedding in Step 2.3, calculate the loss of named entity recognition in the source domain using the cross-entropy loss function;

[0015] Step 3.2: Calculate the contrastive learning loss by using the similarity and dissimilarity of the vectors of all entity spans contained in the two global boundary prediction matrices obtained in Step 2.3;

[0016] Step 3.3: Calculate the KL divergence loss for all entity spans contained in the two global boundary prediction matrices obtained in Step 2.3, so that the generated adversarial examples are more consistent with the distribution predicted by the model itself.

[0017] Step 3.4: Update the model parameters by combining the loss functions from steps 3.1, 3.2, and 3.3 to optimize the joint loss function and train the optimal cross-domain named entity recognition model based on span contrast learning.

[0018] Step 4: Input the target domain test set into the cross-domain named entity recognition model based on span contrast learning, which has been trained, updated and optimized in Step 3, and calculate the score of the target domain entity.

[0019] Further, step 2.2 specifically includes:

[0020] Suppose there is a source dataset D containing n labeled data. S ={x i ,y i} 1,...,n , where x i It is a token sequence, y i It is x i The labels are provided, and the data in the source dataset are obtained from independent and identically distributed sampling from the source domain.

[0021] Target dataset D containing m unlabeled data points T ={x j} 1,...,m , where x j It is a token sequence, and the data in the target dataset is obtained by independent and identically distributed sampling from the target domain;

[0022] The model aims to learn the function f(x; θ) f ,θ y ):x→C, the input of this function is a sequence of tokens, and the output is the corresponding label; where θ f These are the parameters of the pre-trained language model, θ y These are the parameters for category label prediction, where C is the label set;

[0023] This is the model's loss in the classification task. In classification tasks, the goal of model learning is to minimize this loss. The specific formula is as follows:

[0024]

[0025] In the formula: This indicates that both the sequence and the label come from the source domain; within a single domain, adversarial training is an adversarial problem aimed at maximizing the internal loss and minimizing the external loss.

[0026]

[0027] In the formula: δ represents the generated adversarial sample;

[0028]

[0029]

[0030] Where, α adv Used to control the trade-off between the two types of loss, it is usually set to 1;

[0031] The following iterative steps can generate adversarial perturbations;

[0032]

[0033] In the formula, ∈ represents the upper bound of the adversarial perturbation, η represents the adversarial step size, and δ represents the upper bound of the adversarial perturbation. t The adversarial examples generated for the current iteration step. Let represent the gradient of the loss of the classification task at time t with respect to the input at time t. This is the gradient formula. This means that if the disturbance exceeds the range ∈, it must be mapped back to the specified range, ||·|| F Represents the L-infinity norm;

[0034] Generate adversarial examples with domain obfuscation:

[0035]

[0036]

[0037] Here, a domain-specific loss is used to learn the domain classifier using adversarial attacks, where δ0 is the initialized adversarial sample, and θ d These are parameters corresponding to the calculation of domain classification, where d is the domain label; the perturbation δ, f(x+δ; θ) is synthesized by searching the extreme directions that most trouble the domain classifier in the embedding space. f These are domain puzzles created using pre-trained language models. Let represent the gradient of the loss of the domain classification task at time t with respect to the input at time t.

[0038] Furthermore, step 2.3 specifically includes:

[0039] Assume S = [s1, s2, ..., s m [] represents the possible span in the sentence; the span s is denoted as s[i:j], where i and j are the head index and tail index, respectively; the goal of named entity recognition is to identify all s∈E, where E is the set of entity types; given a sentence X=[x1,x2,…x] with n tags. n First, each token in X is associated with its corresponding representation in a pre-trained language model to obtain the sentence representation matrix. Where v is the dimension:

[0040] h1, h2, ... h n =BERT(x1,x2,…x n )

[0041] After obtaining the sentence representation H, the span representation can be computed using two feedforward layers, which depend on the start and end indices of the span:

[0042] q i,α =W q,α h i +b q,α

[0043] k j,α =W k, αh j +b k,α

[0044] In the formula: It is a vector representation used to identify entity tokens of type α, q i,α k j,α W represents the start and end positions of the span s[i:j] of type α. q,α W k,α It is h i and h j The weight, b q,α b k,α As a bias value, the score for span s[i:j] belonging to type α is calculated as follows:

[0045]

[0046] The scoring function for each span is calculated, and the global boundary prediction matrix is ​​then generated using the scoring function.

[0047]

[0048] In the formula: and All are orthogonal matrices.

[0049] Further, step 3.1 specifically includes:

[0050] Calculate the score for each entity using the scoring function obtained in step 2.4;

[0051] Set the cross-entropy loss function as follows:

[0052]

[0053] In the formula: q and k represent the start and end indices of the span, respectively, and P α Q represents the set of spans of entity type α. α s represents the set of spans that are not entities or whose entity type is not α. α (q,k) are the fractions of entities of type α, satisfying s α The segment (q,k)>0 is the output of an entity of type α.

[0054] Further, in step 3.2, for an input sentence, each entity span is represented as a vector, and the similarity and dissimilarity of the vectors of all entity spans contained therein are calculated to calculate the contrast loss;

[0055] The loss function calculation process for contrastive learning is as follows:

[0056]

[0057] In the formula: N is the maximum length of the sentence, M is the number of negative examples, and span(i,j) represents the span. + This is a positive example of the current sentence, representing data augmentation for adversarial training using source domain data, span(i,j). - It is a negative example of the current sentence, that is, a span different from the current token label. The distance between the original sample and the positive and negative samples is calculated using cosine similarity.

[0058] Furthermore, in step 3.3, the loss function for the KL divergence is calculated as follows:

[0059]

[0060] Further, step 3.4 specifically includes:

[0061] An end-to-end neural network model is used for overall training, which includes four loss functions: source domain named entity recognition task loss, domain classifier loss, contrastive learning loss, and KL divergence loss.

[0062] These loss functions are summed to obtain the loss of the cross-domain named entity recognition model based on span contrast learning, and these loss functions are trained jointly.

[0063]

[0064] In the formula: α, λ, β are hyperparameters used to control the weights of various losses.

[0065] Secondly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, it performs the steps of a cross-domain named entity recognition method based on span contrast learning as described above.

[0066] Thirdly, the present invention provides a computer-readable storage medium storing a plurality of computer instructions, the plurality of computer instructions being used to cause a computer to execute a cross-domain named entity recognition method based on span contrast learning as described above.

[0067] Fourthly, the present invention provides a computer program product, which, when executed by a processor, implements a cross-domain named entity recognition method based on span contrast learning as described above.

[0068] The beneficial effects of this invention are:

[0069] This invention addresses the problem of cross-domain named entity recognition, overcomes the shortcomings of existing technologies, and utilizes techniques such as pre-trained language models, adversarial training, and contrastive learning to fully consider and mine domain-invariant features, providing a cross-domain named entity recognition method based on cross-domain contrastive learning.

[0070] 1. By introducing entity boundary information, sequential labels are transformed into a global boundary matrix, which represents the sentence-level target labels, enabling the model to learn explicit span boundary information. In cross-domain learning, both distribution matching and instance-based matching have certain limitations, while contrastive learning can learn domain invariance even when the target domain has no labels.

[0071] 2. Adversarial learning enables the model to learn knowledge of domain confusion, and contrastive learning is used to reduce the domain bias problem.

[0072] 3. At the same time, KL divergence is used to learn a distribution that is close to the model's own prediction, further improving the model's performance.

[0073] This invention is applicable to cross-domain named entity recognition. Attached Figure Description

[0074] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 A flowchart of a cross-domain named entity recognition method based on span contrast learning;

[0076] Figure 2 Model diagram of a cross-domain named entity recognition method based on span contrastive learning;

[0077] Figure 3 Sample image of the domain puzzle;

[0078] Figure 4 A diagram illustrating nested entities identified by multiple heads;

[0079] Figure 5 Schematic diagram of positive sampling;

[0080] Figure 6 Schematic diagram of negative sampling. Detailed Implementation

[0081] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0082] Specific Implementation Method 1: A cross-domain named entity recognition method based on span comparison learning, the method comprising:

[0083] Step 1: Obtain the source domain and target domain datasets, preprocess the datasets, and divide them into training and test sets;

[0084] Step 2: Construct a cross-domain named entity recognition model based on span contrastive learning, specifically including:

[0085] Step 2.1: Obtain the embedded representations of the source domain data and the target domain data, and assign corresponding domain labels to the source domain and the target domain;

[0086] Step 2.2: Construct domain confusion enhancement samples. Embed the source domain and target domain data obtained in Step 2.1 into the pre-trained language model BERT, and use the projective gradient descent (PGD) method to generate adversarial samples. Perform domain classification through adversarial attacks.

[0087] Step 2.3: Generate the global boundary prediction matrix, specifically including:

[0088] The source domain embedding is input into BERT, and the resulting output is used to construct a global boundary prediction matrix using Global Pointer; the source domain embedding and the domain confusion enhancement sample embedding generated in step 2.2 are concatenated and input into BERT, and the resulting output is used to construct a global boundary prediction matrix with adversarial perturbation using Global Pointer;

[0089] Step 3: Train the cross-domain named entity recognition model based on span contrast learning from Step 2, which specifically includes the following steps:

[0090] Step 3.1: Using the global boundary prediction matrix obtained from source domain embedding in Step 2.3, calculate the loss of named entity recognition in the source domain using the cross-entropy loss function;

[0091] Step 3.2: Calculate the contrastive learning loss by using the similarity and dissimilarity of the vectors of all entity spans contained in the two global boundary prediction matrices obtained in Step 2.3;

[0092] Step 3.3: Calculate the KL divergence loss for all entity spans contained in the two global boundary prediction matrices obtained in Step 2.3, so that the generated adversarial examples are more consistent with the distribution predicted by the model itself.

[0093] Step 3.4: Update the model parameters by combining the loss functions from steps 3.1, 3.2, and 3.3 to optimize the joint loss function and train the optimal cross-domain named entity recognition model based on span contrast learning.

[0094] Step 4: Input the target domain test set into the cross-domain named entity recognition model based on span contrast learning, which has been trained, updated and optimized in Step 3, and calculate the score of the target domain entity.

[0095] In this embodiment, to address the problem of cross-domain named entity recognition, the shortcomings of existing technologies are overcome by using techniques such as pre-trained language models, adversarial training, and contrastive learning to fully consider and explore domain-invariant features, and to provide a cross-domain named entity recognition method based on cross-domain contrastive learning.

[0096] First, entity boundary information is introduced, transforming sequential labels into a global boundary matrix. This global boundary matrix represents the sentence-level target labels, enabling the model to learn explicit span boundary information. In cross-domain learning, both distribution matching and instance-based matching have certain limitations, while contrastive learning can learn domain invariance even when the target domain lacks labels.

[0097] Secondly, adversarial learning enables the model to learn knowledge of domain confusion, and contrastive learning is used to reduce the domain bias problem.

[0098] Furthermore, by using KL divergence to learn a distribution that closely approximates the model's own predictions, the model's performance is further improved.

[0099] Specific implementation method two is a further limitation on the cross-domain named entity recognition method based on span contrast learning described in implementation method one. In this implementation method, step 2.2 is further limited, specifically including:

[0100] Step 2.2 specifically includes:

[0101] Suppose there is a source dataset D containing n labeled data. S ={x i ,y i} 1,…,n , where x i It is a token sequence, y i It is x i The labels are provided, and the data in the source dataset are obtained from independent and identically distributed sampling from the source domain.

[0102] Target dataset D containing m unlabeled data points T ={x j} 1,…,m , where x j It is a token sequence, and the data in the target dataset is obtained by independent and identically distributed sampling from the target domain;

[0103] The model aims to learn the function f(x; θ) f ,θ y ):x→C, the input of this function is a sequence of tokens, and the output is the corresponding label; where θ f These are the parameters of the pre-trained language model, θ y These are the parameters for category label prediction, where C is the label set;

[0104] This is the model's loss in the classification task. In classification tasks, the goal of model learning is to minimize this loss. The specific formula is as follows:

[0105]

[0106] In the formula: This indicates that both the sequence and the label come from the source domain; within a single domain, adversarial training is an adversarial problem aimed at maximizing the internal loss and minimizing the external loss.

[0107]

[0108] In the formula: δ represents the generated adversarial sample;

[0109]

[0110]

[0111] Where, α adv Used to control the trade-off between the two types of loss, it is usually set to 1;

[0112] The following iterative steps can generate adversarial perturbations;

[0113]

[0114] In the formula, ∈ represents the upper bound of the adversarial perturbation, η represents the adversarial step size, and δ represents the upper bound of the adversarial perturbation. t The adversarial examples generated for the current iteration step. Let represent the gradient of the loss of the classification task at time t with respect to the input at time t. This is the gradient formula. This means that if the disturbance exceeds the range ∈, it must be mapped back to the specified range, ||·|| F Represents the L-infinity norm;

[0115] Generate adversarial examples with domain obfuscation:

[0116]

[0117]

[0118] Here, a domain-specific loss is used to learn the domain classifier using adversarial attacks, where δ0 is the initialized adversarial sample, and θ d These are parameters corresponding to the calculation of domain classification, where d is the domain label; the perturbation δ, f(x+δ; θ) is synthesized by searching the extreme directions that most trouble the domain classifier in the embedding space. f These are domain puzzles created using pre-trained language models. Let represent the gradient of the loss of the domain classification task at time t with respect to the input at time t.

[0119] In this implementation, internal maximization can be achieved using Projected Gradient Descent (PGD), assuming the loss function is locally linear. The advantage of PGD is that it relies solely on the model itself and can generate samples with varying adversarial strengths, thereby improving the model's generalization ability on unseen data. PGD employs a small-step, multi-step strategy for adversarial processing. Specifically, it performs forward and backward propagation repeatedly, adjusting the loss function according to g... adv The perturbation is calculated, and new adversarial perturbations δ are repeatedly added to the embedding layer g. adv If the value exceeds a certain range, it is mapped back to the given range. Finally, the g calculated in the last step is... adv This is accumulated onto the original gradient. That is, the gradient corresponding to the gradient after t steps of perturbation is accumulated as g. adv Update the original gradient.

[0120] like Figure 3 As shown, domain puzzles can enhance the domain invariance of a model, enabling it to better adapt to unknown data and domains. During training, the model is confused by discarding domain-related information, making it difficult to distinguish data from different domains, and bringing the source (target) data and its corresponding domain puzzles closer together to reduce domain differences.

[0121] Specific implementation method three is a further limitation on the cross-domain named entity recognition method based on span contrast learning described in implementation method one. In this implementation method, step 2.3 is further limited, specifically including:

[0122] Step 2.3 specifically includes:

[0123] Assume S = [s1, s2, ..., s m[] represents the possible span in the sentence; the span s is denoted as s[i:j], where i and j are the head index and tail index, respectively; the goal of named entity recognition is to identify all s∈E, where E is the set of entity types; given a sentence X=[x1,x2,…x] with n tags. n First, each token in X is associated with its corresponding representation in a pre-trained language model to obtain the sentence representation matrix. Where v is the dimension:

[0124] h1, h2, ... h n =BERT(x1,x2,…x n )

[0125] After obtaining the sentence representation H, the span representation can be computed using two feedforward layers, which depend on the start and end indices of the span:

[0126] q i,α =W q,α h i +b q,α

[0127] k j,α =W k,α h j +b k,α

[0128] In the formula: It is a vector representation used to identify entity tokens of type α, q i,α k j,α W represents the start and end positions of the span s[i:j] of type α. q,α W k,α It is h i and h j The weight, b q,α b k,α As a bias value, the score for span s[i:j] belonging to type α is calculated as follows:

[0129]

[0130] The scoring function for each span is calculated, and the global boundary prediction matrix is ​​then generated using the scoring function.

[0131]

[0132] In the formula: and All are orthogonal matrices.

[0133] In this embodiment, such as Figure 4 As shown, multi-head identification of nested entities is to generate all possible entity spans.

[0134] In attention mechanisms, positional encoding comes in two forms: absolute positional encoding and relative positional encoding. While absolute positional encoding can add positional information to word vectors, this positional information is associated with a fixed position and cannot represent the contextual information of that fixed position. To utilize boundary information, [the following is missing from the original text]... Rotational position encoding is applied to entity representation, making the model more sensitive to the relative positions between entities and thus improving entity recognition performance. In this way, scoring functions for each span can be calculated, and these scoring functions are then used to generate a global boundary prediction matrix.

[0135] Specific implementation method four is a further limitation on the cross-domain named entity recognition method based on span contrast learning described in implementation method one. In this implementation method, step 3.1 is further limited, specifically including:

[0136] Step 3.1 specifically includes:

[0137] Calculate the score for each entity using the scoring function obtained in step 2.4;

[0138] Set the cross-entropy loss function as follows:

[0139]

[0140] In the formula: q and k represent the start and end indices of the span, respectively, and P α Q represents the set of spans of entity type α. α s represents the set of spans that are not entities or whose entity type is not α. α (q,k) are the fractions of entities of type α, satisfying s α The segment (q,k)>0 is the output of an entity of type α.

[0141] In this embodiment, the score for each entity is calculated using the scoring function obtained in step 2.4. Furthermore, to address the class imbalance problem in classification, a cross-entropy loss function is designed to encourage the model to learn the boundary information of each training support instance.

[0142] Specific implementation method five is a further limitation on the cross-domain named entity recognition method based on span contrast learning described in implementation method four. In this implementation method, step 3.2 is further limited, specifically including:

[0143] In step 3.2, for an input sentence, each entity span is represented as a vector, and the similarity and dissimilarity of the vectors of all entity spans contained therein are calculated to calculate the contrast loss;

[0144] The loss function calculation process for contrastive learning is as follows:

[0145]

[0146] In the formula: N is the maximum length of the sentence, M is the number of negative examples, and span(i,j) represents the span. + This is a positive example of the current sentence, representing data augmentation for adversarial training using source domain data, span(i,j). - It is a negative example of the current sentence, that is, a span different from the current token label. The distance between the original sample and the positive and negative samples is calculated using cosine similarity.

[0147] In this implementation, contrastive learning is used to bring spans similar to the model closer together and widen the gap between spans dissimilar to the model at the span level, enabling the model to learn more entity span invariant information; such as Figure 5 As shown, for positive sampling, the model can encode the span of the source domain and the span of the domain puzzle as being closer in the representation space, gradually pulling the examples closer to the domain decision boundary as training progresses; for cross-domain negative sampling, contrastive loss may push the negative samples of the source and target domains apart, such as... Figure 6 In the left half of the negative sample set, negative samples of the same category across different domains are far apart, which contradicts the goal of bringing different domains closer together. Therefore, to avoid this cross-domain exclusion, samples from different domains are excluded from the negative sample set.

[0148] By bringing similar spans closer together and distancing dissimilar spans further apart at the span level, the model can learn more entity span invariant information.

[0149] Specific implementation method six is ​​a further limitation on the cross-domain named entity recognition method based on span contrast learning described in implementation method five. In this implementation method, step 3.3 is further limited, specifically including:

[0150] In step 3.3, the loss function calculation process for KL divergence is as follows:

[0151]

[0152] In this embodiment, the loss of KL divergence is calculated by using the span of all entities contained in the two global boundary prediction matrices obtained in step 2.3, so that the generated adversarial examples are more consistent with the distribution predicted by the model itself.

[0153] In adversarial training, to make the model more robust to adversarial examples, the training data is perturbed to generate adversarial examples. These adversarial examples differ from the original samples and may introduce noise or perturbations. To ensure that the generated adversarial examples have a certain degree of similarity and continuity, a latent variable is usually introduced to control the distance between samples. During the generation of adversarial examples, it is desirable for the generated samples to approximate the distribution predicted by the model itself, thereby making the model more robust. The KL distribution is a commonly used metric for comparing the differences between two distributions; the smaller the KL divergence, the closer the two distributions are. Therefore, in adversarial training, the quality of the generated adversarial examples is evaluated by calculating the KL distribution between the distribution of the generated adversarial examples and the model's predicted distribution on the original samples, so that the generated adversarial examples are more consistent with the distribution predicted by the model itself. This way, the generated adversarial examples can be better used to train the model and improve its robustness. Therefore, the method in this embodiment is used for calculating the loss function of KL divergence.

[0154] Specific implementation method seven is a further limitation on the cross-domain named entity recognition method based on span contrast learning described in implementation method six. In this implementation method, step 3.4 is further limited, specifically including:

[0155] Step 3.4 specifically includes:

[0156] An end-to-end neural network model is used for overall training, which includes four loss functions: source domain named entity recognition task loss, domain classifier loss, contrastive learning loss, and KL divergence loss.

[0157] These loss functions are summed to obtain the loss of the cross-domain named entity recognition model based on span contrast learning, and these loss functions are trained jointly.

[0158]

[0159] In the formula: α, λ, β are hyperparameters used to control the weights of various losses.

[0160] In this embodiment, these loss functions are summed to obtain the loss of the cross-domain named entity recognition model based on span contrastive learning. Through joint training of these loss functions, multiple aspects of the model can be optimized, thereby improving its performance and robustness.

[0161] By training end-to-end and optimizing this loss function, we can obtain a robust model that has good classification performance on the original samples and is also effective against high-quality adversarial samples.

[0162] Specific implementation method eight, this implementation method is embodiment 1 of the cross-domain named entity recognition method based on span contrast learning as described above, specifically including:

[0163] The cross-domain named entity recognition method based on span comparison learning designed in this embodiment is implemented through the following steps:

[0164] Step 1: Obtain the source and target domain datasets, preprocess the datasets, and divide them into training and test sets;

[0165] Step 2: Construct a cross-domain named entity recognition model based on span contrastive learning, which includes the following steps:

[0166] Step 2.1: Obtain the embedded representations of the source domain data and the target domain data, and assign corresponding domain labels to the source domain and the target domain;

[0167] Step 2.2: Construct domain confusion enhancement samples. Embed the source and target domain data obtained in Step 2.1 into the pre-trained language model BERT, and use the Projective Gradient Descent (PGD) method to generate adversarial samples for domain classification through adversarial attacks;

[0168] Step 2.3: Generate the global boundary prediction matrix. Input the source domain embedding into BERT, and use the output to construct the global boundary prediction matrix using Global Pointer; concatenate the source domain embedding and the domain confusion enhancement sample embedding generated in Step 2.2 using the concat() function, and input them into BERT, and use the output to construct the global boundary prediction matrix with adversarial perturbation using Global Pointer;

[0169] Step 3: Train the cross-domain named entity recognition model based on span contrast learning from Step 2, which specifically includes the following steps:

[0170] Step 3.1: Using the global boundary prediction matrix obtained from source domain embedding in Step 2.3, calculate the loss of named entity recognition in the source domain using the cross-entropy loss function;

[0171] Step 3.2: Calculate the contrastive learning loss by using the similarity and dissimilarity of the vectors of all entity spans contained in the two global boundary prediction matrices obtained in Step 2.3;

[0172] Step 3.3: Calculate the KL divergence loss using the spans of all entities contained in the two global boundary prediction matrices obtained in Step 2.3, so that the generated adversarial examples are more consistent with the distribution predicted by the model itself;

[0173] Step 3.4: Update the model parameters by combining the loss functions from steps 3.1, 3.2, and 3.3 to optimize the joint loss function and train the optimal cross-domain named entity recognition model based on span contrast learning.

[0174] Step 4: Input the target domain test set into the cross-domain named entity recognition model based on span contrast learning, which has been trained, updated and optimized in Step 3, and calculate the score of the target domain entity.

[0175] Specific implementation method eight, this implementation method is embodiment 2 of the cross-domain named entity recognition method based on span contrast learning as described above, specifically including:

[0176] like Figures 1 to 2 As shown, this invention provides a cross-domain named entity recognition method based on span contrast learning, specifically including the following steps:

[0177] Step 1: Obtain the source and target domain datasets, preprocess the datasets, and divide them into training and test sets;

[0178] Step 1 specifically includes the following steps:

[0179] Step 1.1: Extract text sequences from the source and target domain datasets;

[0180] Step 1.2: Divide the preprocessed dataset into a training set and a test set;

[0181] Step 2: Construct a cross-domain named entity recognition model based on span contrast learning.

[0182] Step 2 specifically includes the following steps:

[0183] Step 2.1: Obtain the embedding representations of the source domain data and the target domain data. First, use independent-hot encoding to obtain the encoding of each label. Use the pre-trained language model BERT to generate source domain and target domain token embeddings, and number the two domains: 0 for the source domain and 1 for the target domain.

[0184] Step 2.2: Construct domain confusion enhancement samples. Embed the source and target domain data obtained in Step 2.1 into the pre-trained language model BERT, and use the Projective Gradient Descent (PGD) method to generate adversarial samples for domain classification through adversarial attacks;

[0185] In step 2.2, assume there is a source dataset D containing n labeled data. S ={x i ,y i} 1,…,nThese data were obtained by independent and identically distributed sampling from the source domain. Additionally, there is a target dataset D containing m unlabeled data points. T ={x j} 1,…,m These data were obtained from independent and identically distributed sampling in the target domain, where x i x j It is a token sequence, y i It is x i The model aims to learn a function f(x; θ) that takes a sequence of tokens as input and outputs the corresponding labels. During domain-specific training, the model learns this function. f ,θ y ):x→C, where θ f These are the parameters of the pre-trained language model, θ y These are the parameters for class label prediction, and C is the label set. In general classification tasks, This is the model's loss in the classification task. In classification tasks, the goal of model learning is to minimize this loss. The specific formula is as follows:

[0186]

[0187] In the formula: This indicates that both the sequence and the label originate from the source domain. Within a single domain, adversarial training is an adversarial problem aimed at maximizing the internal loss and minimizing the external loss.

[0188]

[0189] In the formula: δ is the adversarial sample we generate.

[0190]

[0191]

[0192] In the formula: α adv This is used to control the trade-off between the two losses, and is usually set to 1. Internal maximization can be achieved using Projective Gradient Descent (PGD), assuming the loss function is locally linear. The advantage of PGD is that it relies solely on the model itself and can generate samples with varying adversarial characteristics, thereby improving the model's generalization ability on unseen data. PGD employs a small-step, multi-step strategy for adversarial processing. Specifically, it performs forward and backward propagation repeatedly, adjusting the loss function according to g... adv The perturbation is calculated, and new adversarial perturbations δ are repeatedly added to the embedding layer g. adv If the value exceeds a certain range, it is mapped back to the given range. Finally, the g calculated in the last step is... advThis is accumulated onto the original gradient. That is, the gradient corresponding to the gradient after t steps of perturbation is accumulated as g. adv Update the original gradient. The following iterative steps can generate adversarial perturbations.

[0193]

[0194]

[0195] In the formula: ∈ represents the upper bound of the adversarial perturbation, η represents the adversarial step size, and δ represents the upper bound of the adversarial perturbation. t The adversarial examples generated for the current iteration step. Let represent the gradient of the loss of the classification task at time t with respect to the input at time t. This is the gradient formula. Π ||δ||F≤∈ This means that if the disturbance exceeds the range ∈, it must be mapped back to the specified range, ||·|| F This represents the L-infinity norm.

[0196] like Figure 3 As shown, domain puzzles can enhance the domain invariance of a model, enabling it to better adapt to unknown data and domains. During training, domain-related information is discarded to obfuscate the model, making it difficult to distinguish data from different domains and bringing the source (target) data and its corresponding domain puzzles closer together, thus reducing domain differences. To generate domain obfuscation enhancements, we employ adversarial attacks with perturbations for source and target domain classification tasks. Using the above process for generating adversarial examples, we can generate adversarial examples with domain obfuscation:

[0197]

[0198]

[0199] Here, a domain-specific loss is used to learn the domain classifier using adversarial attacks, where δ0 is the initialized adversarial sample, and θ d These are the parameters corresponding to the calculation of the domain classification, where d is the domain label. The perturbation δ, f(x+δ; θ) is synthesized by searching the extreme directions that most trouble the domain classifier in the embedding space. f ) is a domain puzzle created using a pre-trained language model. Let represent the gradient of the loss of the domain classification task at time t with respect to the input at time t.

[0200] Step 2.3: Generate the global boundary prediction matrix. Input the source domain embedding into BERT, and use the output to construct the global boundary prediction matrix using Global Pointer; concatenate the source domain embedding and the domain confusion enhancement sample embedding generated in Step 2.2 using the concat() function, and input them into BERT, and use the output to construct the global boundary prediction matrix with adversarial perturbation.

[0201] like Figure 4 As shown, multi-head identification of nested entities is to generate all possible entity spans. In step 2.3, assume S = [s1, s2, ..., s...]. m The span s represents the possible spans in the sentence. The span s is denoted as s[i:j], where i and j are the head and tail indices, respectively. The goal of named entity recognition is to identify all s∈E, where E is the set of entity types. Given a sentence X = [x1, x2, ... xj] with n tags... n First, each token in X is associated with its corresponding representation in the pre-trained language model, thus obtaining a new latent vector output matrix. Where v is the dimension:

[0202] h1, h2, ... h n =BERT(x1,x2,…x n )

[0203] After obtaining the sentence representation H, the span representation can be computed using two feedforward layers, which depend on the start and end indices of the span:

[0204] q i,α =W q,α h i +b q,α

[0205] k j,α =W k,α h j +b k,α

[0206] In the formula: It is a vector representation used to identify entity tokens of type α, q i,α k j,α W represents the start and end positions of the span s[i:j] of type α. q,α W k,α It is h i and h j The weight, b q,α b k,α As a bias value, the score for span s[i:j] belonging to type α can be calculated as follows:

[0207]

[0208] In attention mechanisms, positional encoding comes in two forms: absolute positional encoding and relative positional encoding. While absolute positional encoding can add positional information to word vectors, this positional information is associated with a fixed position and cannot represent the contextual information of that fixed position. To utilize boundary information, [the following is missing from the original text]... Rotational position encoding is applied to entity representation, making the model more sensitive to the relative positions between entities and thus improving entity recognition performance. In this way, a scoring function for each span can be calculated, which is then used to generate a global boundary prediction matrix.

[0209]

[0210] In the formula: and All are orthogonal matrices.

[0211] Step 3: Train the cross-domain named entity recognition model based on span contrast learning from Step 2, which specifically includes the following steps:

[0212] Step 3.1: Using the global boundary prediction matrix obtained from source domain embedding in Step 2.3, calculate the loss of named entity recognition in the source domain using the cross-entropy loss function;

[0213] The score for each entity is calculated using the scoring function obtained in step 2.4. Furthermore, to address the class imbalance problem in classification, a cross-entropy loss function is designed to encourage the model to learn the boundary information of each training support instance.

[0214]

[0215] In the formula: q and k represent the start and end indices of the span, respectively, and P α Q represents the set of spans of entity type α. α s represents the set of spans that are not entities or whose entity type is not α. α (q,k) are the fractions of entities of type α, satisfying s α The segment (q,k)>0 is the output of an entity of type α.

[0216] Step 3.2: Calculate the contrastive learning loss by using the similarity and dissimilarity of the vectors of all entity spans contained in the two global boundary prediction matrices obtained in Step 2.3;

[0217] In step 3.2, contrastive learning is used to bring similar spans closer together and widen the gaps between dissimilar spans at the span level, enabling the model to learn more entity span invariant information; such as Figure 5As shown, for positive sampling, the model can encode the span of the source domain and the span of the domain puzzle as being closer in the representation space, gradually pulling the examples closer to the domain decision boundary as training progresses; for cross-domain negative sampling, contrastive loss may push the negative samples of the source and target domains apart, such as... Figure 6 In the left half of the negative sample set, negative samples of the same category across different domains are far apart, which contradicts the goal of bringing different domains closer together. Therefore, to avoid this cross-domain exclusion, samples from different domains are excluded from the negative sample set.

[0218] At the span level, the model brings similar spans closer together and distances dissimilar spans further apart, enabling it to learn more entity span invariant information. For an input sentence, each entity span is represented as a vector, and the similarity and dissimilarity of the vectors of all entity spans it contains are calculated to determine the contrastive loss. Therefore, the loss function calculation process for contrastive learning in step 3.23.3 is as follows:

[0219]

[0220] In the formula: N is the maximum length of the sentence, M is the number of negative examples, and span(i,j) represents the span. + This is a positive example of the current sentence, representing data augmentation for adversarial training using source domain data, span(i,j). - It is a negative example of the current sentence, that is, a span different from the current token label. The distance between the original sample and the positive and negative samples is calculated using cosine similarity.

[0221] Step 3.3: Calculate the KL divergence loss using the spans of all entities contained in the two global boundary prediction matrices obtained in Step 2.3, so that the generated adversarial examples are more consistent with the distribution predicted by the model itself;

[0222] In adversarial training, to make the model more robust to adversarial examples, the training data is perturbed to generate adversarial examples. These adversarial examples differ from the original samples and may introduce noise or perturbations. To ensure that the generated adversarial examples have a certain degree of similarity and continuity, a latent variable is usually introduced to control the distance between samples. During the generation of adversarial examples, it is desirable for the generated samples to approximate the distribution predicted by the model itself, thereby making the model more robust. The KL distribution is a commonly used metric for comparing the differences between two distributions; the smaller the KL divergence, the closer the two distributions are. Therefore, in adversarial training, the quality of the generated adversarial examples is evaluated by calculating the KL distribution between the distribution of the generated adversarial examples and the model's predicted distribution on the original samples, so that the generated adversarial examples are more consistent with the distribution predicted by the model itself. This way, the generated adversarial examples can be better used to train the model and improve its robustness. Therefore, the calculation process of the loss function for KL divergence in step 3.33.4 is as follows:

[0223]

[0224] Step 3.4: Update the model parameters by combining the loss functions from steps 3.1, 3.2, and 3.3 to optimize the joint loss function and train the optimal cross-domain named entity recognition model based on span contrast learning.

[0225] An end-to-end neural network model is used for overall training, which includes four loss functions: the source domain named entity recognition task loss, the domain classifier loss, the contrastive learning loss, and the KL divergence loss. These loss functions are summed to obtain the loss of the cross-domain named entity recognition model based on span contrastive learning. Joint training with these loss functions allows for optimization of multiple aspects of the model, thereby improving its performance and robustness.

[0226]

[0227] In the formula: α, λ, β are hyperparameters used to control the weights of various losses.

[0228] By training end-to-end and optimizing this loss function, we can obtain a robust model that has good classification performance on the original samples and is also effective against high-quality adversarial samples.

[0229] Step 4: Input the target domain test set into the cross-domain named entity recognition model based on span contrast learning, which has been trained, updated and optimized in Step 3, and calculate the score of the target domain entity.

[0230] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate the technical solution of this invention, and the above specific embodiments are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.

Claims

1. A cross-domain named entity recognition method based on span contrastive learning, characterized in that, The method includes: Step 1: Obtain the source domain and target domain datasets, preprocess the datasets, and divide them into training and test sets; Step 2: Construct a cross-domain named entity recognition model based on span contrastive learning, specifically including: Step 2.1: Obtain the embedded representations of the source domain data and the target domain data, and assign corresponding domain labels to the source domain and the target domain; Step 2.2: Construct domain confusion enhancement samples. Embed the source domain and target domain data obtained in Step 2.1 into the pre-trained language model BERT, and use the projective gradient descent (PGD) method to generate adversarial samples. Perform domain classification through adversarial attacks. Step 2.3: Generate the global boundary prediction matrix, which specifically includes: The source domain embedding is input into BERT, and the resulting output is used to construct a global boundary prediction matrix using Global Pointer; the source domain embedding and the domain confusion enhancement sample embedding generated in step 2.2 are concatenated and input into BERT, and the resulting output is used to construct a global boundary prediction matrix with adversarial perturbation using Global Pointer; Step 3: Train the cross-domain named entity recognition model based on span contrast learning from Step 2, which specifically includes the following steps: Step 3.1: Using the global boundary prediction matrix obtained from source domain embedding in Step 2.3, calculate the loss of named entity recognition in the source domain using the cross-entropy loss function; Step 3.2: Calculate the contrastive learning loss by using the similarity and dissimilarity of the vectors of all entity spans contained in the two global boundary prediction matrices obtained in Step 2.3; Step 3.3: Calculate the KL divergence loss for all entity spans contained in the two global boundary prediction matrices obtained in Step 2.3, so that the generated adversarial examples are more consistent with the distribution predicted by the model itself. Step 3.4: Update the model parameters by combining the loss functions from steps 3.1, 3.2, and 3.3 to optimize the joint loss function and train the optimal cross-domain named entity recognition model based on span contrast learning. Step 4: Input the target domain test set into the cross-domain named entity recognition model based on span contrast learning, which has been trained, updated and optimized in Step 3, and calculate the score of the target domain entity.

2. The cross-domain named entity recognition method based on span contrastive learning according to claim 1, characterized in that, Step 2.2 specifically includes: Suppose there is a source dataset D containing n labeled data. S ={x i ,y i } 1,…,n , where x i It is a token sequence, y i It is x i The labels are provided, and the data in the source dataset are obtained from independent and identically distributed sampling from the source domain. Target dataset D containing m unlabeled data points T ={x j } 1,…,m , where x j It is a token sequence, and the data in the target dataset is obtained by independent and identically distributed sampling from the target domain; The model aims to learn the function f(x; θ) f ,θ y ):x→C, the input of this function is a sequence of tokens, and the output is the corresponding label; where θ f These are the parameters of the pre-trained language model, θ y These are the parameters for category label prediction, where C is the label set; This is the model's loss in the classification task. In classification tasks, the goal of model learning is to minimize this loss. The specific formula is as follows: In the formula: This indicates that both the sequence and the label come from the source domain; within a single domain, adversarial training is an adversarial problem aimed at maximizing the internal loss and minimizing the external loss. In the formula: δ represents the generated adversarial sample; Where, α adv Used to control the trade-off between the two types of loss, it is usually set to 1; The following iterative steps can generate adversarial perturbations; In the formula, ∈ represents the upper bound of the adversarial perturbation, η represents the adversarial step size, and δ represents the upper bound of the adversarial perturbation. t The adversarial examples generated for the current iteration step. Let represent the gradient of the loss of the classification task at time t with respect to the input at time t. This is the gradient formula. This means that if the disturbance exceeds the range ∈, it must be mapped back to the specified range, ||·|| F Represents the L-infinity norm; Generate adversarial examples with domain obfuscation: Here, a domain-specific loss is used to learn the domain classifier using adversarial attacks, where δ0 is the initialized adversarial sample, and θ d These are parameters corresponding to the calculation of domain classification, where d is the domain label; the perturbation δ, f(x+δ; θ) is synthesized by searching the extreme directions that most trouble the domain classifier in the embedding space. f These are domain puzzles created using pre-trained language models. Let represent the gradient of the loss of the domain classification task at time t with respect to the input at time t.

3. The cross-domain named entity recognition method based on span contrastive learning according to claim 1, characterized in that, Step 2.3 specifically includes: Assume S = [s1, s2, ..., s m [] represents the possible span in the sentence; the span s is denoted as s[i:j], where i and j are the head index and tail index, respectively; the goal of named entity recognition is to identify all s∈E, where E is the set of entity types; given a sentence X=[x1,x2,…x] with n tags. n First, each token in X is associated with its corresponding representation in a pre-trained language model to obtain the sentence representation matrix. Where v is the dimension: h1,h2,…h n =BERT(x1,x2,…x n ) After obtaining the sentence representation H, the span representation can be computed using two feedforward layers, which depend on the start and end indices of the span: q i，α =W q，α h i +b q，α k j，α =W k，α h j +b k，α In the formula: It is a vector representation used to identify entity tokens of type α, q i,α k j,α W represents the start and end positions of the span s[i:j] of type α. q,α W k,α It is h i and h j The weight, b q,α b k,α As a bias value, the score for span s[i:j] belonging to type α is calculated as follows: The scoring function for each span is calculated, and the global boundary prediction matrix is ​​then generated using the scoring function. In the formula: and All are orthogonal matrices.

4. The cross-domain named entity recognition method based on span contrastive learning according to claim 1, characterized in that, Step 3.1 specifically includes: Calculate the score for each entity using the scoring function obtained in step 2.4; Set the cross-entropy loss function as follows: In the formula: q and k represent the start and end indices of the span, respectively, and P α Q represents the set of spans of entity type α. α s represents the set of spans that are not entities or whose entity type is not α. α (q,k) are the fractions of entities of type α, satisfying s α The segment (q,k)>0 is the output of an entity of type α.

5. The cross-domain named entity recognition method based on span contrastive learning according to claim 4, characterized in that, In step 3.2, for an input sentence, each entity span is represented as a vector, and the similarity and dissimilarity of the vectors of all entity spans contained therein are calculated to calculate the contrast loss; The loss function calculation process for contrastive learning is as follows: In the formula: N is the maximum length of the sentence, M is the number of negative examples, and span(i,j) represents the span. + This is a positive example of the current sentence, representing data augmentation for adversarial training using source domain data, span(i,j). - It is a negative example of the current sentence, that is, a span different from the current token label. The distance between the original sample and the positive and negative samples is calculated using cosine similarity.

6. The cross-domain named entity recognition method based on span contrastive learning according to claim 5, characterized in that, In step 3.3, the loss function calculation process for KL divergence is as follows:

7. The cross-domain named entity recognition method based on span contrastive learning according to claim 6, characterized in that, Step 3.4 specifically includes: An end-to-end neural network model is used for overall training, which includes four loss functions: source domain named entity recognition task loss, domain classifier loss, contrastive learning loss, and KL divergence loss. These loss functions are summed to obtain the loss of the cross-domain named entity recognition model based on span contrast learning, and these loss functions are trained jointly. In the formula: α, λ, β are hyperparameters used to control the weights of various losses.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, The steps of the method according to any one of claims 1 to 7 are performed when the processor runs the computer program stored in the memory.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of computer instructions, which are used to cause a computer to perform the method of any one of claims 1 to 7.

10. A computer program product, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images