Cross-domain named entity recognition method based on knowledge sharing
The cross-domain NER method addresses sparse annotation issues by using a cross-self-attention encoder and entropy regularization to enhance feature exchange and label prediction, improving recognition accuracy in specific domains.
Patent Information
- Application Number
- CN202411369012.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2024-12-13
AI Technical Summary
Existing cross-domain named entity recognition (NER) models face challenges in effectively utilizing limited labeled data, especially in specific domains, due to sparse annotations and differing label distributions, leading to reduced performance and recognition accuracy.
A cross-domain NER method using a cross-self-attention encoder to exchange query matrices, calculate maximum mean difference with label constraints, and apply L2 regularization to constrain relative entropy, facilitating knowledge sharing between source and target domains to enhance feature representation and label prediction.
Improves model performance by enabling effective knowledge transfer and accurate label prediction in target domains with sparse annotations, enhancing recognition accuracy in specific fields.
Smart Images
Figure CN119129599A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of information extraction, and in particular relates to a cross-domain named entity recognition method based on knowledge sharing. Background Art
[0002] Named Entity Recognition (NER) tasks are divided into general domain named NER and specific domain NER. Specific domains include finance, justice, biology, medicine, military, etc. Although more and more high-quality datasets are used for model training and evaluation of Chinese NER, the fields involved in most datasets are general domains, and the entity category names are relatively fixed. There are few public datasets in specific fields, and most of them are built by individuals during research. Data privacy issues also mean that the built datasets cannot be shared, and the cost of manual labeling is very high. These limitations have led to sparse annotations in datasets in specific fields.
[0003] At present, NER models based on deep learning often require a large amount of labeled data to better train the model. The small scale of text makes it impossible for deep learning models to fully learn the hidden features of the data. Therefore, the problem of limited text tags reduces the performance and recognition effect of deep learning models. On the other hand, some studies define the source domain and the target domain. Among them, the source domain uses a large-scale general domain dataset, and the target domain uses a specific dataset with limited resources. The parameter sharing model is trained by jointly training the source domain data and the target domain data to make the parameters of the source domain and the target domain as close as possible to achieve the purpose of improving the recognition effect of the target domain. However, simple parameter sharing cannot enable the target domain to absorb the characteristics of the source domain text corpus. Moreover, when the label distribution of the two domain data is different, simply sharing the CRF layer may lead to different label results for the same type of labels in the source and target domains. Summary of the invention
[0004] Purpose of the invention: It can be seen from the analysis of the above prior art that in the cross-domain named entity recognition task, the model needs to be able to exchange source domain and target domain features and share source domain knowledge. In order to achieve effective recognition of named entities in a specific domain (target domain) with sparse annotations, the present invention provides a cross-domain named entity recognition method based on knowledge sharing.
[0005] Technical solution: In view of the problems and shortcomings in the prior art, the present invention provides a cross-domain named entity recognition method based on knowledge sharing, which uses a cross-self-attention encoder to exchange the query matrices of the source domain and the target domain, converts the embedding vector into an attention vector, and increases the shared information from the source domain task to improve the model performance; uses the maximum mean difference with label constraints to measure the feature representation difference between the source domain and the target domain, calculates the overall maximum mean difference between the two sample vector representations with the same label, realizes the feature exchange between the source domain and the target domain, and shares the source domain knowledge; calculates the inverse relative entropy of the label probabilities of the two domains in the conditional random fields of the source domain and the target domain, which is used to measure the distance between the label probability distributions of the two domains, and uses L2 regularization to constrain the relative entropy of the label probabilities in the two domains to make the distributions between the same labels approximate and obtain the same label output results.
[0006] The specific steps include:
[0007] Step 1: For the input text sequence, use the Cross-Transformer to exchange the query matrices of the source domain and the target domain, convert the embedding vector into an attention vector, increase the shared information from the source domain task, and enable the target domain to absorb the knowledge of the source domain;
[0008] Step 2: In the knowledge sharing stage, the maximum mean difference (MMD) with label constraints is used to measure the difference in feature representation between the source domain and the target domain. The overall maximum mean difference between the vector representations of two samples with the same label is calculated to achieve feature exchange between the source domain and the target domain and share source domain knowledge.
[0009] Step 3: Calculate the inverse relative entropy of the label probabilities of the two domains in the conditional random fields (CRFs) of the source domain and the target domain to measure the distance between the label probability distributions of the two domains. Use L2 regularization to constrain the relative entropy of the label probabilities in the two domains so that the distributions between the same labels are approximated and the same label output results are obtained.
[0010] In step 1, the cross self-attention encoder is used to exchange the query matrix of the source domain and the target domain. The specific steps to convert the embedding vector into the attention vector are as follows:
[0011] The query matrix Q in the source domain S , key matrix K S Sum value matrix V S Input cross self-attention encoder, the matrix Q corresponding to the target domain T ,K T ,V T Obtained by embedding linear transformation:
[0012]
[0013] Among them, E S With E T are the embeddings of the source domain and the target domain respectively, I is the unit matrix, W S(T),Q With W S(T),V is a trainable parameter.
[0014] In the source domain and the target domain, for any two Chinese characters x in the text i With x j , i and j represent the positions of the two Chinese characters in the text sentence, head[i] represents the head, and tail[i] represents the tail. The calculation formulas for the four relative distances of the corresponding words are:
[0015]
[0016] in, Represents x i Head and x j The distance between the heads, Represents x i Head and x j The distance between the tails of Represents x i tail and x j The distance between the heads, Represents x i tail and x j The relative position encoding is a simple nonlinear transformation of the four distances:
[0017]
[0018]
[0019] Among them, W r and W R is a trainable parameter, Represents the concatenation operator.
[0020] Based on the relative position encoding of the self-attention encoder Cross-Transformer calculated by the above formula, the attention vectors of the source domain and the target domain are calculated:
[0021]
[0022] in, is a trainable parameter.
[0023] The attention scores of the source domain and the target domain after the exchange query matrix are calculated:
[0024] Att S(A T ,V S )=Softmax(A T )V S
[0025] Att T (A S ,V T )=Softmax(A S )V T
[0026] Among them, A S With A T are the attention vectors of the source domain and the target domain respectively, V S With V T are the value matrices of the source domain and the target domain respectively.
[0027] Finally, randomly initialize the parameter matrix B max_len×max_len , adding random attention sampling, not limited to local information, gives the final score:
[0028]
[0029] Among them, V S With V T are the value matrices of the source domain and the target domain respectively.
[0030] In step 2, the MMD with label constraints is used to measure the difference in feature representation between the source domain and the target domain. The specific steps for calculating the overall maximum mean difference between two sample vector representations with the same label are as follows:
[0031] Given a source domain N s Sample training set: Target area N t Sample training set: And N t <<N s The context information of the source domain and the target domain is not entirely the same, resulting in differences in feature representation between the two domains. knowledge and improve target areas In order to reduce the difference between the source domain and the target domain, MMD with label constraints is used to keep the source domain and target domain vector states in a similar distribution.
[0032] Defining Tag Classes is the set of matching labels between the source domain and the target domain. The overall maximum mean difference (MMD) between two sample vector representations with the same label y is:
[0033]
[0034] in, is a hidden representation v s A collection of is a hidden representation v t A collection of represents the number of samples in the source domain, Represents the number of samples in the target domain.
[0035] Finally, the loss function of MMD is defined as:
[0036]
[0037] Among them, μ y is the corresponding coefficient, V * is the value matrix output from step 1.4.
[0038] In step 3, the inverse relative entropy of the label probabilities of the two domains is calculated to measure the distance between the label probability distributions of the two domains. The specific steps of using L2 regularization to constrain the relative entropy of the label probabilities in the two domains are as follows:
[0039] Calculate the reverse relative entropy of the source domain and target domain label probabilities to further determine the law between the label distributions of the two domains. The calculation formula for relative entropy is:
[0040]
[0041] Among them, y is the label class set, p s (y|V * ) is the label probability of the source domain, p t (y|V * ) is the label probability of the target domain. The relative entropy can be obtained by logarithmic transformation:
[0042]
[0043] Where H(·) represents the distribution entropy.
[0044] By reducing p t (y|V * ) to p s (y|V * ) reduces the distance between the two by increasing the relative entropy upper limit and adding the L2 constraint to the parameters corresponding to the same labels in the source and target domains:
[0045]
[0046] Where c is a constant, W s With W tare the parameter vectors of the source domain and the target domain respectively, A s With A t They are the attention vectors of the source domain and the target domain output in step 2 respectively.
[0047] The parameter regularization loss function in CRF is defined as:
[0048]
[0049] The negative log-likelihood loss function of the source domain and target domain training samples is:
[0050]
[0051] Where ε is the balance coefficient.
[0052] The loss function is defined as:
[0053]
[0054] In the formula is the MMD loss obtained in step 2.3, and α, β, and γ are the hyperparameters of the balanced loss term.
[0055] Finally, CRF predicts the label sequence with the highest probability, which is used to classify and annotate the text.
[0056] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the cross-domain named entity recognition method based on knowledge sharing as described above is implemented.
[0057] A computer-readable storage medium stores a computer program for executing the cross-domain named entity recognition method based on knowledge sharing as described above.
[0058] Beneficial effects: Compared with the prior art, the cross-domain named entity recognition method based on knowledge sharing provided by the present invention has the following advantages:
[0059] (1) Use the cross-attention encoder to exchange the query matrices of the source domain and the target domain, so that the target domain can absorb the knowledge of the source domain, realize the knowledge transfer between the two domains, and increase the shared information from the source domain tasks to improve the model performance.
[0060] (2) The maximum mean difference with label constraints is used to measure the difference in feature representation between the source domain and the target domain. The overall maximum mean difference between the vector representations of two samples with the same label is calculated to achieve feature exchange between the source domain and the target domain, share the source domain knowledge, and improve the label prediction accuracy of the target domain.
[0061] (3) The present invention solves the problem of reduced performance and recognition effect of deep learning models caused by limited text tags, can effectively solve the entity recognition problem in specific fields with sparse annotations, and has high recognition accuracy and good recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 is a flow chart of a method according to an embodiment of the present invention;
[0063] Figure 2 It is a schematic diagram of MMD with label constraints in an example implementation of the present invention;
[0064] Figure 3 This is a CRF label parameter constraint diagram in an example implementation of the present invention. DETAILED DESCRIPTION
[0065] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.
[0066] The MSRA dataset released by Microsoft Research Asia is used as the source domain dataset, and the crime fact entity recognition dataset LAIC2022 released by the 5th "China Law Research Cup" Judicial Artificial Intelligence Challenge is used as the target domain dataset. Among them, the MSRA dataset is large in scale and has rich text annotations, while the number of sentences and characters in LAIC2022 is small, and the annotations are sparse.
[0067] Figure 1 The overall process of the cross-domain named entity recognition method based on knowledge sharing is given, including the following steps:
[0068] Step 1: Use the cross self-attention encoder to exchange the query matrix of the source domain and the target domain to convert the embedding vector into an attention vector. The encoder uses a layer of Transformer with 8 heads, where the FFN size is 480. The total number of model training iterations is up to 100 times, and the batch size is set to 16.
[0069] Input Transformer source domain Q S ,K S ,V S , the matrix Q corresponding to the target domain T ,K T ,V T Obtained by embedding linear transformation:
[0070]
[0071] Among them, ES With E T are the embeddings of the source domain and the target domain respectively, I is the unit matrix, W S(T),Q With W S(T),V is a trainable parameter.
[0072] In the source domain and the target domain, for any two Chinese characters x in the text i With x j , i and j represent the positions of the two Chinese characters in the text sentence, head[i] represents the head, and tail[i] represents the tail. The calculation formulas for the four relative distances of the corresponding words are:
[0073]
[0074] in, Represents x i Head and x j The distance between the heads, Represents x i Head and x j The distance between the tails of Represents x i tail and x j The distance between the heads, Represents x i tail and x j The relative position encoding is a simple nonlinear transformation of the four distances:
[0075]
[0076] Among them, W r ,W R is a trainable parameter, Represents the concatenation operator.
[0077] Based on the relative position encoding of the self-attention encoder Cross-Transformer calculated by the above formula, the attention vectors of the source domain and the target domain are calculated:
[0078]
[0079] in, is a trainable parameter.
[0080] The attention scores of the source domain and the target domain after the exchange query matrix are calculated:
[0081] Att S (A T ,V S )=Softmax(A T )V S
[0082] Att T (A S ,V T )=Softmax(A S )V T
[0083] Among them, A S With A T are the attention vectors of the source domain and the target domain respectively, V S With V T are the value matrices of the source domain and the target domain respectively.
[0084] Finally, randomly initialize the parameter matrix B max_len×max_len , adding random attention sampling, not limited to local information, gives the final score:
[0085]
[0086] Among them, V S With V T are the value matrices of the source domain and the target domain respectively.
[0087] Step 2: Use MMD with label constraints to measure the difference in feature representation between the source domain and the target domain, and calculate the overall maximum mean difference between the vector representations of two samples with the same label.
[0088] Given a source domain N s Sample training set: Target area N t Sample training set: And N t <<N s In order to reduce the difference between the source domain and the target domain, MMD with label constraints is used as Figure 2 As shown, the source domain and target domain vector states are kept in similar distribution.
[0089] Defining Tag Classes is the set of matching labels between the source domain and the target domain. The overall maximum mean difference (MMD) between two sample vector representations with the same label y is:
[0090]
[0091] in, is a hidden representation v s A collection of is a hidden representation v t A collection of represents the number of samples in the source domain, Represents the number of samples in the target domain.
[0092] Finally, the loss function of MMD is defined as:
[0093]
[0094] Among them, μ y is the corresponding coefficient, V * is the value matrix output from step 1.4.
[0095] Step 3: Calculate the reverse relative entropy of the label probabilities of the two domains in the CRF of the MSRA dataset and the LAIC2022 dataset to measure the distance between the label probability distributions of the two domains. Use L2 regularization to constrain the relative entropy of the label probabilities in the two domains. The CRF label parameter constraints are as follows: Figure 3 .
[0096] Calculate the reverse relative entropy of the source domain and target domain label probabilities to further determine the law between the label distributions of the two domains. The calculation formula for relative entropy is:
[0097]
[0098] Among them, y is the label class set, p s (y|V * ) is the label probability of the source domain, p t (y|V * ) is the label probability of the target domain. The relative entropy can be obtained by logarithmic transformation:
[0099]
[0100] Where H(·) represents the distribution entropy.
[0101] By reducing p t (yV * ) to p s (y|V * ) reduces the distance between the two by increasing the relative entropy upper limit and adding the L2 constraint to the parameters corresponding to the same labels in the source and target domains:
[0102]
[0103] Where c is a constant, W s With W t are the parameter vectors of the source domain and the target domain respectively, A s With A t They are the attention vectors of the source domain and the target domain output in step 2 respectively.
[0104] The parameter regularization loss function in CRF is defined as:
[0105]
[0106] The negative log-likelihood loss function of the source domain and target domain training samples is:
[0107]
[0108] Where ε is the balance coefficient.
[0109] The final loss function is defined as:
[0110]
[0111] In the formula, is the MMD loss obtained in step 2.3, α, β, and γ are the hyperparameters of the balanced loss term, where α = 0.025, β = 0.075, and γ = 0.25.
[0112] Finally, CRF predicts the label sequence with the highest probability, which is used to classify and annotate the text.
[0113] Obviously, those skilled in the art should understand that the various steps of the cross-domain named entity recognition method based on knowledge sharing of the above-mentioned embodiment of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.
Claims
1. A cross-domain named entity recognition method based on knowledge sharing, characterized in that: The steps include: Step 1: For the input text sequence, use the Cross-Transformer to exchange the query matrices of the source domain and the target domain, convert the embedding vector into an attention vector, increase the shared information from the source domain task, and enable the target domain to absorb the knowledge of the source domain; Step 2: In the knowledge sharing stage, the maximum mean difference (MMD) with label constraints is used to measure the difference in feature representation between the source domain and the target domain. The overall maximum mean difference between the vector representations of two samples with the same label is calculated to achieve feature exchange between the source domain and the target domain and share the source domain knowledge. Step 3: Calculate the inverse relative entropy of the label probabilities of the two domains in the conditional random field (CRF) of the source domain and the target domain to measure the distance between the label probability distributions of the two domains. Use L2 regularization to constrain the relative entropy of the label probabilities in the two domains so that the distributions between the same labels are approximate and the same label output results are obtained.
2. According to the cross-domain named entity recognition method based on knowledge sharing according to claim 1, it is characterized in that: In step 1, a cross-attention encoder Cross-Transformer is used to exchange the query matrices of the source domain and the target domain to convert the embedding vector into an attention vector. The specific steps are as follows: The query matrix Q in the source domain S , key matrix K S Sum value matrix V S Input Cross-Transformer, the matrix Q corresponding to the target domain T ,K T ,V T Obtained by embedding linear transformation: Among them, E S With E T are the embeddings of the source domain and the target domain respectively, I is the unit matrix, W S(T),Q With W S(T),V is a trainable parameter. In the source domain and the target domain, for any two Chinese characters x in the text i With x j , i and j represent the positions of the two Chinese characters in the text sentence, head[i] represents the head, and tail[i] represents the tail. The calculation formulas for the four relative distances of the corresponding words are: in, Represents x i Head and x j The distance between the heads, Represents x i Head and x j The distance between the tails of Represents x i tail and x j The distance between the heads, Represents x i tail and x j The relative position encoding is a simple nonlinear transformation of the four distances: Among them, W r ,W R is a trainable parameter and ⊕ represents a concatenation operator. Based on the relative position encoding of the self-attention encoder Cross-Transformer calculated by the above formula, the attention vectors of the source domain and the target domain are calculated: in, is a trainable parameter. The attention scores of the source domain and the target domain after the exchange query matrix are calculated: To S (A T ,V S )=Softmax(A T )V S To T (A S ,V T )=Softmax(A S )V T Among them, A S With A T are the attention vectors of the source domain and the target domain respectively, V S With V T are the value matrices of the source domain and the target domain respectively. Finally, randomly initialize the parameter matrix B max _len×max_ len , adding random attention sampling, not limited to local information, gives the final score: Among them, V S With V T are the value matrices of the source domain and the target domain respectively.
3. According to the cross-domain named entity recognition method based on knowledge sharing according to claim 1, it is characterized in that: In step 2, the MMD with label constraints is used to measure the difference in feature representation between the source domain and the target domain, and the overall maximum mean difference between the two sample vector representations with the same label is calculated. The specific steps are as follows: Given a source domain N s Sample training set: Target area N t Sample training set: And N t <<N s The context information of the source domain and the target domain is not entirely the same, resulting in differences in feature representation between the two domains. knowledge and improve target areas In order to reduce the difference between the source domain and the target domain, MMD with label constraints is used to keep the source domain and target domain vector states in a similar distribution. Defining Tag Classes is the set of matching labels between the source domain and the target domain. The overall maximum mean difference (MMD) between two sample vector representations with the same label y is: in, is a hidden representation v s A collection of is a hidden representation v t A collection of represents the number of samples in the source domain, Represents the number of samples in the target domain. Finally, the loss function of MMD is defined as: Among them, μ y is the corresponding coefficient, V * is the value matrix output from step 1.
4.
4. The cross-domain named entity recognition method based on knowledge sharing according to claim 1 is characterized in that: In step 3, the reverse relative entropy of the label probabilities of the two domains is calculated in the conditional random field CRF of the source domain and the target domain to measure the distance between the label probability distributions of the two domains. The relative entropy of the label probabilities in the two domains is constrained by L2 regularization. The specific steps are as follows: Calculate the reverse relative entropy of the source domain and target domain label probabilities to further determine the law between the label distributions of the two domains. The calculation formula for relative entropy is: Among them, y is the label class set, p s (y|V * ) is the label probability of the source domain, p t (y|V * ) is the label probability of the target domain. The relative entropy can be obtained by logarithmic transformation: Where H(·) represents the distribution entropy. By reducing p t (y|V * ) to p s (y|V * ) reduces the distance between the two by increasing the relative entropy upper limit and adding the L2 constraint to the parameters corresponding to the same labels in the source and target domains: Where c is a constant, W s With W t are the parameter vectors of the source domain and the target domain respectively, A s With A t They are the attention vectors of the source domain and the target domain output in step 2 respectively. The parameter regularization loss function in CRF is defined as: The negative log-likelihood loss function of the source domain and target domain training samples is: Where ε is the balance coefficient. The final loss function is defined as: In the formula, is the MMD loss obtained in step 2.3, and α, β, and γ are the hyperparameters of the balanced loss term. Finally, CRF predicts the label sequence with the highest probability, which is used to classify and annotate the text.
5. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the cross-domain named entity recognition method based on knowledge sharing as described in any one of claims 1 to 4 is implemented.
6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the cross-domain named entity recognition method based on knowledge sharing as described in any one of claims 1 to 4.