An adversarial reading comprehension nested named entity recognition method
Through the adversarial reading comprehension method, combined with BERT and Star-Transformer, the problems of lack of professional knowledge, noise impact and insufficient information mining in nested named entity recognition in complex electromechanical equipment are solved, achieving higher recognition accuracy and stability.
Patent Information
- Application Number
- CN202410606985.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-16
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-05-16
AI Technical Summary
In the field of complex electromechanical equipment, the existing machine reading comprehension model lacks professional knowledge in nested named entity recognition, influence of noise data, insufficient learning of label annotation information and insufficient mining of internal text information, resulting in insufficient recognition accuracy and Semantic comprehension error.
Adversarial reading comprehension nested named entity recognition method, using BERT model and Star-Transformer fine-tuned by the complex electromechanical equipment field dictionary, combining attention mechanism, average pooling and cosine similarity for label annotation information, and through the FGM algorithm Compute noise interference, generate text feature representations of slight perturbations, calculate the probability of the entity's start and end positions, and finally output the predicted entity range and its categories.
Effectively learn knowledge in complex electromechanical fields, improve the stability and generalization of the model in noise data, comprehensively mine the semantic information of label annotation, capture the long-distance dependencies within the sentence, and improve the accuracy of nested named entity recognition.
Smart Images

Figure BDA0004842322880000033 
Figure BDA0004842322880000034 
Figure BDA0004842322880000035
Abstract
Description
Technical Field
[0001] The present invention belongs to entity recognition technology, and in particular to an adversarial reading comprehension nested named entity recognition method. Background Art
[0002] Complex electromechanical equipment is usually composed of a series of hierarchically organized components and systems, forming a highly structured framework. In this hierarchical structure, the entire equipment can be divided into multiple subsystems, which are further composed of smaller components, which may also contain more sophisticated components. This hierarchical structure not only clarifies the subordinate relationship between components and parts, but also reflects the mutual nesting between them. For example, "the temperature of the left wheelset axle box rises" is a specific fault phenomenon entity. In this entity, "temperature rise" is a fault mode entity, and "wheelset axle box" is a fault structure entity with a hierarchical structure. In the complex electromechanical equipment system, "wheelset axle box" includes two key structures: "wheelset" and "axle end", and the next level of "axle end" includes the "axle box" structure.
[0003] This situation of mutual nesting between entities undoubtedly brings challenges to the extraction of entities in text data in the field of complex electromechanical equipment. Since traditional sequence annotation methods cannot solve the problem of identifying nested named entities, the machine reading comprehension method represented by the LEAR model in the general field has become an effective means for nested named entity recognition. However, due to the professional nature of data in the field of complex electromechanical equipment, the current model that uses machine reading comprehension to realize nested entity recognition has the following problems:
[0004] 1. Lack of pre-training of professional domain knowledge. The pre-training models used by general domain models usually do not involve professional terms in the field of complex electromechanical equipment, resulting in insufficient accuracy of the models in specific recognition tasks.
[0005] 2. Impact of noisy data. When processing data in the field of complex electromechanical equipment, existing models often lead to incorrect information extraction due to the loss of characters. For example: "Vehicle 8 reported 68C4 (fault code), at least one wheelset bearing measured temperature failure on May 9, 2011, and the axle temperature sensor was replaced on May 11, 2011." In this sentence, "temperature failure" and "May 11, 2011" will cause semantic misunderstanding due to the missing comma, so that the model may mistakenly identify "May 11, 2011" as the fault time rather than the maintenance time.
[0006] 3. Insufficient learning of label annotation information. The current machine reading comprehension method is to connect the question and text information as input (the format is: [CLS] question [SEP] text [SEP]). Since the label annotation information (i.e., the category description information) is integrated into the text representation based on the self-attention mechanism, and the "attention" of the self-attention mechanism will be distracted by the text and cannot be fully focused on the question part, the model cannot effectively distinguish the importance of each word in the text to different question labels.
[0007] 4. Insufficient mining of text internal information. The existing machine reading comprehension method uses a pre-trained model to pre-embed text information and questions, but does not deeply mine the rich contextual information in the text information. Summary of the invention
[0008] In view of the above problems, the present invention provides an adversarial reading comprehension nested named entity recognition method.
[0009] The present invention provides an adversarial reading comprehension nested named entity recognition method, comprising the following steps:
[0010] Step 1: Use the BERT model (Bidirectional Encoder Representations from Transformers, BERT) that has been fine-tuned on the dictionary of complex electromechanical equipment to initially embed the input text information and label annotations, and then perform further encoding operations on the text information through Star-Transformer to capture deeper semantic features.
[0011] Step 2: The label annotation information is comprehensively analyzed using the attention mechanism, average pooling and cosine similarity to fully explore the semantic information in the label annotation; after the text information and the label annotation information are fused, the model uses the FGM algorithm to calculate the noise interference and add it to obtain the text feature representation containing small disturbances.
[0012] Step 3: Calculate the probability of each token in the text being the starting or ending position of each entity, and finally output the predicted entity range and its corresponding category.
[0013] Step 4: Use the loss function to measure the difference between the model output and the true label, and minimize this difference to train the model.
[0014] Furthermore, step 1 is specifically as follows:
[0015] Step 1.1: Implement the initial embedding of text information and label annotation information.
[0016] Given the input text information X and the label annotation Y, the initial embedding is performed using BERT fine-tuned with a dictionary in the field of complex electromechanical equipment, and the following representation results are obtained:
[0017] E X =bert(X) (1)
[0018] E Y =bert(Y) (2)
[0019] Among them, E X ∈R n×d and E Y ∈R |C|×m×d , n represents the length of the text information X, m represents the length of the label annotation Y, |C| represents the size of the category set C, and d represents the vector dimension of the BERT encoder.
[0020] Step 1.2: Deepen the internal information of the text to obtain richer text information.
[0021] After the initial embedding of the text information, the Star-Transformer encoder is introduced to encode the text information of length n; let E X =[e1,e2,...,e n ], where e i ∈R 1×d represents the embedding representation of the i-th token, and at the same time, s t ∈R 1×d represents the state of the relay node at step t, H t ∈R n×d Denote the states of all n satellite nodes at step t, and use H 0 =E X and 0 =average(E X ) to initialize the state, and average represents the averaging operation.
[0022] Furthermore, step 2 is specifically as follows:
[0023] Step 2.1: Map the text information embedding representation and the label annotation embedding representation to the same feature space.
[0024] First, H X and E Y They are input into the fully connected layer respectively, the purpose of which is to map the text information embedding representation and the label annotation embedding representation to the same feature space, calculated by the following formula:
[0025] h X_fc =W X ·H X (3)
[0026] h Y_fc =W Y ·E Y (4)
[0027] Among them, W X ∈R d×d and W Y ∈R d×d are learnable parameters of the fully connected layer.
[0028] Step 2.2: Use the attention mechanism to aggregate the label annotation information into the text information.
[0029] For any 1≤i≤n, let x i is the i-th token of the text information X; h X_fc The i-th row of ; for any 1≤j≤m and category c∈C, let is the jth token of the label annotation Y of category c, so the attention mechanism is used to calculate the embedding representation h of the label annotation Y_att , and its calculation formula is as follows:
[0030]
[0031]
[0032] The calculation process of formulas (5) and (6) is repeated for all categories to obtain the embedding representation h of the label annotation. Y_att .
[0033] Step 2.3: Aggregate the label annotation information into text information using average pooling.
[0034] The embedding representation h of the label annotation is calculated using the average pooling method Y_ave , and its calculation formula is as follows:
[0035]
[0036] The calculation process of formula (7) is repeated for all categories to obtain the embedding representation h of the label annotation. Y_ave ;
[0037] Step 2.4: Aggregate the label annotation information to the text information using cosine similarity calculation.
[0038] By calculating the text feature h X_fc Annotate with label h Y_fc The cosine similarity between them is calculated and the calculated cosine similarity score is multiplied by the label annotation vector to obtain the weighted label feature representation hY_cos , and its calculation formula is as follows:
[0039]
[0040]
[0041] in, Yes Y_fc The corresponding embedding representation of category c is obtained by repeating the calculation process of formulas (8) and (9) for all categories to obtain the embedding representation h of the label annotation. Y_cos .
[0042] Step 2.5: The information learned in steps 2.2 to 2.4 is integrated into the text features through learnable weight parameters.
[0043] Through the attention mechanism, average pooling and cosine similarity, different label annotation information is obtained respectively. The following calculation formula is used to integrate the label annotation information into the text information:
[0044] h fused =h X_fc +α·h Y_att +β·h Y_ave +λ·h Y_cos (10)
[0045] Among them, α, β, and λ are all learnable parameters.
[0046] Step 2.6: Add noise interference.
[0047] After the text information is fused with the label annotation information, the FGM algorithm is used to generate adversarial perturbations and add the perturbations to the fused text information h fused The calculation formula is as follows:
[0048]
[0049] h adv =r adv +h fused (12)
[0050] Among them, r adv is the current input perturbation; ε is a configurable hyperparameter that determines the magnitude of the adversarial perturbation; To obtain the gradient of x, x represents the model input; L(θ,x,y) is the loss corresponding to the sample, y is the corresponding label, and θ is all the parameters of the current model.
[0051] Furthermore, step 3 is specifically as follows:
[0052] According to the label sequence output by the model, the starting and ending positions of each entity are determined to obtain the span information of the entity; the start / end tag mode is used to annotate the target range to be extracted; specifically, for each token x i The following calculation formula is used to calculate the probability of the starting position:
[0053]
[0054] in, Indicates x i is the probability of the span of category c at the start position; M s ∈R |C|×d and b s ∈R d All are learnable parameters; ° represents primitive multiplication; f o (·) is a function that sums the rows of the input means; similarly, use Indicates x i is the probability of the ending position, and the calculation process is similar.
[0055] In nested named entity recognition, spans in the same category can be nested or overlapped, which makes the heuristic matching principle no longer work. Therefore, the solution in the BERT-MRC model is adopted; a binary classifier is used to predict a pair of candidate start / end positions as the span; specifically, for any category c, the following binary classifier is defined:
[0056]
[0057] Among them, 1≤i,j≤n; M∈R 1×2d is a learnable parameter; when Coming soon i , x j The range prediction is class c.
[0058] Furthermore, step 4 is specifically as follows:
[0059] In order to measure the difference between the model output and the true label, and to minimize this difference to train the model; therefore, refer to the definition of the relevant loss function in the LEAR model, and its calculation formula is as follows:
[0060]
[0061]
[0062]
[0063] L total =μ(L s +L e )+vLmatch (18)
[0064] Among them, L s represents the initial loss, L e Indicates the end of loss, L match represents the matching loss, L total represents the final total loss; CE(x,y) represents the cross entropy; S c ∈{0,1} n , if and only if x i is the actual starting position of category c. E c ∈{0,1} n , if and only if x i Is the actual end position of category c? M c ∈R n×n is a binary matrix if and only if token x i and x j When all are classified as the true range of category c, W c ∈R n×n is also a binary matrix if and only if or When there is μ,v are learnable hyperparameters in the final loss function.
[0065] The beneficial technical effects of the present invention are:
[0066] 1. In the field of complex electromechanical equipment, the structural hierarchy for the same product family is generally fixed. Therefore, the corresponding product family metastructure tree is used to construct a domain dictionary, and the BERT model is used for fine-tuning, so that relevant knowledge in the field can be effectively learned.
[0067] 2. Introduce the adversarial training FGM algorithm to improve the stability and generalization of the model when facing noisy data.
[0068] 3. The three strategies (attention mechanism, cosine similarity and average pooling) are jointly used to integrate the label annotation information into the text feature representation, so that the semantic information in the label annotation can be more comprehensively mined and utilized.
[0069] 4. Introduce Star-Transformer to effectively capture the long-distance dependencies within sentences, thereby obtaining a richer representation of text information. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1 A flowchart for the overall structure of the model network. DETAILED DESCRIPTION
[0071] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0072] The overall structure of the model network of the adversarial reading comprehension nested named entity recognition method of the present invention is as follows: Figure 1 As shown. The following steps are included:
[0073] Step 1: Use the BERT model (Bidirectional Encoder Representations from Transformers, BERT) that has been fine-tuned on the dictionary of complex electromechanical equipment to initially embed the input text information and label annotations, and then perform further encoding operations on the text information through Star-Transformer to capture deeper semantic features.
[0074] Step 1.1: Implement the initial embedding of text information and label annotation information.
[0075] Given the input text information X and the label annotation Y, the initial embedding is performed using BERT fine-tuned with a dictionary in the field of complex electromechanical equipment, and the following representation results are obtained:
[0076] E X =bert(X) (1)
[0077] E Y =bert(Y) (2)
[0078] Among them, E X ∈R n×d and E Y ∈R |C|×m×d , n represents the length of the text information X, m represents the length of the label annotation Y, |C| represents the size of the category set C, and d represents the vector dimension of the BERT encoder.
[0079] Step 1.2: Deepen the internal information of the text to obtain richer text information.
[0080] In order to achieve in-depth mining of the internal information of the text and obtain richer text information, after the initial embedding of the text information, the Star-Transformer encoder is introduced to encode the text information of length n; let E X =[e1,e2,...,e n ], where e i ∈R 1×d represents the embedding representation of the i-th token, and at the same time, let s t ∈R 1×d represents the state of the relay node at step t, H t∈R n×d Denote the states of all n satellite nodes at step t, and use H 0 =E X and 0 =average(E X ) to initialize the state, and average represents the average operation. The overall network update of the introduced Star-Transformer is shown in Algorithm 1:
[0081]
[0082] Note:
[0083] 1) MultiAtt(a,b) represents the multi-head attention mechanism;
[0084] 2)LayerNorm(x) represents the layer normalization operation.
[0085] Step 2: The label annotation information is comprehensively analyzed using the attention mechanism, average pooling and cosine similarity to fully explore the semantic information in the label annotation; after the text information and the label annotation information are fused, the model uses the FGM algorithm to calculate the noise interference and add it to obtain the text feature representation containing small disturbances.
[0086] Step 2.1: Map the text information embedding representation and the label annotation embedding representation to the same feature space.
[0087] First, H X and E Y They are input into the fully connected layer respectively, the purpose of which is to map the text information embedding representation and the label annotation embedding representation to the same feature space, calculated by the following formula:
[0088] h X_fc =W X ·H X (3)
[0089] h Y_fc =W Y ·E Y (4)
[0090] Among them, W X ∈R d×d and W Y ∈R d×d are learnable parameters of the fully connected layer.
[0091] Step 2.2: Use the attention mechanism to aggregate the label annotation information into the text information.
[0092] For any 1≤i≤n, let x i is the i-th token of the text information X; h X_fc The i-th row of ; for any 1≤j≤m and category c∈C, let is the jth token of the label annotation Y of category c, so the attention mechanism is used to calculate the embedding representation h of the label annotation Y_att , and its calculation formula is as follows:
[0093]
[0094]
[0095] The calculation process of formulas (5) and (6) is repeated for all categories to obtain the embedding representation h of the label annotation. Y_att .
[0096] Step 2.3: Aggregate the label annotation information into text information using average pooling.
[0097] The embedding representation h of the label annotation is calculated using the average pooling method Y_ave , and its calculation formula is as follows:
[0098]
[0099] The calculation process of formula (7) is repeated for all categories to obtain the embedding representation h of the label annotation. Y_ave ;
[0100] Step 2.4: Aggregate the label annotation information to the text information using cosine similarity calculation.
[0101] By calculating the text feature h X_fc Annotate with label h Y_fc The cosine similarity between them is calculated and the calculated cosine similarity score is multiplied by the label annotation vector to obtain the weighted label feature representation h Y_cos , and its calculation formula is as follows:
[0102]
[0103]
[0104] in, Yes Y_fc The corresponding embedding representation of category c is obtained by repeating the calculation process of formulas (8) and (9) for all categories to obtain the embedding representation h of the label annotation. Y_cos .
[0105] Step 2.5: The information learned in steps 2.2 to 2.4 is integrated into the text features through learnable weight parameters.
[0106] Through the attention mechanism, average pooling and cosine similarity, different label annotation information is obtained respectively. The following calculation formula is used to integrate the label annotation information into the text information:
[0107] h fused =h X_fc +α·h Y_att +β·h Y_ave +λ·h Y_cos (10)
[0108] Among them, α, β, and λ are all learnable parameters.
[0109] Step 2.6: Add noise interference.
[0110] After the text information is fused with the label annotation information, adversarial training is introduced to improve the model's ability to cope with noisy data and improve the model's generalization ability. The essence of adversarial training is to select a perturbation that can cause the model to incur greater losses as an attack, then add the maximum perturbation to the input sample, and update the parameters in the direction of minimizing the loss containing the perturbation. The FGM algorithm is used to generate adversarial perturbations and add the perturbations to the fused text information h. fused The calculation formula is as follows:
[0111]
[0112] h adv =r adv +h fused (12)
[0113] Among them, r adv is the current input perturbation; ε is a configurable hyperparameter that determines the magnitude of the adversarial perturbation; To obtain the gradient of x, x represents the model input; L(θ,x,y) is the loss corresponding to the sample, y is the corresponding label, and θ is all the parameters of the current model.
[0114] Step 3: Calculate the probability of each token in the text being the starting or ending position of each entity, and finally output the predicted entity range and its corresponding category.
[0115] According to the label sequence output by the model, the starting and ending positions of each entity are determined to obtain the span information of the entity; the start / end tag mode is used to annotate the target range to be extracted; specifically, for each token x i The following calculation formula is used to calculate the probability of the starting position:
[0116]
[0117] in, Indicates x i is the probability of the span of category c at the start position; M s ∈R |C|×d and b s ∈R d All are learnable parameters; represents primitive multiplication; f o (·) is a function that sums the rows of the input means; similarly, use Indicates x i is the probability of the ending position, and the calculation process is similar.
[0118] In nested named entity recognition, spans in the same category can be nested or overlapped, which makes the heuristic matching principle no longer work. Therefore, the solution in the BERT-MRC model is adopted; a binary classifier is used to predict a pair of candidate start / end positions as the span; specifically, for any category c, the following binary classifier is defined:
[0119]
[0120] Among them, 1≤i,j≤n; M∈R 1×2d is a learnable parameter; when Coming soon i , x j The range prediction is class c.
[0121] Step 4: Use the loss function to measure the difference between the model output and the true label, and minimize this difference to train the model.
[0122] In order to measure the difference between the model output and the true label, and to minimize this difference to train the model; therefore, refer to the definition of the relevant loss function in the LEAR model, and its calculation formula is as follows:
[0123]
[0124]
[0125]
[0126] L total =μ(L s +L e )+νL match (18)
[0127] Among them, L s represents the initial loss, L e Indicates the end of loss, L match represents the matching loss, L totalrepresents the final total loss; CE(x,y) represents the cross entropy; S c ∈{0,1} n , if and only if x i is the actual starting position of category c. E c ∈{0,1} n , if and only if x i Is the actual end position of category c? M c ∈R n×n is a binary matrix if and only if token x i and x j When all are classified as the true range of category c, W c ∈R n×n is also a binary matrix if and only if or When there is μ, ν are learnable hyperparameters in the final loss function.
[0128] In the process of nested named entity recognition, the main types of errors include correct recognition of the named entity boundary but incorrect labeling of the named entity type, or incorrect recognition of the named entity boundary but correct main named entity words and category labels. Specific evaluation indicators include: precision, recall, and F1 value.
[0129] 1. Precision indicates the accuracy of the recognition result. The calculation formula is shown in (19):
[0130]
[0131] 2. Recall rate indicates the degree to which positive samples are accurately predicted in the entire corpus. The calculation is shown in formula (20):
[0132]
[0133] 3. The F1 value is the weighted harmonic mean of precision and recall, and is calculated as shown in formula (21):
[0134]
[0135] Example:
[0136] In order to evaluate the effectiveness of the proposed NNER-ARC model, this paper selects some representative nested named entity recognition in recent years for comparison, mainly including:
[0137] Seq2Seq-BERT (2020): Treats the nested named entity recognition problem as a sequence-to-sequence problem.
[0138] BERT-MRC (2020): Treats the named entity recognition problem as a machine reading comprehension question answering approach for extraction.
[0139] LEAR (2021): Improves on the BERT-MRC model by encoding text and label annotations separately, and integrating label knowledge into text representation for entity extraction through a designed semantic fusion module.
[0140] W2NER (2022): Converts the named entity recognition task into predicting word-word relationship categories, making it possible to simultaneously handle three named entity tasks: non-nested entities, nested entities, and non-continuous entities.
[0141] CNN-NER (2023): Utilizes Convolutional Neural Networks (CNNs) to model spatial relationships in the score matrix for entity extraction.
[0142] The complex electromechanical equipment fault data set used in the experiment comes from a high-speed train partial fault data set provided by a research institute. The entity types in the data are defined, labeled and downloaded using the open source data annotation tool Label Studio. The downloaded data set uses the random.shuffle() function in Python to shuffle the data order and divide the data according to a certain ratio. The detailed statistical information is shown in Table 1.
[0143] Table 1 Statistics of complex electromechanical equipment fault data set
[0144]
[0145] There are 7 types of entities in this dataset, and the label annotations corresponding to each type of entity are shown in Table 2.
[0146] Table 2 Label annotations corresponding to entity types in the complex electromechanical equipment fault dataset
[0147]
[0148]
[0149] In terms of experimental settings, the data division strategy adopted is: 80% of the data is used as a training set, 10% as a validation set, and the remaining 10% as a test set. The model uses BERT-large to initially embed text information and label annotations, and the resulting embedding dimension is 1024; in terms of the choice of optimizer, the AdamW algorithm is used.
[0150] Regarding the setting of learning rate, a dynamic adjustment strategy is adopted, which is divided into two stages: warm-up period and linear decay period. In the warm-up period, the learning rate will gradually increase, which helps the model to quickly converge to a more ideal state in the early stage of training. Subsequently, in the linear decay period, the learning rate will decrease linearly according to the preset total number of training steps and warm-up steps. The specific parameter settings are shown in Table 3.
[0151] Table 3 Parameter settings
[0152]
[0153] The comparison of the results of different nested named entity recognition models is shown in Table 4.
[0154] Table 4 Comparison of results of different models for complex electromechanical equipment fault dataset
[0155]
[0156] Since the F1 value in the evaluation index takes into account both precision and recall, it is used as an indicator for comprehensive evaluation of model performance. The NNER-ARC model improves the best effect by 0.09% compared with the baseline model, verifying the effectiveness of the improvements made by the model of the present invention.
[0157] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the above specific implementation methods. Therefore, any modification or equivalent replacement of the present invention; and all technical solutions and improvements thereof that do not depart from the spirit and scope of the invention are included in the scope of the claims of the present invention.
Claims
1. A method for nested named entity recognition in adversarial reading comprehension, characterized in that: The following steps are involved: Step 1: Initially embed the input text information and label annotations using the BERT model fine-tuned by the complex mechatronics domain dictionary, and then further encode the text information through Star-Transformer to capture deeper semantic features; Step 1.1: Implement the initial embedding of text information and label annotation information; Given the input text information X and the label annotation Y, the initial embedding is performed using BERT fine-tuned with a dictionary in the field of complex electromechanical equipment, and the following representation results are obtained: E X =bert(X) (1) AND Y =bert(Y) (2) Among them, E X ∈R n×d and E Y ∈R C×m×d , n represents the length of text information X, m represents the length of label annotation Y, |CI represents the size of category set C, and d represents the vector dimension of BERT encoder; Step 1.2: Deepen the internal information of the text to obtain richer text information; After the initial embedding of the text information, the Star-Transformer encoder is introduced to encode the text information of length n; let E X =[e1, e2, ..., e n ], where e i ∈R 1×d represents the embedding representation of the i-th token, and at the same time, let s t ∈R 1×d represents the state of the relay node at step t, H t ∈R n×d Denote the states of all n satellite nodes at step t, and use H 0 =E X and 0 =average(E X ) to initialize the state, average means the average operation; Step 2: The label annotation information is analyzed by using the attention mechanism, average pooling and cosine similarity to fully explore the semantic information in the label annotation. After the text information and the label annotation information are fused, the model uses the FGM algorithm to calculate the noise interference and add it to obtain the text feature representation containing small disturbances. Step 2.1: Map the text information embedding representation and the label annotation embedding representation to the same feature space; First, H X and E Y They are input into the fully connected layer respectively, the purpose of which is to map the text information embedding representation and the label annotation embedding representation to the same feature space, calculated by the following formula: h X_fc =W X ·H X (3) h Y_fc =W Y ·HAVE BEEN Y (4) Among them, W X ∈R d×d and W Y ∈R d×d are learnable parameters of the fully connected layer; Step 2.2: Use the attention mechanism to aggregate the label annotation information into the text information; For any 1≤i≤n, let x i is the i-th token of the text information X; for The i-th row of ; for any 1≤j≤m and category c∈C, let is the jth token of the label annotation Y of category c, so the attention mechanism is used to calculate the embedding representation h of the label annotation Y_att , and its calculation formula is as follows: The calculation process of formulas (5) and (6) is repeated for all categories to obtain the embedding representation h of the label annotation. Y_att ; Step 2.3: Aggregate the label annotation information into text information using average pooling; The embedding representation h of the label annotation is calculated using the average pooling method Y_ave , and its calculation formula is as follows: The calculation process of formula (7) is repeated for all categories to obtain the embedding representation h of the label annotation. Y_ave ; Step 2.4: Aggregate the label annotation information into the text information using cosine similarity calculation; By calculating the text feature h X_fc Annotate with label h Y_fc The cosine similarity between them is calculated and the calculated cosine similarity score is multiplied by the label annotation vector to obtain the weighted label feature representation h Y_cos , and its calculation formula is as follows: in, Yes Y_fc The corresponding embedding representation of category c is obtained by repeating the calculation process of formulas (8) and (9) for all categories to obtain the embedding representation h of the label annotation. Y_cos ; Step 2.5: The information learned from steps 2.2 to 2.4 is integrated into the text features through learnable weight parameters; Through the attention mechanism, average pooling and cosine similarity, different label annotation information is obtained respectively. The following calculation formula is used to integrate the label annotation information into the text information: h fused =h X_fc +a·h Y_att +β·h Y_ave +λ·h Y_cos (10) Among them, α, β, and λ are all learnable parameters; Step 2.6: Add noise interference; After the text information is fused with the label annotation information, the FGM algorithm is used to generate adversarial perturbations and add the perturbations to the fused text information h fused The calculation formula is as follows: h adv =r adv +h fused (12) Among them, r adv is the current input perturbation; ε is a configurable hyperparameter that determines the magnitude of the adversarial perturbation; To obtain the gradient of x, x represents the model input; L(θ, x, y) is the loss corresponding to the sample, y is the corresponding label, and θ is all the parameters of the current model; Step 3: Calculate the probability of each token in the text being the starting or ending position of each entity, and finally output the predicted entity range and its corresponding category; According to the label sequence output by the model, the starting and ending positions of each entity are determined to obtain the span information of the entity; the start / end tag mode is used to annotate the target range to be extracted; specifically, for each token x i The following calculation formula is used to calculate the probability of the starting position: in, Indicates x i is the probability of the span of category c at the start position; M s ∈R |C|×d and b s ∈R d All are learnable parameters; represents primitive multiplication; f o (·) is a function that sums the rows of the input means; similarly, use Indicates x i is the probability of the ending position, and the calculation process is similar; In nested named entity recognition, spans in the same category can be nested or overlapped, which makes the heuristic matching principle no longer work. Therefore, the solution in the BERT-MRC model is adopted; a binary classifier is used to predict a pair of candidate start / end positions as the span; specifically, for any category c, the following binary classifier is defined: Among them, 1≤i, j≤n; M∈R 1×2d is a learnable parameter; when Coming soon i , x j The range prediction is category C; Step 4: Use the loss function to measure the difference between the model output and the true label, and minimize this difference to train the model.
2. According to claim 1, a method for nested named entity recognition in adversarial reading comprehension is characterized in that: The step 4 is specifically as follows: In order to measure the difference between the model output and the true label, and to minimize this difference to train the model; therefore, refer to the definition of the relevant loss function in the LEAR model, and its calculation formula is as follows: L total =μ(L s +L e )+vL match (18) Among them, L s represents the initial loss, L e Indicates the end of loss, L match represents the matching loss, L total represents the final total loss; CE(x,y) represents the cross entropy; S c ∈{0, 1} n , if and only if x i is the actual starting position of category c. E c ∈{0, 1} n , if and only if x i Is the actual end position of category c? M c ∈R n×n is a binary matrix if and only if token x i and x j When all are classified as the true range of category c, W c ∈R n×n is also a binary matrix if and only if or When there is μ, v are learnable hyperparameters in the final loss function.
Citation Information
Patent Citations
Named entity recognition model training method, named entity recognition method and medium
CN111738003A
Named entity recognition method based on global pointer and adversarial training
CN115688782A
Named entity identification method and system
CN117933258A
Fine-tuning multi-head network from a single transformer layer of pre-trained language model
US20230115321A1
