A Named Entity Joint Recognition Method and System Based on Pointer Networks

Abstract

This invention discloses a joint named entity recognition method and system based on pointer networks. The method inputs the vectors of character integer indices and position indices after text segmentation into a BERT model to obtain a fusion context vector. The fusion context vector is then input into a sequence decoding model and a pointer decoding model for joint training to obtain entity probability distributions, entity start-point probability distributions, and entity end-point probability distributions. The joint training involves: training the sequence decoding model to stability, then jointly training the sequence decoding model and the pointer decoding model to stability, and finally training the pointer decoding model to stability. Finally, joint decoding is performed to obtain entity labels. This invention proposes a joint model based on BERT's CRF and pointer networks. Through a multi-task alternating training strategy, both the CRF and pointer networks are fully trained, effectively reducing the entity boundary recognition error rate in NER tasks and improving the model's generalization ability.

Description

Technical Field

[0001] This invention relates to a named entity recognition method and system, and more particularly to a named entity joint recognition method and system based on pointer networks. Background Technology

[0002] Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), serving as a crucial tool for numerous NLP tasks such as information extraction, question answering systems, syntactic analysis, and machine translation. The accuracy of NER directly impacts the performance of downstream tasks. The evaluation criteria for NER performance primarily focus on the correctness of entity boundary delineation and entity type labeling. Entity boundary errors are among the most common error types; reducing their frequency can significantly improve NER model metrics. However, current techniques using Pointer Networks (SPAN) instead of Conditional Random Fields (CRF) improve boundary recognition accuracy but sacrifice the advantages of CRF in sequence labeling, thus affecting the overall sentence recognition accuracy. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide a named entity joint recognition method with high recognition accuracy. The second purpose of this invention is to provide a named entity joint recognition system with high recognition accuracy.

[0004] Technical solution: The named entity joint recognition method based on pointer networks described in this invention includes the following steps:

[0005] (1) Perform word segmentation on the text and obtain the integer index and position index of each character after word segmentation;

[0006] (2) Input the vector sum of the integer index and the position index into the BERT model to obtain the vector sum of the fusion context;

[0007] (3) Jointly train the vector and input sequence decoding model and the pointer decoding model of the fusion context to obtain the entity probability distribution, entity start probability distribution and entity end probability distribution;

[0008] The joint training is as follows: after training the sequence decoding model to stabilize it, the sequence decoding model and the pointer decoding model are jointly trained to stabilize them, and finally the pointer decoding model is trained to stabilize it.

[0009] (4) Jointly decode the entity probability distribution, entity start probability distribution and entity end probability distribution to obtain entity labels.

[0010] Furthermore, in the joint training described in step (3), the path loss of the sequence decoding model is calculated, the cross-entropy loss of the entity start probability distribution, the entity end probability distribution and the text label is calculated, and the parameters of the sequence decoding model and the pointer decoding model are updated through the backpropagation algorithm.

[0011] Furthermore, the loss function for joint training in step (3) is:

[0012]

[0013] in, For hyperparameters, l span The cross-entropy loss is calculated using the entity origin probability distribution, entity destination probability distribution, and text label. crf This is the inverse of the path score for the sequence decoding model.

[0014] Furthermore, the joint training described in step (3) is as follows: Let Train the sequence decoding model, and after the sequence decoding model converges, let After jointly training the sequence decoding model and the pointer decoding model and achieving convergence, let... Train the pointer decoding model.

[0015] Furthermore, the cross-entropy loss between the entity start probability distribution, the entity end probability distribution, and the text label is:

[0016]

[0017] Where the subscript c represents the entity category, k is the number of entity categories, and y c p is a one-hot vector of the text label. start Let p be the probability distribution of the entity's origin. end This represents the probability distribution of the entity's endpoint.

[0018] Further, the joint decoding in step (4) is as follows:

[0019]

[0020] in, This is the normalized result of the entity probability distribution p. The probability distribution p of the entity's origin start The normalization and extension results, Let p be the probability distribution of the entity endpoint. end The normalization and extension results.

[0021] For p start The normalized result of the transformation is m*l*(2k+1), p startThe probability of an entity starting in p corresponds one-to-one with the probability of an entity starting in p. start The probability of other classes in p is expanded to the probability of the entity body in p being other classes;

[0022] For p end The normalized result of the transformation is m*l*(2k+1), p end The probability of the middle entity ending is expanded to p, the probability of the middle entity's torso. end The probabilities of other classes in p are expanded to the probabilities of the entity starting point and other classes in p. Further, the integer index of the character in step (1) includes: the integer index of the character in the pre-trained vocabulary and the integer index of the character in the sentence, with values ​​of {0,1}.

[0023] Further, the pointer decoding model in step (3) includes a first fully connected layer and a second fully connected layer. The vector sum of the fusion context enters the first fully connected layer to obtain the entity starting point probability distribution, and then is concatenated with itself as a vector. It then enters the second fully connected layer and obtains the entity ending point probability distribution after layer normalization.

[0024] The sequence decoding module is a discriminant model that combines the maximum entropy model and the hidden Markov model.

[0025] The pointer-based named entity joint recognition system of the present invention includes:

[0026] The text segmentation module is used to segment text into words and obtain the integer index and position index of each character after segmentation.

[0027] The joint training module is used to input the vectors of the integer index and the position index into the BERT model to obtain the vector sum of the fusion context; and to jointly train the vector sum of the fusion context and the input sequence decoding model and the pointer decoding model to obtain the entity probability distribution, the entity start probability distribution and the entity end probability distribution.

[0028] The joint training is as follows: after training the sequence decoding model to stabilize it, the sequence decoding model and the pointer decoding model are jointly trained to stabilize them, and finally the pointer decoding model is trained to stabilize it.

[0029] The joint decoding module is used to jointly decode the entity probability distribution, the entity start probability distribution, and the entity end probability distribution to obtain entity labels.

[0030] The computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the named entity joint recognition method based on pointer networks.

[0031] Beneficial effects: Compared with the prior art, the advantages of the present invention are: (1) The boundary enhancement named entity recognition algorithm based on pointer network disclosed in the present invention proposes a joint model based on BERT CRF and pointer network, which combines the advantages of CRF in sequence labeling and utilizes the accuracy of pointer network in boundary recognition, which can effectively reduce the error rate of entity boundary recognition in NER tasks; (2) Multi-task joint training can improve the generalization ability of the model. In addition, compared with the approach of using model fusion or introducing additional prior information, this model has a faster inference speed, smaller memory footprint, and is more suitable for processing massive samples under field conditions; (3) Considering the similarity of input requirements between CRF and pointer network, a multi-task alternating training strategy is designed, which can enable the model to converge quickly and effectively; after convergence, the weights of BERT and CRF are fixed, and the pointer network is trained separately, so that both CRF and pointer network are fully trained in the end. Attached Figure Description

[0032] Figure 1 This is a flowchart of the named entity joint identification method of the present invention.

[0033] Figure 2 This is a loss curve of CRF and SPAN trained separately in an embodiment of the present invention.

[0034] Figure 3 This is a diagram showing the entity extraction results in an embodiment of the present invention. Detailed Implementation

[0035] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0036] In this example, the input text is: "[Tongcheng Travel] Booking successful: Please bring your ID card to the front desk and report *** for check-in. 2022 / 5 / 7, * City Jiangcheng Business Hotel, Standard Double Room, 1 room 1 night, Total price ¥128.0, Address: No. 2, ** Street, ** Middle Road, 2-2 to 2-18B, Tel: 023-4040****. Changes and cancellations are allowed before 12:00 on May 6th. Changes and cancellations after 12:00 on May 6th will incur a deduction of the full room rate."

[0037] like Figure 1 As shown, the named entity joint recognition method based on pointer networks includes the following steps:

[0038] Step 1: Tokenize the text and obtain the token_id corresponding to each character after tokenization, the sentence_type_id corresponding to the sentence type, and the position_id corresponding to the position. This step is accomplished using the BertTokenizer class in the transformers module.

[0039] The token_id is the integer index of the character in the pre-trained vocabulary, the sentence_type_id is the integer index of the character in the sentence, and the value is {0,1}. The position_id is the absolute position index of the character in the sentence.

[0040] Step 2: Obtain the corresponding embedding based on token_id, sentence_type_id, and position_id, and input it into the pre-trained BERT model to obtain an embedding that incorporates the context; the pre-trained BERT model is the BERT-Base model provided by Google.

[0041] The embedding is the sum of vectors obtained by querying the embedding space based on token_id, sentence_type_id, and position_id respectively. The embedding with context is an m*l*768 matrix, where m and l are the number of sentences in the batch and the maximum sentence length, respectively, and 768 is the dimension of the embedding space.

[0042] In this embodiment, the number of sentences in the batch is 1, and the maximum sentence length is 128.

[0043] Step 3: Input the embedding with context into the sequence decoding model and the pointer decoding model for decoding, and obtain the probability distribution p of the entity after decoding by the sequence decoding model and the probability distribution p of the entity's starting point after decoding by the pointer decoding model, respectively. start and the probability distribution of the entity endpoint p end In this embodiment, the parameters of the sequence decoding model and the pointer decoding model are pre-trained parameters.

[0044] The sequence decoding model is a discriminative model that combines the characteristics of maximum entropy models and hidden Markov models, suitable for sequence prediction tasks. The pointer decoding model consists of two fully connected layers: one to determine the start position of an entity and the other to determine the end position. The input, after passing through the BERT model, becomes an embedding that integrates contextual semantics and prior information. This embedding is then copied twice: one copy enters the sequence decoding model as the emission probability to obtain the entity's probability distribution p; the other copy enters the start layer of the pointer decoding model to obtain the entity's origin probability distribution p. start Then, it is concatenated with its own vector, and then enters the final layer. After layer normalization, the probability distribution p of the entity's endpoint is obtained. end .

[0045] The decoded entity probability distribution p is an m*l*(2k+1) matrix, representing the probability distribution of entity labels predicted by the sequence decoding model; the entity starting probability distribution pstart Let p be an m*l*(k+1) matrix, representing the confidence level of the entity starting point predicted by the pointer decoding model; p is the probability distribution of the entity ending point. end Let be an m*l*(k+1) matrix, representing the confidence level of the entity endpoint predicted by the pointer decoding model; k is the number of entity categories.

[0046] The joint model of sequence decoding and pointer decoding, based on sequence decoding, only adds 607,619 parameters by sharing BERT parameters, an increase of 0.56% in parameter count. In contrast, a model fusion algorithm with the same effect requires 108,917,791 additional parameters, an increase of over 100%. Comparatively, the joint decoding model offers faster inference speed, smaller memory footprint, and is more suitable for handling massive amounts of samples in real-world conditions.

[0047] Step 4: Joint training, calculate the CRF path loss and the entity origin probability distribution p separately. start and the probability distribution of the entity endpoint p end The model parameters are updated using the backpropagation algorithm based on the cross-entropy loss of the text labels and the training strategy. The text labels are m*l matrices, with each element representing a real entity label.

[0048] The challenge of joint training lies in the initial stage, where the CRF loss is several orders of magnitude larger than the SPAN loss. If training begins by directly adding them together, the gradient of the SPAN module becomes negligible, resulting in ineffective parameter updates and reduced decoding performance. Even after the CRF loss decreases to the order of magnitude of the SPAN loss, weighted training is still implemented. However, due to the shorter training time of the SPAN module, the decoding performance will ultimately be inferior to that of the standalone SPAN model. The loss curves during CRF and SPAN training are shown below. Figure 2 As shown.

[0049] Therefore, considering the similarity in input requirements between CRF and SPAN, this invention designs a multi-task alternating training strategy, which enables the model to converge quickly and effectively. After convergence, the weights of BERT and CRF are fixed, and the SPAN network is trained separately, ultimately ensuring that both CRF and SPAN are fully trained.

[0050] In this embodiment, the joint model improves the whole sentence accuracy of the test set from 95.5% in the BERT-CRF single-task model to 97.9%.

[0051] Step 5, Inference Phase: Determine the probability distribution p of the entities after CRF decoding and the probability distribution p of the entity starting point after SPAN decoding. start and the probability distribution of the entity endpoint p endThe three items are decoded to obtain the prediction result pred, and finally the entity label is obtained.

[0052] The prediction result pred is an m*l matrix, and each element can be decoded into an entity label.

[0053] In this embodiment, the entity extraction result is as follows: Figure 3 As shown.

[0054] Based on the same inventive concept, the pointer-network-based named entity joint recognition system of the present invention includes:

[0055] The text segmentation module is used to segment text into words and obtain the integer index and position index of each character after segmentation.

[0056] The joint training module is used to input the vectors of the integer index and the position index into the BERT model to obtain the vector sum of the fusion context; and to jointly train the vector sum of the fusion context and the input sequence decoding model and the pointer decoding model to obtain the entity probability distribution, the entity start probability distribution and the entity end probability distribution.

[0057] The joint training is as follows: after training the sequence decoding model to stabilize it, the sequence decoding model and the pointer decoding model are jointly trained to stabilize them, and finally the pointer decoding model is trained to stabilize it.

[0058] The joint decoding module is used to jointly decode the entity probability distribution, the entity start probability distribution, and the entity end probability distribution to obtain entity labels.

[0059] Based on the same inventive concept, the computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the named entity joint recognition method based on pointer networks.

[0060] The computer-readable storage medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer.

Claims

1. A named entity joint recognition method based on pointer networks, characterized in that, Includes the following steps: (1) Segment the text into words and obtain the integer index of the character and the position index of the character after segmentation; the integer index of the character includes: the integer index of the character in the pre-trained vocabulary and the integer index of the character in the sentence, with a value of {0,1}; (2) Input the vector sum of the integer index and the position index into the BERT model to obtain the vector sum of the fusion context; (3) Jointly train the vector and input sequence decoding model and the pointer decoding model of the fusion context to obtain the entity probability distribution, entity start probability distribution and entity end probability distribution; The joint training is as follows: after training the sequence decoding model to stabilize it, the sequence decoding model and the pointer decoding model are jointly trained to stabilize them, and finally the pointer decoding model is trained to stabilize it. (4) Jointly decode the entity probability distribution, entity start probability distribution and entity end probability distribution to obtain entity labels.

2. The named entity joint recognition method based on pointer networks according to claim 1, characterized in that, In the joint training described in step (3), the path loss of the sequence decoding model is calculated, the cross-entropy loss of the entity start probability distribution, the entity end probability distribution and the text label is calculated, and the parameters of the sequence decoding model and the pointer decoding model are updated through the backpropagation algorithm.

3. The named entity joint recognition method based on pointer networks according to claim 2, characterized in that, The loss function for joint training in step (3) is: in, For hyperparameters, l span The cross-entropy loss is calculated using the entity origin probability distribution, entity destination probability distribution, and text label. crf This is the inverse of the path score for the sequence decoding model.

4. The named entity joint recognition method based on pointer networks according to claim 3, characterized in that, The joint training mentioned in step (3) is as follows: Let Train the sequence decoding model, and after the sequence decoding model converges, let After jointly training the sequence decoding model and the pointer decoding model and achieving convergence, let... Train the pointer decoding model.

5. The named entity joint recognition method based on pointer networks according to claim 2, characterized in that, The cross-entropy loss between the entity start probability distribution, the entity end probability distribution, and the text label is: Where the subscript c represents the entity category, k is the number of entity categories, and y c p is a one-hot vector of the text label. start Let p be the probability distribution of the entity's origin. end This represents the probability distribution of the entity's endpoint.

6. The named entity joint recognition method based on pointer networks according to claim 1, characterized in that, The joint decoding in step (4) is as follows: in, This is the normalized result of the entity probability distribution p. The probability distribution p of the entity's starting point start The normalization and extension results, Let p be the probability distribution of the entity endpoint. end The normalization and extension results.

7. The named entity joint recognition method based on pointer networks according to claim 1, characterized in that, The pointer decoding model in step (3) includes a first fully connected layer and a second fully connected layer. The vector sum of the fusion context enters the first fully connected layer to obtain the entity starting point probability distribution, and then is concatenated with its own vector. Then it enters the second fully connected layer, and after layer normalization, the entity ending point probability distribution is obtained. The sequence decoding module is a discriminant model that combines the maximum entropy model and the hidden Markov model.

8. A named entity joint recognition system based on pointer networks, characterized in that, include: The text segmentation module is used to segment text into words and obtain the integer index and position index of each character after segmentation. The integer index of the character includes: the integer index of the character in the pre-trained vocabulary and the integer index of the character in the sentence, with a value of {0,1}; The joint training module is used to input the vectors of the integer index and the position index into the BERT model to obtain the vector sum of the fusion context; and to jointly train the vector sum of the fusion context and the input sequence decoding model and the pointer decoding model to obtain the entity probability distribution, the entity start probability distribution and the entity end probability distribution. The joint training is as follows: after training the sequence decoding model to stabilize it, the sequence decoding model and the pointer decoding model are jointly trained to stabilize them, and finally the pointer decoding model is trained to stabilize it. The joint decoding module is used to jointly decode the entity probability distribution, the entity start probability distribution, and the entity end probability distribution to obtain entity labels.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the named entity joint recognition method based on pointer networks according to any one of claims 1-7.

Images