A named entity recognition method for Chinese electronic medical records based on active learning
Through an active learning method, the naming entity recognition model is optimized using cross-entropy loss, the problem of naming entity recognition in Chinese electronic medical records is solved, and the recognition accuracy and efficiency are improved.
Patent Information
- Application Number
- CN202211100112.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-09
- Publication Date
- 2025-02-11
- Estimated Expiration
- 2042-09-09
AI Technical Summary
The prior art is difficult to effectively identify and extract named entities in Chinese electronic medical records, and the lack of high-quality labeled data affects the training and verification of the model.
Using an active learning method, the main network and subnet of the model are identified through the trained named entity, and the cross entropy loss is used as a new label to gradually optimize the model parameters until the preset F1 value condition is reached.
It improves the accuracy and efficiency of identifying named entities in Chinese electronic medical records, reduces the workload of manual labeling, and can effectively identify the samples with the richest amount of information.
Smart Images
Figure CN115440330B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of text annotation, and in particular relates to a Chinese electronic medical record named entity recognition method based on active learning. Background Art
[0002] With the development of detection technology and disease diagnosis methods in the medical field, doctors are increasingly using electronic documents to record relevant information about patients and various disease characterization data. As a result, a large number of electronic medical record texts have been generated. How to effectively extract effective information from a large amount of electronic medical record text data so that medical staff can use the medical data analysis system conveniently and quickly is now a problem faced by artificial intelligence in the medical field. Faced with a huge amount of electronic medical record text information, doctors must quickly and effectively extract the most effective information for disease diagnosis, such as the name of the disease, the site of onset, the corresponding symptoms, etc. However, medical electronic reports usually contain some less relevant information, so it is very necessary to develop relevant technologies to help doctors quickly eliminate the interference of irrelevant information and extract effective information.
[0003] Among them, named entity recognition of Chinese electronic medical records is the upstream work of medical information processing. Named entity recognition refers to the identification of entities with specific meanings in text. However, the resources of annotated Chinese electronic medical records are currently very scarce, and how to obtain effective and high-quality annotated data has become an urgent problem to be solved. Active learning is a subfield of machine learning. The idea of active learning is to selectively select samples for manual annotation based on a certain screening strategy through unlabeled data, thereby reducing the workload of manual annotation and maximizing model performance. The existing widely used active learning strategies can be roughly divided into the following types:
[0004] (1) Uncertainty-based active learning strategy: Uncertainty sampling can be used to identify unlabeled samples near the decision boundary of the current machine learning model. The most informative examples here are the ones that the classifier is most uncertain about. The samples that the model is most uncertain about may be data near the classification boundary. And our model learning algorithm will obtain more information about the class boundary by observing these most difficult samples to classify.
[0005] (2) Active learning strategy based on representativeness: Representative sampling is a common strategy based on data distribution. The algorithm ensures that the queried samples can cover the entire data distribution to ensure the diversity of the labeled data.
[0006] However, the uncertainty-based active learning strategy has a very strong assumption that all samples are independent and identically distributed, which is not always the case in the real world. Compared with the uncertainty-based active learning strategy, the representativeness-based active learning strategy can more effectively select samples with different characteristics, but it cannot effectively identify the samples with the richest information for the model; and in the scenario of named entity recognition of Chinese electronic medical records, there are very high requirements for sentence-level accuracy. Summary of the invention
[0007] In order to solve the above technical problems, the present invention proposes a Chinese electronic medical record named entity recognition method based on active learning, comprising:
[0008] Input the Chinese electronic medical record text to be recognized into the main network of the trained named entity recognition model for recognition, and obtain the recognition result;
[0009] The named entity recognition model includes: a main network and a sub-network;
[0010] The training process of the named entity recognition model includes:
[0011] S1: Obtain unlabeled Chinese electronic medical record texts, select parts from them for manual annotation, and divide them into training set and validation set in a ratio of 4:1;
[0012] S2: Input the text in the training set into the main network of the named entity recognition model, train the main network, obtain the F1 value based on the recognition results of the trained main network on the validation set, save the model parameters with the largest F1, and fix the parameters of the main network;
[0013] S3: Calculate the cross entropy loss of the tokens in the training set;
[0014] S4: Use the cross entropy loss of the text token as the new label of the training set and the validation set, input the training set with the cross entropy loss label into the sub-network of the named entity recognition model, train the sub-network, and save the parameters of the sub-network model with the smallest loss on the validation set according to the loss of the sub-network on the validation set after training, and fix the parameters of the sub-network;
[0015] S5: Input the unlabeled text into the sub-network of the named entity recognition model with fixed parameters, obtain the loss prediction value of each token of the text, select the text with the largest text loss prediction value, annotate the text with the text loss prediction value, and add it to the training set and validation set. Repeat the above process until the F1 value of the main network of the named entity recognition model on the validation set reaches the preset conditions of the model, and the model training is completed.
[0016] Preferably, the Chinese electronic medical record text to be recognized is input into a trained named entity recognition model for recognition, and a recognition result is obtained, which specifically includes:
[0017] Step 1: Divide the Chinese electronic medical record text to be recognized into text segments with a length less than or equal to 256. If the length of the text is less than 256, the sequence is padded with padding.
[0018] Step 2: Perform token embedding, sentence embedding, and position embedding on the divided text;
[0019] Step 3: Input the encoded text into the named entity recognition model, obtain the output of the main network of the named entity recognition model, perform Viterbi decoding, and obtain the final recognition result.
[0020] Preferably, the main network structure of the named entity recognition model is BERT+BILSTM+CRF, and the sub-network structure is BERT+BILSTM+LINEAR, wherein the weights of BERT+LSTM are shared with the main network.
[0021] Preferably, it is characterized in that training the main network specifically includes:
[0022] Input the text in the training set into the BERT layer of the main network for text segmentation, encode the segmented text through the BILSTM layer to obtain the encoding vector of each word in the text, input the encoding vector of each word in the text into the CRF layer to obtain the predicted labels corresponding to all words in the text, and calculate the predicted label scores of all words in the text. According to the predicted label scores, softmax is used to calculate the probability of the correct label text, and the probability of the correct label text is maximized. According to the maximized probability of the correct label text, the encoding vector of each word in the text is decoded to obtain the optimal text sequence, and the main network under the output optimal sequence is used as the optimal main network obtained by training.
[0023] Furthermore, the predicted label score of the words in the text is calculated, expressed as:
[0024]
[0025] Among them, s(x,y) represents the predicted label score of the word in the text, x represents the text sequence, y represents the predicted label sequence, and n is the sequence length. Indicates that the i-th character in the input sequence is predicted to be the y-th character i label scores, and A represents the transfer matrix.
[0026] Furthermore, softmax is used to calculate the probability of the correct label text based on the predicted label score, and the probability of the correct label text is maximized, which is expressed as:
[0027]
[0028] Among them, P(y|x) represents the probability of the correct label text, s(x,y) represents the predicted label score of the word in the sample text, and s(x,y ～ ) represents the correct label score, represents the sum of all possible label scores, x represents the text sequence, y represents the predicted label sequence, Y x represents the set of all possible label sequences, y ～ Represents a label sequence in the set of all possible label sequences.
[0029] Preferably, the F1 value is obtained according to the recognition result of the main network on the validation set after training, which is expressed as:
[0030]
[0031] Among them, c represents the total number of entity categories, F 1k represents the F1 value of the k-th entity.
[0032] Preferably, the cross entropy loss of the token of the text is calculated, expressed as:
[0033]
[0034] Among them, t i is the cross entropy loss of the i-th token in a sample, y ij Indicates whether the i-th token belongs to the j-th category. If yes, it is 1, otherwise it is 0. ij represents the probability that the i-th token belongs to the j-th category, P represents the output matrix of bilstm, c is the number of labels, P i,j represents the output of bilstm on the jth dimension of the i-th token, and e represents the natural base.
[0035] Preferably, the training of the sub-network is expressed as:
[0036]
[0037] Among them, L sub represents the loss of the sub-network, S represents the sentence length, B represents the size of a training batch, and t ij represents the cross entropy loss of the i-th token in the j-th sample in a batch, Represents the sub-network's predicted value of the loss of the i-th token in the j-th sample in a batch.
[0038] The beneficial effects of the present invention are as follows: by letting the model predict its own loss, it can find the sample that can make the biggest change to itself. The model does not care about the specific uncertainty or the difference of the samples, but only cares about the difference between its own predicted value and the true label. At the same time, in order to meet the high requirements for sentence-level named entity recognition accuracy in the scenario of Chinese electronic medical records, the sampling strategy is adjusted to the maximum value of all token losses in a sentence for screening, because in the sentence-level accuracy evaluation, any token prediction error will be classified as an incorrect prediction, which is similar to the idea of filling the last short board. The value of the token with the largest loss is used as the loss value of the sample for sorting and screening. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A flowchart of the training process of the named entity recognition model of the present invention;
[0040] Figure 2 It is a schematic diagram of the model structure of the present invention. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] A named entity recognition method for Chinese electronic medical records based on active learning, comprising:
[0043] Input the Chinese electronic medical record text to be recognized into the main network of the trained named entity recognition model for recognition, and obtain the recognition result;
[0044] The named entity recognition model includes: a main network and a sub-network;
[0045] The training process of the named entity recognition model includes: Figure 1 As shown:
[0046] S1: Obtain unlabeled Chinese electronic medical record texts, select parts from them for manual annotation, and divide them into training set and validation set in a ratio of 4:1;
[0047] S2: Input the text in the training set into the main network of the named entity recognition model, train the main network, obtain the F1 value based on the recognition results of the trained main network on the validation set, save the model parameters with the largest F1, and fix the parameters of the main network;
[0048] S3: Calculate the cross entropy loss of the tokens in the training set;
[0049] S4: Use the cross entropy loss of the text token as the new label of the training set and the validation set, input the training set with the cross entropy loss label into the sub-network of the named entity recognition model, train the sub-network, and save the parameters of the sub-network model with the smallest loss on the validation set according to the loss of the sub-network on the validation set after training, and fix the parameters of the sub-network;
[0050] S5: Input the unlabeled text into the sub-network of the named entity recognition model with fixed parameters, obtain the loss prediction value of each token of the text, select the text with the largest text loss prediction value, annotate the text with the text loss prediction value, and add it to the training set and validation set. Repeat the above process until the F1 value of the main network of the named entity recognition model on the validation set reaches the preset conditions of the model, and the model training is completed.
[0051] Preprocessing of a large amount of unlabeled data, because the BERT model can only receive up to 512 tokens, long texts need to be segmented to meet the BERT input limit. Sentences are segmented by punctuation marks. In order to improve the reasoning speed while maintaining the richness of information, the sentence input length is 256. When segmenting sentences, punctuation marks are first used to ensure the integrity of the sentence, and then the length of the entire sentence is ensured to be less than or equal to 256. If it is less than 256, it is supplemented by padding.
[0052] The named entity recognition model, such as Figure 2 As shown, it includes two parts: a backbone network and a sub-network. The main network structure is BERT+BILSTM+CRF, and the sub-network structure is BERT+BILSTM+LINEAR, where the weights of BERT+LSTM are shared with the main network.
[0053] The Chinese electronic medical record text to be recognized is input into the trained named entity recognition model for recognition, and the recognition results are obtained, including:
[0054] Step 1: Divide the Chinese electronic medical record text to be recognized into text segments with a length less than or equal to 256. If the length of the text is less than 256, the sequence is padded with padding.
[0055] Step 2: Perform token embedding, sentence embedding, and position embedding on the divided text;
[0056] Step 3: Input the encoded text into the named entity recognition model, obtain the output of the main network of the named entity recognition model, perform Viterbi decoding, and obtain the final recognition result.
[0057] Training the main network includes:
[0058] Input the text in the training set into the BERT layer of the main network for text segmentation, encode the segmented text through the BILSTM layer to obtain the encoding vector of each word in the text, input the encoding vector of each word in the text into the CRF layer to obtain the predicted labels corresponding to all words in the text, and calculate the predicted label scores of all words in the text. According to the predicted label scores, softmax is used to calculate the probability of the correct label text, and the probability of the correct label text is maximized. According to the maximized probability of the correct label text, the encoding vector of each word in the text is decoded to obtain the optimal text sequence, and the main network under the output optimal sequence is used as the optimal main network obtained by training.
[0059] Calculate the predicted label scores for all words in the text, expressed as:
[0060]
[0061] Where P∈R n×c is the output matrix of bilstm, n is the sequence length, c is the number of labels, Indicates that the i-th character in the input sequence is predicted to be the y-th character i label scores, A represents the transfer matrix, A∈R c+2×c+2 , A ij represents the probability (score) of transferring from label i to label j.
[0062] According to the predicted label score, softmax is used to calculate the probability of the correct label text, and the probability of the correct label text is maximized, which is expressed as:
[0063]
[0064] Among them, s(x,y) represents the correct label score, s(x,y ～ ) represents the predicted label score, Represents the sum of all possible label score indices.
[0065] The F1 value is obtained based on the recognition results of the main network on the validation set after training, expressed as:
[0066]
[0067] Among them, c represents the total number of entity categories, F 1k represents the F1 value of the k-th entity.
[0068] Calculate the cross entropy loss of the text token, expressed as:
[0069]
[0070] Among them, t i is the cross entropy loss of the i-th token in a sample, y ij Indicates whether the i-th token belongs to the j-th category. If yes, it is 1, otherwise it is 0. ij represents the probability that the i-th token belongs to the j-th category, P∈R n×c is the output matrix of bilstm, n is the sequence length, c is the number of labels, P i,j represents the output of bilstm on the jth dimension of the i-th token.
[0071] Use the subnetwork to predict the loss of text data, and get the loss prediction value of each token in the sentence for each sample Select the largest loss value as the screening basis for the sample irank =max(T i ), and then all the predicted samples l irank The values are sorted in descending order, the largest top-k ones are selected, the text of the text loss prediction value is annotated, and added to the training set and validation set. The main network and sub-network are trained repeatedly until the prediction index of the main network meets the preset conditions.
[0072] For the training of the sub-network, the learning rate of the linear layer of the sub-network is set to 1e-3, and the optimizer uses AdamW. At the same time, the parameters of the main network are fixed, that is, only the linear layer is updated, which is expressed as:
[0073]
[0074] Among them, L sub represents the loss of the sub-network, S represents the sentence length, that is, the number of tokens, B represents the size of a training batch, and t i is the cross entropy loss of the i-th token in a sample, is the predicted value of the loss of the i-th token in a sentence by the sub-network.
[0075] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for named entity recognition in Chinese electronic medical records based on active learning, characterized in that: include: Input the Chinese electronic medical record text to be recognized into the main network of the trained named entity recognition model for recognition, and obtain the recognition result; The named entity recognition model includes: a main network and a sub-network; The training process of the named entity recognition model includes: S1: Obtain unlabeled Chinese electronic medical record texts, select parts from them for manual annotation, and divide them into training set and validation set in a ratio of 4:1; S2: Input the text in the training set into the main network of the named entity recognition model, train the main network, obtain the F1 value based on the recognition results of the trained main network on the validation set, save the model parameters with the largest F1, and fix the parameters of the main network; Training the main network includes: Input the text in the training set into the BERT layer of the main network for text segmentation, encode the segmented text through the BILSTM layer to obtain the encoding vector of each word in the text, input the encoding vector of each word in the text into the CRF layer, obtain the predicted labels corresponding to all words in the text, and calculate the predicted label scores of all words in the text. According to the predicted label scores, softmax is used to calculate the probability of the correct label text, and the probability of the correct label text is maximized. According to the maximized probability of the correct label text, the encoding vector of each word in the text is decoded to obtain the optimal text sequence, and the main network with the optimal sequence output is used as the optimal main network obtained by training; Calculate the predicted label score of the words in the text, expressed as: Among them, s(x,y) represents the predicted label score of the word in the text, x represents the text sequence, y represents the predicted label sequence, and n is the sequence length. Indicates that the i-th character in the input sequence is predicted to be the y-th character i label scores, A represents the transfer matrix; According to the predicted label score, softmax is used to calculate the probability of the correct label text, and the probability of the correct label text is maximized, which is expressed as: Among them, P(y|x) represents the probability of the correct label text, s(x,y ～ ) represents the correct label score, represents the sum of all possible label scores, Y x represents the set of all possible label sequences, y ～ represents a label sequence in the set of all possible label sequences; The F1 value is obtained based on the recognition results of the main network on the validation set after training, expressed as: Among them, c represents the total number of entity categories, F 1k represents the F1 value of the kth entity; Calculate the cross entropy loss of the text token, expressed as: Among them, t i is the cross entropy loss of the i-th token in a sample, y ij Indicates whether the i-th token belongs to the j-th category. If yes, it is 1, otherwise it is 0. ij represents the probability that the i-th token belongs to the j-th category, P represents the output matrix of bilstm, P i,j represents the output of bilstm on the jth dimension of the i-th token, and e represents the natural base; S3: Calculate the cross entropy loss of the tokens in the training set; S4: Use the cross entropy loss of the text token as the new label of the training set and the validation set, input the training set with the cross entropy loss label into the sub-network of the named entity recognition model, train the sub-network, and save the parameters of the sub-network model with the smallest loss on the validation set according to the loss of the sub-network on the validation set after training, and fix the parameters of the sub-network; S5: Input the unlabeled text into the sub-network of the named entity recognition model with fixed parameters, obtain the loss prediction value of each token of the text, select the text with the largest text loss prediction value, annotate the text with the text loss prediction value, and add it to the training set and validation set. Repeat the above process until the F1 value of the main network of the named entity recognition model on the validation set reaches the preset conditions of the model, and the model training is completed.
2. According to claim 1, a method for named entity recognition in Chinese electronic medical records based on active learning is characterized in that: The Chinese electronic medical record text to be recognized is input into the trained named entity recognition model for recognition, and the recognition results are obtained, including: Step 1: Divide the Chinese electronic medical record text to be recognized into text segments with a length less than or equal to 256. If the length of the text is less than 256, the sequence is padded with padding. Step 2: Perform token embedding, sentence embedding, and position embedding on the divided text; Step 3: Input the encoded text into the named entity recognition model, obtain the output of the main network of the named entity recognition model, perform Viterbi decoding, and obtain the final recognition result.
3. According to the method of active learning based Chinese electronic medical record named entity recognition in claim 1, it is characterized in that: The main network structure of the named entity recognition model is BERT+BILSTM+CRF, and the sub-network structure is BERT+BILSTM+Linear, where the weights of BERT+LSTM are shared with the main network.
4. According to the method of active learning based Chinese electronic medical record named entity recognition in claim 1, it is characterized in that: The training of the sub-network is expressed as: Among them, L sub represents the loss of the sub-network, S represents the sentence length, B represents the size of a training batch, and t ij represents the cross entropy loss of the i-th token in the j-th sample in a batch, Represents the sub-network's predicted value of the loss of the i-th token in the j-th sample in a batch.
Citation Information
Patent Citations
Image processing method, device, electronic device and storage medium thereof
CN108875693A