Mask enhanced named entity recognition method based on large model

By adopting the mask enhancement method of large models in medical named entity recognition, sharing parameters to perform named entity recognition and masking tasks, the problems of semantics and insufficient context dependence of pre-trained models in medical text are solved, and recognition is improved Accuracy and context capture capabilities reduce misidentification and misidentification.

CN118940763AActive Publication Date: 2024-11-12JIANGNAN UNIV +3

Patent Information

Application Number
CN202411377901.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-30
Publication Date
2024-11-12
Estimated Expiration
2044-09-30

AI Technical Summary

Technical Problem

The existing pre-trained models have semantic deficiencies and insufficient context-dependent capture capabilities in medical named entity recognition, resulting in misidentification and missed recognition problems.

Method used

Using a mask-enhanced named entity recognition method based on large models, the sharing parameters of named entity recognition tasks and predict mask tasks are performed, and the loss function is calculated using the BERT model and the conditional random field model, and gradient update is performed to improve model performance, combined with mask The strategy masks the training input sequences to enhance contextual understanding.

Benefits of technology

It improves the accuracy and context-dependent capture capabilities of medical named entity recognition, reduces misidentification and misidentification, and achieves a more comprehensive semantic understanding.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940763A_ABST
    Figure CN118940763A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of natural language processing, and provides a mask enhanced named entity recognition method based on a large model, and the method comprises the steps: collecting to-be-recognized text data; preprocessing to obtain an input sequence, and inputting the trained recognition model to obtain a recognition result; the recognition model training process comprises the following steps of: performing mask processing on a training input sequence based on a set mask strategy to obtain a mask input sequence, sending the mask input sequence into a BERT model to obtain an entity, and representing a feature by a mask context; executing a named entity identification task and a mask prediction task and sharing parameters to obtain an entity prediction value and a mask prediction value; calculating a first loss function based on the entity context representation feature and the entity predictor, and calculating a second loss function based on the mask predictor; updating model parameters; evaluating the performance of the model, and repeatedly training until the performance meets set requirements. According to the method, semantics can be fully understood, the generalization ability is high, the context dependency capturing ability is high, and the misrecognition and missed recognition conditions are few.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a mask-enhanced named entity recognition method based on a large model. Background Art

[0002] Named entity recognition is one of the key technologies in natural language processing, which aims to automatically identify and classify entities with specific meanings from unstructured text. In the past few decades, the research on named entity recognition technology in general fields has been relatively mature, but in specific fields, especially in the medical field, research still faces many challenges.

[0003] Named entity recognition of medical texts is of great significance in applications such as medical information processing, medical literature retrieval, and medical record analysis. In the medical field, entities are diverse and highly professional, including but not limited to disease names, symptoms, drugs, treatment methods, body parts, etc. Medical texts often contain a large number of professional terms and abbreviations, and the same entity may appear in different forms, which makes the medical named entity recognition task more complex and challenging than the named entity recognition task in general fields.

[0004] With the rapid development of pre-trained models, their application to medical named entity recognition has become an important tool to improve the performance of the task. However, pre-trained models are usually trained on large-scale general corpora, which, although extensive, often lack domain-specific medical knowledge.

[0005] There is a problem of missing semantics when using these pre-trained models, which reduces accuracy and generalization ability. For example, existing pre-trained models may have a good understanding of common vocabulary, but have a relatively limited understanding of medical terms (such as drug names, disease names, symptom descriptions, etc.). This results in the inability of existing pre-trained models to accurately capture the precise semantics of medical terms when faced with medical data, which in turn affects the effect of entity recognition.

[0006] At the same time, the medical named entity recognition task requires the pre-trained model to be able to distinguish fine-grained semantic differences. For example, the same symptom may represent different medical entities in different contexts, or different symptoms may have similar descriptions. Existing pre-trained models have difficulty distinguishing subtle semantic differences in these cases, resulting in misidentification or missed recognition.

[0007] In addition, medical texts are highly context-dependent, and different patient descriptions, doctors’ diagnoses, and treatment recommendations may be expressed in different ways. Existing pre-trained models often fail to capture these context dependencies, which is particularly evident in long-distance dependencies at the sentence or paragraph level. Summary of the invention

[0008] To this end, the technical problem to be solved by the present invention is to overcome the problems of semantic loss, insufficient context dependency capture capability, misidentification and missed recognition in the medical named entity recognition method in the prior art when using a pre-trained model, and provide a mask enhanced named entity recognition method based on a large model, which can fully understand the semantics, has a strong context dependency capture capability, and has fewer misidentification and missed recognition.

[0009] In order to solve the above technical problems, the present invention provides a mask-enhanced named entity recognition method based on a large model, comprising:

[0010] Step S1: Collect text data to be recognized;

[0011] Step S2: preprocessing the text data to be recognized to obtain an input sequence;

[0012] Step S3: inputting the input sequence into the trained recognition model to obtain a recognition result;

[0013] The training process of the recognition model includes:

[0014] Collecting training text data; preprocessing the training text data to obtain a training input sequence; performing mask processing on the training input sequence based on a set mask strategy to obtain a masked input sequence;

[0015] Sending the training input sequence and the mask input sequence into the BERT model to obtain entity context representation features and mask context representation features;

[0016] Performing a named entity recognition task and a mask prediction task and sharing parameters; in the named entity recognition task, using a NER-Classifier to identify and classify the entity context representation features to obtain an entity prediction value; in the mask prediction task, using an MLM-Classifier to perform token prediction on the mask context representation features to obtain a mask prediction value;

[0017] Calculating a first loss function based on the entity context representation feature and the entity prediction value, and calculating a second loss function based on the mask prediction value; performing gradient calculation based on the first loss function and the second loss function in each training cycle to update the parameters of the recognition model;

[0018] At the end of each training cycle, a performance evaluation is performed based on the entity prediction value and the mask prediction value; and the training is repeated until the performance of the recognition model meets the set requirements.

[0019] In one embodiment of the present invention, the method for performing masking processing on the training input sequence based on a set masking strategy to obtain a masked input sequence includes: constructing an entity masking model, performing masking processing on some tokens in the training input sequence based on the set masking strategy, and constructing a masking matrix and a masked input sequence:

[0020] First, the entities in the training input sequence x for:

[0021] ;

[0022] Then, for The tokens spanned by each entity in adopt the mask strategy:

[0023] ;

[0024] Next, Apply the masking strategy to all entities in to obtain the masked input sequence. Correspondingly, the mask matrix value corresponding to the spanning position contained in each entity should be set to 1, indicating that the part has been masked:

[0025] ;

[0026] Wherein, b represents the starting position of entity u, and q is the span length of the entity u; Represents the masked entity u, random token is a randomly selected token in the vocabulary, [MASK] represents a mask token, and p represents the selection probability of the mask strategy; M mask,i Represents the mask matrix value at sequence position i.

[0027] In one embodiment of the present invention, a method for calculating a first loss function based on the entity context representation feature and the entity prediction value includes:

[0028] Using the conditional random field model, the first loss function Perform the calculation:

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] Among them, emissions(t,y) represents the score of selecting label y at sequence position t, W y is the weight of NER-Classifier when using the conditional random field model, h t is the BERT output representation at position t in the sequence, b y is the fully connected layer bias of NER-Classifier when using the conditional random field model; T yt-1,yt Indicates that from y t-1 Transfer to y t The score, y t-1 is the label of sequence position t-1, y t is the label of the sequence position t, trans() represents the transfer matrix; P(y│x) represents the conditional probability, Z(x) represents the normalization factor, exp() represents the exponential function, Σ represents the summation function, n is the sequence length; log() represents taking the logarithm.

[0035] In one embodiment of the present invention, based on the first loss function The method of performing gradient calculation with the second loss function to update the parameters of the recognition model is:

[0036] The first step is to freeze the parameters of the MLM-Classifier based on the first loss function Perform gradient calculation to update the parameters of the BERT model, the parameters of the NER-Classifier, and the parameters of the conditional random field model. In the second step, freeze the weights of the NER-Classifier and the weights of the conditional random field model, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the parameters of the MLM-Classifier.

[0037] In one embodiment of the present invention, a method for calculating a first loss function based on the entity context representation feature and the entity prediction value includes:

[0038] Using the Softmax layer, the first loss function Perform the calculation:

[0039] ;

[0040] ;

[0041] ;

[0042] Among them, z t represents the label score at sequence position t, W sis the weight of NER-Classifier when using the Softmax layer, h t is the BERT output representation at position t in the sequence, b s is the bias of NER-Classifier when using the Softmax layer; p(y t =c│h t ) is the probability that the token at position t in the sequence corresponds to label c, z t,c is the logit value corresponding to the label c, exp() represents the exponential function, Σ represents the sum function, n is the sequence length; L is the number of labels; is an indicator function, and when the true label of the sequence position t is c, the indicator function value is 1, otherwise it is 0.

[0043] In one embodiment of the present invention, based on the first loss function The method of performing gradient calculation with the second loss function to update the parameters of the recognition model is:

[0044] The first step is to freeze the parameters of the MLM-Classifier based on the first loss function Perform gradient calculation to update the parameters of the BERT model and the parameters of the NER-Classifier; secondly, freeze the weight of the NER-Classifier, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the parameters of the MLM-Classifier.

[0045] In one embodiment of the present invention, a second loss function is calculated based on the mask prediction value. The method is:

[0046] ;

[0047] ;

[0048] ;

[0049] Among them, z i m represents the mask prediction value, W m is the weight of the MLM-Classifier, b m is the bias of the MLM-Classifier, h i m representing features for the mask context; Represents the mask prediction value z im The distribution probability of is the predicted label, exp() represents the exponential function, Σ represents the sum function, V is the vocabulary size, z i,j m is the prediction score of the masked token at sequence position i; |M mask | is the mask matrix M mask The number of non-zero elements in , n is the length of the sequence; is an indicator function. When the true label of sequence position m is j, the value of the indicator function is 1, otherwise it is 0; M mask,i is the mask value of the sequence position i. If the sequence position i is masked, M mask,i The value of is 1, otherwise it is 0; log() means taking the logarithm.

[0050] In one embodiment of the present invention, the masking strategy is: 80% probability of using mask [MASK] to replace the input of the target position, 10% probability of using a random token to replace the token of the target position, and 10% probability of not making any changes.

[0051] In one embodiment of the present invention, when the entity context representation features are identified and classified by NER-Classifier to obtain the entity prediction value, a label sequence serving as the entity prediction value is found based on the Viterbi algorithm.

[0052] In one embodiment of the present invention, at the end of each training cycle, performance evaluation is performed based on the entity prediction value and the mask prediction value, and evaluation indicators of the performance evaluation include weighted precision, weighted recall and weighted score.

[0053] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0054] The present invention provides a mask-enhanced named entity recognition method based on a large model. When training the recognition model, the named entity recognition task and the mask prediction task are performed and parameters are shared, and the parameters of the recognition model are updated based on a first loss function and a second loss function. Medical named entity recognition is performed based on the trained recognition model, which can fully understand the semantics, has a strong ability to capture contextual dependencies, and has fewer misrecognitions and missed recognitions. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0056] Figure 1 A flowchart of a method for mask-enhanced named entity recognition based on a large model in an embodiment of the present invention; Figure 2 Schematic diagram of the experimental environment in an embodiment of the present invention; Figure 3 A schematic diagram of experimental parameters set in an embodiment of the present invention; Figure 4 BERT-CRF model diagram using the entity mask model in an embodiment of the present invention; Figure 5 This is a diagram of a BERT-CRF model using a random mask model in an embodiment of the present invention; Figure 6 BERT-Softmax model diagram using the entity mask model in an embodiment of the present invention; Figure 7 This is a diagram of a BERT-Softmax model using a random mask model in an embodiment of the present invention; Figure 8 Schematic diagram of experimental results in an embodiment of the present invention; Fig. 9 A comparison chart of the weighted scores F1 of the six models in the embodiment of the present invention on the CMeEE experimental data set; Fig.10 This is a comparison chart of the weighted scores F1 of the six models in the embodiments of the present invention on the CHIP2020 experimental dataset; Fig.11 Schematic diagram of the final prediction result of a sentence example in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0059] It should be noted that the descriptions of "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0060] With the rapid development of pre-trained models, applying them to medical named entity recognition has become an important tool to improve task performance. However, existing pre-trained models have semantic missing problems when performing medical named entity recognition tasks, and their accuracy and generalization ability are poor. At the same time, the medical named entity recognition task requires the pre-trained model to be able to distinguish fine-grained semantic differences and have strong contextual dependence, while existing pre-trained models often fail to capture these contextual dependencies, which is particularly evident in long-distance dependencies at the sentence level or paragraph level.

[0061] To this end, an embodiment of the present application provides a mask-enhanced named entity recognition method based on a large model.

[0062] This embodiment provides a mask-enhanced named entity recognition method based on a large model. Figure 1 As shown, including:

[0063] Step S1: Collect text data to be recognized;

[0064] Step S2: preprocessing the text data to be recognized to obtain an input sequence;

[0065] Step S3: inputting the input sequence into the trained recognition model to obtain a recognition result;

[0066] The training process of the recognition model includes:

[0067] Collecting training text data; preprocessing the training text data to obtain a training input sequence; performing mask processing on the training input sequence based on a set mask strategy to obtain a masked input sequence;

[0068] Sending the training input sequence and the mask input sequence into the BERT model to obtain entity context representation features and mask context representation features;

[0069] Performing a named entity recognition task and a mask prediction task and sharing parameters; in the named entity recognition task, using a NER-Classifier to identify and classify the entity context representation features to obtain an entity prediction value; in the mask prediction task, using an MLM-Classifier to perform token prediction on the mask context representation features to obtain a mask prediction value;

[0070] Calculating a first loss function based on the entity context representation feature and the entity prediction value, and calculating a second loss function based on the mask prediction value; performing gradient calculation based on the first loss function and the second loss function in each training cycle to update the parameters of the recognition model;

[0071] At the end of each training cycle, a performance evaluation is performed based on the entity prediction value and the mask prediction value; and the training is repeated until the performance of the recognition model meets the set requirements.

[0072] The present embodiment provides a large-model-based mask-enhanced named entity recognition method, which executes named entity recognition tasks and mask prediction tasks and shares parameters when training the recognition model, and performs gradient calculation based on the first loss function and the second loss function to update the parameters of the recognition model, thereby improving the performance of the recognition model in medical named entity recognition, being able to fully understand semantics, having strong generalization ability, strong context-dependency capture ability, high recognition accuracy, and fewer misrecognitions and missed recognitions.

[0073] Next, a mask-enhanced named entity recognition method based on a large model provided in this embodiment is introduced in detail:

[0074] 1. Training process of recognition model:

[0075] Step 01: Collect training text data; pre-process the training text data to obtain a training input sequence.

[0076] Optionally, numbers, brackets and spaces in the training text data are removed, entity information is parsed into a start index, an end index and an entity type and stored in a dictionary form; word segmentation is performed, the stored training text data is converted into a format required by the input model, and padded or trimmed according to a specified maximum length.

[0077] Optionally, initialize the named entity recognition (NER) label to all "O", set the corresponding label to the start label B-dis and the middle label I-dis of the entity type according to the entity information, and add the entity type; convert the NER label into an ID, and fill or trim it according to the specified maximum length.

[0078] Optionally, the training text data is converted into a training input sequence by a word segmenter.

[0079] Step 02: Masking the training input sequence based on the set masking strategy to obtain a masked input sequence.

[0080] Optionally, the mask strategy is:

[0081] (1) With a probability of 80%, the mask [MASK] is used to replace the input of the target position;

[0082] (2) There is a 10% probability of using a random token to replace the token at the target position;

[0083] (3) There is a 10% probability that no change will be made.

[0084] Specifically, the replacement in (1) requires the recognition model to infer the masked real words through context information during training. The use of the mask [MASK] can force the recognition model to learn to make predictions in the absence of information. The replacement in (2) increases the diversity of training data and can simulate the noise and changes that may occur in actual applications. By replacing the words at the target position with random words, the recognition model will learn to process noisy data during training, which is conducive to understanding semantic information. Part of the original data is retained in (3), which can ensure that the recognition model can learn how to deal with semantic omissions and noise changes while maintaining memory of real words during training.

[0085] Optionally, a random mask model is constructed, and based on the mask strategy, some tokens in the training input sequence are masked to construct a mask matrix and a masked input sequence:

[0086] First, initialize the mask probability matrix P mask =0.15×1 mask ; Among them, 0.15 is the overall probability of mask, 1 mask is an all-1 matrix, the length of which is the same as the length of the training input sequence.

[0087] Then, the mask matrix is ​​generated using the Bernoulli distribution :

[0088] ;

[0089] Among them, M mask,i represents the mask value of sequence position i, and p represents the selection probability of the mask strategy.

[0090] Next, for the training input sequence x i Each token in adopts the masking strategy to obtain the masked input sequence x i ':

[0091] ;

[0092] Among them, random token is a randomly selected token from the vocabulary.

[0093] Preferably, an entity mask model is constructed, and based on the mask strategy, some tokens in the training input sequence are masked to construct a mask matrix and a masked input sequence:

[0094] First, the entities in the training input sequence x for:

[0095] ;

[0096] Then, for The tokens spanned by each entity in adopt the mask strategy:

[0097] ;

[0098] Next, Apply the masking strategy to all entities in to obtain the masked input sequence. Correspondingly, the mask matrix value corresponding to the spanning position contained in each entity should be set to 1, indicating that the part has been masked:

[0099] ;

[0100] Wherein, b represents the starting position of entity u, and q is the span length of the entity u; Represents the masked entity u, random token is a randomly selected token in the vocabulary, [MASK] represents a mask token, and p represents the selection probability of the mask strategy; M mask,i Represents the mask matrix value at sequence position i.

[0101] Step 03: Send the training input sequence and the mask input sequence into the BERT model to obtain entity context representation features and mask context representation features.

[0102] Optionally, the training input sequence is fed into the BERT model to generate a context representation feature for each token, that is, the entity context representation feature H, H=[h1,h2,…,h n ], the dimension of h is D; the mask context representation feature H is obtained m The process is the same as the process of obtaining the entity context representation feature H.

[0103] Step 04: Execute a named entity recognition task and a mask prediction task and share parameters; in the named entity recognition task, use NER-Classifier to identify and classify the entity context representation features to obtain an entity prediction value; in the mask prediction task, use MLM-Classifier to perform token prediction on the mask context representation features to obtain a mask prediction value.

[0104] Step 04-1 Named Entity Recognition Task:

[0105] Optionally, a Conditional Random Field (CRF) model is used to calculate the conditional probability P(y│x):

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] in, represents the score of selecting label y at sequence position t, W y is the weight of NER-Classifier when using the CRF model, h t is the BERT output representation at position t in the sequence, b y is the fully connected layer bias of NER-Classifier when using the CRF model; T yt-1,yt Indicates that from y t-1 Transfer to y t The score, y t-1 is the label of sequence position t-1, y t is the label at position t in the sequence, trans() represents the transfer matrix; Z(x) represents the normalization factor, which ensures that the sum of the probabilities of possible label sequences is 1; exp() represents the exponential function, Σ represents the sum function, and n is the sequence length; log() represents taking the logarithm;

[0111] The Viterbi algorithm is used to find the most likely tag sequence as the entity prediction value :

[0112] ;

[0113] Optionally, apply a Softmax layer to generate the probability of each label, the probability that the token at sequence position t corresponds to label c for:

[0114] ;

[0115] ;

[0116] Among them, z t represents the label score at sequence position t, W s is the weight of NER-Classifier when using the Softmax layer, b sis the bias of NER-Classifier when using the Softmax layer; t,c is the logit value corresponding to the label c;

[0117] The Viterbi algorithm is used to find the most likely tag sequence as the entity prediction value :

[0118] ;

[0119] Step 04-2 Predict mask task:

[0120] Optionally, the mask prediction value z i m for:

[0121] ;

[0122] Among them, W m is the weight of the MLM-Classifier, b m is the bias of the MLM-Classifier, h i m A context representation feature is provided for the mask.

[0123] Step 05: Calculate a first loss function based on the entity context representation feature and the entity prediction value, and calculate a second loss function based on the mask prediction value; perform gradient calculation based on the first loss function and the second loss function in each training cycle to update the parameters of the recognition model.

[0124] Step 05-1 First loss function:

[0125] Optionally, when the CRF model is used, the first loss function for:

[0126] ;

[0127] Optionally, when the Softmax layer is applied, the first loss function is:

[0128] ;

[0129] Where L is the number of labels, is an indicator function, and when the true label of the sequence position t is c, the indicator function value is 1, otherwise it is 0.

[0130] Step 05-2 Second loss function:

[0131] Optionally, the second loss function for:

[0132] ;

[0133] ;

[0134] in, Represents the mask prediction value z i m The distribution probability of is the predicted label, V is the vocabulary size, z i,j m is the prediction score of the masked token at sequence position i; |M mask | is the mask matrix M mask The number of non-zero elements in , is an indicator function, which has a value of 1 when the true label of sequence position m is j, and 0 otherwise.

[0135] Step 05-3 Optimize parameters:

[0136] Optionally, when the CRF model is adopted:

[0137] The first step is to freeze the parameters of the MLM-Classifier based on the first loss function Perform gradient calculation to update the parameters of the BERT model, the parameters of the NER-Classifier, and the parameters of the conditional random field model. In the second step, freeze the weights of the NER-Classifier and the weights of the conditional random field model, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the parameters of the MLM-Classifier.

[0138] Optionally, when applying the Softmax layer:

[0139] The first step is to freeze the parameters of the MLM-Classifier based on the first loss function Perform gradient calculation to update the parameters of the BERT model and the parameters of the NER-Classifier; secondly, freeze the weight of the NER-Classifier, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the parameters of the MLM-Classifier.

[0140] Step 06: At the end of each training cycle, a performance evaluation is performed based on the entity prediction value and the mask prediction value; and the training is repeated until the performance of the recognition model meets the set requirements.

[0141] Optionally, the evaluation indicators of the performance evaluation include weighted precision, weighted recall and weighted score.

[0142] Specifically, weighted precision measures the accuracy of the model when predicting the positive class, that is, how high the proportion of actual positive classes is among all samples predicted to be positive; the higher the weighted precision, the smaller the false alarm rate of the model.

[0143] Specifically, the weighted recall rate measures the model's ability to detect positive samples, that is, how high the proportion of samples correctly identified by the model is among all actual positive samples; the higher the weighted recall rate, the higher the rate at which the model identifies positive samples.

[0144] Specifically, the weighted score is the harmonic mean of the weighted precision and the weighted recall, and is a comprehensive evaluation indicator of the two; the higher the weighted score, the better the overall performance of the model.

[0145] 2. Experimental verification of beneficial effects:

[0146] 1. Experimental data set:

[0147] For example, experiments are conducted on two experimental datasets: CMeEE and CHIP2020:

[0148] The CMeEE dataset contains 504 common pediatric diseases, 7085 body parts, 12907 clinical manifestations and 4354 medical procedures, which can be divided into 15,000 training sets, 5000 and 3000 test sets; CHIP2020 contains 47194 sentences and 938 files, with an average of 2355 words per file, which can be divided into 15,000 training sets and 5000 test sets; both benchmark datasets contain nine categories of medical entity labels: diseases, clinical manifestations, medical procedures, medical equipment, drugs, medical test items, body, departments, and microorganisms.

[0149] 2. Experimental environment:

[0150] For example, the experimental environment is as follows Figure 2 shown.

[0151] 3. Experimental parameters:

[0152] For example, the experimental parameters are as follows: Figure 3 Settings shown.

[0153] Among them, *MBERT-CRF includes EntMBERT-CRF model and RanDMBERT-CRF model, *MBERT-Softmax includes EntMBERT-Softmax model and RanDBERT-Softmax model; the following six models are based on the BERT model:

[0154] The BERT-CRF model only uses the CRF model; the BERT-Softmx model only uses the Softmax layer;

[0155] The EntMBERT-CRF model uses the entity mask model and the CRF model, please refer to Figure 4 As shown; the RanDMBERT-CRF model uses the random mask model and the CRF model, please refer to Figure 5 As shown; the EntMBERT-Softmax model uses the entity mask model and the Softmax layer, please refer to Figure 6 As shown; the RanDBERT-Softmax model uses the random mask model and the Softmax layer, please refer to Figure 7 As shown; among them, E [CLS] Indicates the special placeholder [CLS], E1, E2, E N Indicates that they correspond to token1, token2, token N Characterization of E [SEP] A token that represents a special placeholder [SEP].

[0156] 4. Experimental results:

[0157] For example, the experimental results are as follows Figure 8 As shown in Figure 2, the EntMBERT-CRF model and the EntMBERT-Softmax model showed significant performance improvements on both experimental datasets; please refer to Fig. 9 and Fig.10 As shown in the figure, the EntMBERT-CRF model and the EntMBERT-Softmax model both show higher weighted scores and faster convergence speeds on both experimental datasets.

[0158] Wherein, Precision represents the weighted precision, Recall represents the weighted recall, and F1 represents the weighted score.

[0159] It can be seen that when performing named entity recognition tasks and predicting mask tasks and sharing parameters, building an entity mask model and masking the entities can focus on the entities and their contexts, thereby more effectively learning entity-related features and achieving the goal of medical named entity recognition.

[0160] 5. Example analysis:

[0161] For example, the sentence "The patient visited the doctor due to acute headache and hypertension. Examination revealed stenosis of the left carotid artery. The doctor recommended the use of metoprolol and pril, and performed ultrasound examination and blood pressure monitoring." is used as an example.

[0162] For example, the entities and categories expected to be identified are as follows: headache, hypertension, carotid stenosis, metoprolol, pril, ultrasound examination, blood pressure test; please refer to the final prediction results Fig.11 As shown in the figure, the EntMBERT-CRF model can recognize all entities. Although the EntMBERT-Softmax model cannot recognize all entities, the recognition effect of the EntMBERT-Softmax model is still better than that of the BERT-Softmax model and the RanDMBERT-Softmax model which also use the Softmax layer.

[0163] It can be seen that when performing named entity recognition tasks and predicting mask tasks and sharing parameters, building an entity mask model and masking the entities can focus on the entities and their contexts, thereby more effectively learning entity-related features and achieving the goal of medical named entity recognition.

[0164] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0165] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0166] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0168] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A mask-enhanced named entity recognition method based on a large model, characterized in that: include: Step S1: Collect text data to be recognized; Step S2: preprocessing the text data to be recognized to obtain an input sequence; Step S3: inputting the input sequence into the trained recognition model to obtain a recognition result; The training process of the recognition model includes: Collecting training text data; preprocessing the training text data to obtain a training input sequence; performing mask processing on the training input sequence based on a set mask strategy to obtain a masked input sequence; Sending the training input sequence and the mask input sequence into the BERT model to obtain entity context representation features and mask context representation features; Performing a named entity recognition task and a mask prediction task and sharing parameters; in the named entity recognition task, using a NER-Classifier to identify and classify the entity context representation features to obtain an entity prediction value; in the mask prediction task, using an MLM-Classifier to perform token prediction on the mask context representation features to obtain a mask prediction value; Calculating a first loss function based on the entity context representation feature and the entity prediction value, and calculating a second loss function based on the mask prediction value; performing gradient calculation based on the first loss function and the second loss function in each training cycle to update the parameters of the recognition model; At the end of each training cycle, a performance evaluation is performed based on the entity prediction value and the mask prediction value; and the training is repeated until the performance of the recognition model meets the set requirements.

2. According to the large model-based mask-enhanced named entity recognition method of claim 1, it is characterized in that: The method of performing mask processing on the training input sequence based on the set mask strategy to obtain the masked input sequence includes: Construct an entity mask model, mask some tokens in the training input sequence based on the set mask strategy, and construct a mask matrix and a masked input sequence: First, the entities in the training input sequence x for: ; Then, for The tokens spanned by each entity in adopt the mask strategy: ; Next, Apply the masking strategy to all entities in to obtain the masked input sequence. Correspondingly, the mask matrix value corresponding to the spanning position contained in each entity should be set to 1, indicating that the part has been masked: ; Wherein, b represents the starting position of entity u, and q is the span length of the entity u; Represents the masked entity u, random token is a randomly selected token in the vocabulary, [MASK] represents a mask token, and p represents the selection probability of the mask strategy; M mask,i Represents the mask matrix value at sequence position i.

3. According to the large model-based mask-enhanced named entity recognition method of claim 1, it is characterized in that: The method for calculating the first loss function based on the entity context representation feature and the entity prediction value includes: Using the conditional random field model, the first loss function Perform the calculation: ; ; ; ; ; Among them, emissions(t,y) represents the score of selecting label y at sequence position t, W y is the weight of NER-Classifier when using the conditional random field model, h t is the BERT output representation at position t in the sequence, b y is the fully connected layer bias of NER-Classifier when using the conditional random field model; T yt-1,yt Indicates that from y t-1 Transfer to y t The score, y t-1 is the label of sequence position t-1, y t is the label of the sequence position t, trans() represents the transfer matrix; P(y│x) represents the conditional probability, Z(x) represents the normalization factor, exp() represents the exponential function, Σ represents the summation function, n is the sequence length; log() represents taking the logarithm.

4. According to the large model-based mask-enhanced named entity recognition method of claim 3, it is characterized in that: Based on the first loss function The method of performing gradient calculation with the second loss function to update the parameters of the recognition model is: The first step is to freeze the parameters of the MLM-Classifier based on the first loss function Perform gradient calculation to update the parameters of the BERT model, the parameters of the NER-Classifier, and the parameters of the conditional random field model. In the second step, freeze the weights of the NER-Classifier and the weights of the conditional random field model, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the parameters of the MLM-Classifier.

5. The method for mask-enhanced named entity recognition based on a large model according to claim 1, characterized in that: The method for calculating the first loss function based on the entity context representation feature and the entity prediction value includes: Using the Softmax layer, the first loss function Perform the calculation: ; ; ; Among them, z t represents the label score at sequence position t, W s is the weight of NER-Classifier when using the Softmax layer, h t is the BERT output representation at position t in the sequence, b s is the bias of NER-Classifier when the Softmax layer is used; p(y t =c│h t ) is the probability that the token at position t in the sequence corresponds to label c, z t,c is the logit value corresponding to the label c, exp() represents the exponential function, Σ represents the sum function, n is the sequence length; L is the number of labels; is an indicator function, and when the true label of the sequence position t is c, the indicator function value is 1, otherwise it is 0.

6. The method for mask-enhanced named entity recognition based on a large model according to claim 5, characterized in that: Based on the first loss function The method of performing gradient calculation with the second loss function to update the parameters of the recognition model is: The first step is to freeze the parameters of the MLM-Classifier based on the first loss function Perform gradient calculation to update the parameters of the BERT model and the parameters of the NER-Classifier; secondly, freeze the weight of the NER-Classifier, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the parameters of the MLM-Classifier.

7. The method for mask-enhanced named entity recognition based on a large model according to claim 1, characterized in that: Calculate a second loss function based on the mask prediction value The method is: ; ; ; Among them, z i m represents the mask prediction value, W m is the weight of the MLM-Classifier, b m is the bias of the MLM-Classifier, h i m representing features for the mask context; Represents the mask prediction value z i m The distribution probability of is the predicted label, exp() represents the exponential function, Σ represents the sum function, V is the vocabulary size, z i,j m is the prediction score of the masked token at sequence position i; |M mask | is the mask matrix M mask The number of non-zero elements in , n is the length of the sequence; is an indicator function. When the true label of sequence position m is j, the value of the indicator function is 1, otherwise it is 0; M mask,i is the mask value of the sequence position i. If the sequence position i is masked, M mask,i The value of is 1, otherwise it is 0; log() means taking the logarithm.

8. The method for mask-enhanced named entity recognition based on a large model according to claim 1, characterized in that: The masking strategy is: 80% probability of using mask [MASK] to replace the input of the target position, 10% probability of using a random token to replace the token of the target position, and 10% probability of not making any changes.

9. The method for mask-enhanced named entity recognition based on a large model according to claim 1, characterized in that: When the entity context representation features are identified and classified by NER-Classifier to obtain the entity prediction value, a label sequence serving as the entity prediction value is found based on the Viterbi algorithm.

10. The method for mask-enhanced named entity recognition based on a large model according to claim 1, characterized in that: At the end of each training cycle, a performance evaluation is performed based on the entity prediction value and the mask prediction value, and the evaluation indicators of the performance evaluation include weighted precision, weighted recall and weighted score.

Citation Information

Patent Citations

  • Medical consultation named entity identification method based on adversarial multi-task learning

    CN111222339A

  • Named entity recognition model of dense entity text and training method thereof

    CN117236338A

  • Small sample named entity recognition method based on metric learning

    CN118536509A

  • Method and apparatus of NER-oriented chinese clinical text data augmentation

    US20240013000A1

Cited By

  • Named entity recognition method for man-machine conversation system

    CN119808786A

  • Text entity recognition model construction method and equipment based on large model data enhancement

    CN120995985A