Masked large model-based enhanced named entity recognition method

Abstract

The application relates to the technical field of natural language processing, and provides a mask enhancement named entity recognition method based on a large model, which comprises the following steps: collecting to-be-recognized text data; preprocessing to obtain an input sequence, inputting the trained recognition model to obtain a recognition result; the recognition model training process comprises the following steps: performing mask processing on the training input sequence based on a set mask strategy to obtain a mask input sequence, inputting the mask input sequence into a BERT model to obtain entity and mask context representation features; performing a named entity recognition task and a mask prediction task and sharing parameters to obtain entity prediction values and mask prediction values; calculating a first loss function based on the entity context representation features and the entity prediction values, and calculating a second loss function based on the mask prediction values; updating model parameters; evaluating model performance, and repeating training until the performance reaches a set requirement. The application can fully understand semantics, has strong generalization capability, strong context dependency capturing capability, and less misrecognition and missed recognition.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a mask-enhanced named entity recognition method based on a large model. Background Technology

[0002] Named entity recognition (NER) is a key technology in natural language processing, aiming to automatically identify and classify entities with specific meanings from unstructured text. While NER technology has matured relatively well in general fields over the past few decades, research in specific domains, particularly in medicine, still faces numerous challenges.

[0003] Named entity recognition (NENT) in medical texts is of great significance in applications such as medical information processing, medical literature retrieval, and medical record analysis. In the medical field, entities are diverse and highly specialized, including but not limited to disease names, symptoms, drugs, treatment methods, and body parts. Medical texts often contain a large number of technical terms and abbreviations, and the same entity may appear in different forms, making medical NENT recognition tasks more complex and challenging than general domain NENT recognition tasks.

[0004] With the rapid development of pre-trained models, their application in medical named entity recognition has become an important tool for improving task performance. However, pre-trained models are usually trained on large-scale general corpora, which, while extensive, often lack domain-specific medical knowledge.

[0005] Using these pre-trained models presents a semantic gap problem, reducing accuracy and generalization ability. For example, existing pre-trained models may have a good understanding of common vocabulary, but their understanding of medical terms (such as drug names, disease names, symptom descriptions, etc.) is relatively limited. This means that when faced with medical data, existing pre-trained models cannot accurately capture the precise semantics of medical terms, thus affecting the performance of entity recognition.

[0006] Meanwhile, the medical named entity recognition task requires pre-trained models to be able to distinguish fine-grained semantic differences. For example, the same symptom may represent different medical entities in different contexts, or different symptoms may have similar descriptions. Existing pre-trained models struggle to distinguish subtle semantic differences in these situations, leading to misidentification or missed identification.

[0007] In addition, medical texts are highly context-dependent; different patient descriptions, doctors' diagnoses, and treatment recommendations may be expressed differently. Existing pre-trained models often underperform in capturing these contextual dependencies, which is particularly evident in long-distance dependencies at the sentence or paragraph level. Summary of the Invention

[0008] Therefore, the technical problem to be solved by the present invention is to overcome the problems of semantic loss, insufficient ability to capture context dependence, misidentification and missed identification in the existing medical named entity recognition methods when using pre-trained models. The present invention provides a mask-enhanced named entity recognition method based on a large model, which can fully understand semantics, has a strong ability to capture context dependence, and has fewer misidentification and missed identification cases.

[0009] To address the aforementioned technical problems, this invention provides a mask-enhanced named entity recognition method based on a large model, comprising:

[0010] Step S1: Collect the text data to be recognized;

[0011] Step S2: Preprocess the text data to be recognized to obtain the input sequence;

[0012] Step S3: Input the input sequence into the trained recognition model to obtain the recognition result;

[0013] The training process of the recognition model includes:

[0014] Collect training text data; preprocess the training text data to obtain a training input sequence; perform masking processing on the training input sequence based on a set masking strategy to obtain a masked input sequence;

[0015] The training input sequence and the masked input sequence are fed into the BERT model to obtain entity context representation features and masked context representation features.

[0016] The system performs named entity recognition and prediction mask tasks with shared parameters. In the named entity recognition task, NER-Classifier is used to identify and classify the entity context representation features to obtain entity prediction values. In the prediction mask task, MLM-Classifier is used to perform token prediction on the mask context representation features to obtain mask prediction values.

[0017] A first loss function is calculated based on the entity context representation features and the entity prediction value, and a second loss function is calculated based on the mask prediction value; gradient calculation is performed based on the first loss function and the second loss function in each training cycle to update the parameters of the recognition model;

[0018] At the end of each training cycle, a performance evaluation is performed based on the entity prediction value and the mask prediction value; training is repeated until the performance of the recognition model meets the set requirements.

[0019] In one embodiment of the present invention, a method for masking the training input sequence based on a set masking strategy to obtain a masked input sequence includes: constructing an entity masking model, masking a portion of the tokens in the training input sequence based on the set masking strategy, and constructing a masking matrix and a masked input sequence.

[0020] First, the entities in the training input sequence x for:

[0021] ;

[0022] Then, for The masking strategy is used for the token spanned by each entity in the data:

[0023] ;

[0024] Next, regarding All entities in the sequence are subjected to the masking strategy described above to obtain the masked input sequence. Accordingly, the mask matrix value corresponding to the spanning position contained in each entity should be set to 1, indicating that the part has been masked.

[0025] ;

[0026] Where b represents the starting position of entity u, and q represents the span length of entity u; Represents the entity u after masking, random token It is a token randomly selected from the vocabulary, [MASK] represents the mask token, and p represents the selection probability of the masking strategy; M mask,i The mask matrix value represents the position i in the sequence.

[0027] In one embodiment of the present invention, a method for calculating a first loss function based on the entity context representation features and the entity prediction value includes:

[0028] Using a conditional random field model, the first loss function is... Perform the calculation:

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] Where emissions(t,y) represents the score for selecting label y at sequence position t, W y The weights h of the NER-Classifier when using the conditional random field model are described above. t b is the BERT output representation at position t of the sequence. y This refers to the bias of the fully connected layer of the NER-Classifier when using the conditional random field model; T yt-1，yt Indicates from y t-1 Transfer to y t The score, y t-1 It is the label at sequence position t-1, y t Here, t is the label of the sequence position, trans() represents the transition matrix, P(y│x) represents the conditional probability, Z(x) represents the normalization factor, exp() represents the exponential function, Σ represents the summation function, n is the sequence length, and log() represents taking the logarithm.

[0035] In one embodiment of the present invention, based on the first loss function The method for calculating the gradient with the second loss function to update the parameters of the recognition model is as follows:

[0036] The first step is to freeze the parameters of the MLM-Classifier based on the first loss function. First, perform gradient calculation to update the parameters of the BERT model, the NER-Classifier, and the Conditional Random Field model. Second, freeze the weights of the NER-Classifier and the Conditional Random Field model, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the MLM-Classifier.

[0037] In one embodiment of the present invention, a method for calculating a first loss function based on the entity context representation features and the entity prediction value includes:

[0038] Using a Softmax layer, the first loss function is... Perform the calculation:

[0039] ;

[0040] ;

[0041] ;

[0042] Among them, z t W represents the label score at sequence position t. sThe weights of the NER-Classifier when using the Softmax layer, h t b is the BERT output representation at position t of the sequence. s This refers to the bias of the NER-Classifier when using the Softmax layer; p(y t =c│h t ) is the probability that the token at position t in the sequence corresponds to the tag c, z t,c This corresponds to the logit value of the label c, where exp() represents the exponential function, Σ represents the summation function, n is the sequence length, and L is the number of labels. It is an indicator function. When the actual label of the sequence position t is c, the indicator function has a value of 1, otherwise it is 0.

[0043] In one embodiment of the present invention, based on the first loss function The method for calculating the gradient with the second loss function to update the parameters of the recognition model is as follows:

[0044] The first step is to freeze the parameters of the MLM-Classifier based on the first loss function. First, perform gradient calculation to update the parameters of the BERT model and the NER-Classifier. Second, freeze the weights of the NER-Classifier, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the MLM-Classifier.

[0045] In one embodiment of the present invention, a second loss function is calculated based on the mask prediction value. The method is as follows:

[0046] ;

[0047] ;

[0048] ;

[0049] Among them, z i m W represents the predicted mask value. m b is the weight of the MLM-Classifier. m h represents the deviation of the MLM-Classifier. i m The mask context represents the features; This represents the mask prediction value z. im The probability distribution, This is for predicting labels, where exp() represents the exponential function, Σ represents the summation function, V is the vocabulary size, and z is the z-value. i,j m It is the predicted score of the masked token at sequence position i; |M mask | is the mask matrix M mask The number of non-zero elements in the sequence, where n is the sequence length; It is an indicator function; its value is 1 when the actual label of sequence position m is j, and 0 otherwise; M mask,i It is the mask value of the sequence position i. If the sequence position i is masked, M mask,i The value is 1 if it is not 0 otherwise; log() means taking the logarithm.

[0050] In one embodiment of the present invention, the masking strategy is as follows: there is an 80% probability of replacing the input at the target location with a mask [MASK], a 10% probability of replacing the token at the target location with a random token, and a 10% probability of not making any changes.

[0051] In one embodiment of the present invention, when using NER-Classifier to identify and classify the entity context representation features to obtain entity prediction values, a label sequence as the entity prediction value is found based on the Viterbi algorithm.

[0052] In one embodiment of the present invention, a performance evaluation is performed at the end of each training cycle based on the entity prediction value and the mask prediction value. The evaluation metrics for the performance evaluation include weighted precision, weighted recall, and weighted score.

[0053] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0054] This invention provides a mask-enhanced named entity recognition method based on a large model. When training the recognition model, the named entity recognition task and the prediction mask task are performed and the parameters are shared. The parameters of the recognition model are updated based on a first loss function and a second loss function. Medical named entity recognition based on the trained recognition model can fully understand the semantics, has a strong ability to capture contextual dependencies, and has fewer misidentifications and omissions. Attached Figure Description

[0055] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0056] Figure 1 This is a flowchart illustrating the steps of a mask-enhanced named entity recognition method based on a large model, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the experimental environment in an embodiment of the present invention; Figure 3 This is a schematic diagram of the experimental parameters set in an embodiment of the present invention; Figure 4 This is a diagram of the BERT-CRF model using the entity masking model in this embodiment of the invention; Figure 5 This is a diagram of the BERT-CRF model using a random masking model in an embodiment of the present invention; Figure 6 This is a diagram of the BERT-Softmax model using the entity masking model in this embodiment of the invention; Figure 7 This is a diagram of the BERT-Softmax model using a random masking model in an embodiment of the present invention; Figure 8 This is a schematic diagram of the experimental results in an embodiment of the present invention; Figure 9 This is a comparison chart of the weighted F1 scores of the six models in this embodiment of the invention on the CMeEE experimental dataset; Figure 10 This is a comparison chart of the weighted F1 scores of six models on the CHIP2020 experimental dataset in this embodiment of the invention; Figure 11 This is a schematic diagram of the final prediction result of a sentence example in an embodiment of the present invention. Detailed Implementation

[0057] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0058] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0059] It should be noted that the use of terms such as "first" and "second" in the embodiments of this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0060] With the rapid development of pre-trained models, applying them to medical named entity recognition has become an important tool for improving task performance. However, existing pre-trained models suffer from semantic gaps when performing medical named entity recognition tasks, resulting in poor accuracy and generalization ability. Furthermore, medical named entity recognition tasks require pre-trained models to distinguish fine-grained semantic differences and exhibit strong contextual dependence, which existing pre-trained models often fail to capture, particularly in long-distance dependencies at the sentence or paragraph level.

[0061] Therefore, embodiments of this application provide a mask-enhanced named entity recognition method based on a large model.

[0062] This embodiment provides a mask-enhanced named entity recognition method based on a large model. Please refer to [link / reference]. Figure 1 As shown, it includes:

[0063] Step S1: Collect the text data to be recognized;

[0064] Step S2: Preprocess the text data to be recognized to obtain the input sequence;

[0065] Step S3: Input the input sequence into the trained recognition model to obtain the recognition result;

[0066] The training process of the recognition model includes:

[0067] Collect training text data; preprocess the training text data to obtain a training input sequence; perform masking processing on the training input sequence based on a set masking strategy to obtain a masked input sequence;

[0068] The training input sequence and the masked input sequence are fed into the BERT model to obtain entity context representation features and masked context representation features.

[0069] The system performs named entity recognition and prediction mask tasks with shared parameters. In the named entity recognition task, NER-Classifier is used to identify and classify the entity context representation features to obtain entity prediction values. In the prediction mask task, MLM-Classifier is used to perform token prediction on the mask context representation features to obtain mask prediction values.

[0070] A first loss function is calculated based on the entity context representation features and the entity prediction value, and a second loss function is calculated based on the mask prediction value; gradient calculation is performed based on the first loss function and the second loss function in each training cycle to update the parameters of the recognition model;

[0071] At the end of each training cycle, a performance evaluation is performed based on the entity prediction value and the mask prediction value; training is repeated until the performance of the recognition model meets the set requirements.

[0072] This embodiment provides a mask-enhanced named entity recognition method based on a large model. During the training of the recognition model, it performs named entity recognition tasks and prediction mask tasks while sharing parameters. It performs gradient calculation based on the first loss function and the second loss function to update the parameters of the recognition model, thereby improving the performance of the recognition model in medical named entity recognition. It can fully understand semantics, has strong generalization ability, strong context dependency capture ability, high recognition accuracy, and fewer misidentifications and omissions.

[0073] The following section provides a detailed description of the mask-enhanced named entity recognition method based on a large model provided in this embodiment:

[0074] I. Training process of the recognition model:

[0075] Step 01: Collect training text data; preprocess the training text data to obtain the training input sequence.

[0076] Optionally, the numbers, parentheses, and spaces in the training text data are removed, and the entity information is parsed into a start index, an end index, and an entity type and stored in dictionary form; word segmentation is performed, and the stored training text data is converted into the format required by the input model, and padded or cropped according to the specified maximum length.

[0077] Optionally, initialize the Named Entity Recognition (NER) tags to all "O", set the corresponding tags as the start tag B-dis and the middle tag I-dis of the entity type according to the entity information, and add the entity type; convert the NER tags to IDs, and pad or trim them according to the specified maximum length.

[0078] Optionally, the training text data can be converted into a training input sequence using a word segmenter.

[0079] Step 02: Perform masking processing on the training input sequence based on the set masking strategy to obtain the masked input sequence.

[0080] Optionally, the masking strategy is:

[0081] (1) There is an 80% probability that the input at the target location will be replaced with a mask [MASK];

[0082] (2) There is a 10% probability that a random token will be used to replace the token at the target location;

[0083] (3) The 10% probability remains unchanged.

[0084] Specifically, the substitution in (1) requires the recognition model to infer the masked real words through contextual information during training. Using the mask [MASK] can force the recognition model to learn to make predictions in the absence of information. The substitution in (2) increases the diversity of training data and can simulate noise and changes that may occur in actual applications. By replacing the words at the target position with random words, the recognition model will learn to process noisy data during training, which is beneficial for understanding semantic information. In (3), some original data is retained, which can ensure that the recognition model can learn how to handle semantic loss and noise changes during training, and can also maintain the memory of real words.

[0085] Optionally, a random masking model is constructed, and a portion of the tokens in the training input sequence are masked based on the masking strategy to construct a mask matrix and the masked input sequence:

[0086] First, initialize the mask probability matrix P. mask =0.15×1 mask Where 0.15 is the overall probability of the mask, and 1 mask It is a matrix of all 1s, and the length of the matrix of all 1s is the same as the length of the training input sequence.

[0087] Then, the mask matrix is ​​generated using the Bernoulli distribution. :

[0088] ;

[0089] Among them, M mask,i Let p represent the mask value at sequence position i, and p represent the selection probability of the masking strategy.

[0090] Next, for the training input sequence x i Each token in the sequence is masked using the masking strategy described above to obtain the masked input sequence x. i ':

[0091] ;

[0092] Among them, random token It is a token randomly selected from the vocabulary.

[0093] Preferably, an entity masking model is constructed, and based on the masking strategy, a portion of the tokens in the training input sequence are masked to construct a mask matrix and the masked input sequence:

[0094] First, the entities in the training input sequence x for:

[0095] ;

[0096] Then, for The masking strategy is used for the token spanned by each entity in the data:

[0097] ;

[0098] Next, regarding All entities in the sequence are subjected to the masking strategy described above to obtain the masked input sequence. Accordingly, the mask matrix value corresponding to the spanning position contained in each entity should be set to 1, indicating that the part has been masked.

[0099] ;

[0100] Where b represents the starting position of entity u, and q represents the span length of entity u; Represents the entity u after masking, random token It is a token randomly selected from the vocabulary, [MASK] represents the mask token, and p represents the selection probability of the masking strategy; M mask,i The mask matrix value represents the position i in the sequence.

[0101] Step 03: Feed the training input sequence and the mask input sequence into the BERT model to obtain entity context representation features and mask context representation features.

[0102] Optionally, the training input sequence is fed into the BERT model to generate contextual representation features for each token, i.e., the entity contextual representation features H, where H = [h1, h2, ..., h...]. n The dimension of h is D; the mask context representation feature H is obtained. m The process is similar to the process of obtaining the entity context representation feature H.

[0103] Step 04: Perform the named entity recognition task and the prediction mask task with shared parameters; in the named entity recognition task, use NER-Classifier to identify and classify the entity context representation features to obtain the entity prediction value; in the prediction mask task, use MLM-Classifier to perform token prediction on the mask context representation features to obtain the mask prediction value.

[0104] Step 04-1 Named Entity Recognition Task:

[0105] Optionally, a Conditional Random Field (CRF) model is used to calculate the conditional probability P(y│x):

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] in, W represents the score of selecting label y at sequence position t. y The weights of the NER-Classifier when using the CRF model, h t b is the BERT output representation at position t of the sequence. y This refers to the fully connected layer bias of the NER-Classifier when using the CRF model; T yt-1，yt Indicates from y t-1 Transfer to y t The score, y t-1 It is the label at sequence position t-1, y t Here, is the label at position t in the sequence; trans() represents the transition matrix; Z(x) represents the normalization factor, which ensures that the sum of the probabilities of possible label sequences is 1; exp() represents the exponential function; Σ represents the summation function; n is the sequence length; and log() represents taking the logarithm.

[0111] The Viterbi algorithm is used to find the most likely label sequence as the entity prediction value. :

[0112] ;

[0113] Optionally, a Softmax layer is applied to generate the probability of each tag, and the probability of the token at sequence position t corresponding to tag c. for:

[0114] ;

[0115] ;

[0116] Among them, z t W represents the label score at sequence position t. s The weights of the NER-Classifier when using the Softmax layer, b sThis refers to the bias of the NER-Classifier when using the Softmax layer; z t,c It corresponds to the logit value of the label c;

[0117] The Viterbi algorithm is used to find the most likely label sequence as the entity prediction value. :

[0118] ;

[0119] Step 04-2: Predict the mask task:

[0120] Optionally, the mask prediction value z i m for:

[0121] ;

[0122] Among them, W m b is the weight of the MLM-Classifier. m h represents the deviation of the MLM-Classifier. i m The mask context representation features.

[0123] Step 05: Calculate a first loss function based on the entity context representation features and the entity prediction value, and calculate a second loss function based on the mask prediction value; perform gradient calculation based on the first loss function and the second loss function in each training cycle to update the parameters of the recognition model.

[0124] Step 05-1 First Loss Function:

[0125] Optionally, when using the CRF model, the first loss function for:

[0126] ;

[0127] Optionally, when the Softmax layer is applied, the first loss function is:

[0128] ;

[0129] Where L is the number of tags. It is an indicator function. When the actual label of the sequence position t is c, the indicator function has a value of 1, otherwise it is 0.

[0130] Step 05-2 Second Loss Function:

[0131] Optionally, the second loss function for:

[0132] ;

[0133] ;

[0134] in, This represents the mask prediction value z. i m The probability distribution, These are predicted labels, V is the vocabulary size, and z is the number of labels. i,j m It is the predicted score of the masked token at sequence position i; |M mask | is the mask matrix M mask The number of non-zero elements in the neutron. It is an indicator function. When the actual label of sequence position m is j, the value of the indicator function is 1, and otherwise it is 0.

[0135] Step 05-3 Optimize parameters:

[0136] Optionally, when using the CRF model:

[0137] The first step is to freeze the parameters of the MLM-Classifier based on the first loss function. First, perform gradient calculation to update the parameters of the BERT model, the NER-Classifier, and the Conditional Random Field model. Second, freeze the weights of the NER-Classifier and the Conditional Random Field model, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the MLM-Classifier.

[0138] Optionally, when the Softmax layer is applied:

[0139] The first step is to freeze the parameters of the MLM-Classifier based on the first loss function. First, perform gradient calculation to update the parameters of the BERT model and the NER-Classifier. Second, freeze the weights of the NER-Classifier, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the MLM-Classifier.

[0140] Step 06: At the end of each training cycle, perform a performance evaluation based on the entity prediction value and the mask prediction value; repeat the training until the performance of the recognition model reaches the set requirements.

[0141] Optionally, the performance evaluation metrics include weighted precision, weighted recall, and weighted score.

[0142] Specifically, weighted precision measures the accuracy of a model when it predicts a positive class, that is, how high the proportion of samples that are actually positive out of all those predicted as positive; the higher the weighted precision, the lower the false positive rate of the model.

[0143] Specifically, weighted recall measures the ability of a model to detect positive samples, that is, what proportion of all actual positive samples the model correctly identifies; the higher the weighted recall, the higher the model's rate of identifying positive samples.

[0144] Specifically, the weighted score is the harmonic mean of the weighted precision and the weighted recall, and is a comprehensive evaluation index of the two; the higher the weighted score, the better the overall performance of the model.

[0145] II. Experimental verification of the beneficial effects:

[0146] 1. Experimental Dataset:

[0147] For example, experiments were conducted on two experimental datasets: CMeEE and CHIP2020.

[0148] The CMeEE dataset contains 504 common pediatric diseases, 7085 body parts, 12907 clinical manifestations, and 4354 medical procedures, which can be divided into a training set of 15000 entries and a test set of 5000 entries and 3000 entries. The CHIP2020 dataset contains 47194 sentences and 938 files, with an average of 2355 characters per file, which can be divided into a training set of 15000 entries and a test set of 5000 entries. Both benchmark datasets contain nine categories of medical entity labels: diseases, clinical manifestations, medical procedures, medical equipment, drugs, medical test items, body, departments, and microorganisms.

[0149] 2. Experimental environment:

[0150] For example, the experimental environment is as follows Figure 2 As shown.

[0151] 3. Experimental parameters:

[0152] For example, the experimental parameters are as follows Figure 3 The settings are shown.

[0153] Among them, *MBERT-CRF includes the EntMBERT-CRF model and the RanDMBERT-CRF model, and *MBERT-Softmax includes the EntMBERT-Softmax model and the RanDBERT-Softmax model; the following six models all adopt the BERT model:

[0154] The BERT-CRF model only uses the CRF model; the BERT-Softmx model only uses the Softmax layer.

[0155] The EntMBERT-CRF model uses the entity mask model and the CRF model, please refer to... Figure 4 As shown; the RanDMBERT-CRF model uses the aforementioned random mask model and the aforementioned CRF model. Please refer to [the documentation / reference]. Figure 5 As shown; the EntMBERT-Softmax model uses the aforementioned entity mask model and the aforementioned Softmax layer. Please refer to [the documentation / reference]. Figure 6 As shown; the RanDBERT-Softmax model uses the aforementioned random masking model and the aforementioned Softmax layer. Please refer to [the documentation / reference]. Figure 7 As shown; where E [CLS] The special placeholders [CLS] are represented by E1, E2, and E... N This indicates that they correspond to token1, token2, and token respectively. N The characterization of E [SEP] This represents the special placeholder [SEP].

[0156] 4. Experimental Results:

[0157] For example, the experimental results are as follows Figure 8 As shown, both the EntMBERT-CRF model and the EntMBERT-Softmax model exhibit significant performance improvements on both experimental datasets; please refer to... Figure 9 and Figure 10 As shown, both the EntMBERT-CRF model and the EntMBERT-Softmax model exhibit high weighted scores and fast convergence speeds on both experimental datasets.

[0158] Wherein, Precision represents the weighted precision, Recall represents the weighted recall, and F1 represents the weighted score.

[0159] It is evident that when performing named entity recognition and prediction masking tasks while sharing parameters, constructing an entity masking model and masking entities allows us to focus on the entity and its context, thereby learning entity-related features more effectively and achieving the goal of medical named entity recognition.

[0160] 5. Case Analysis:

[0161] For example, consider the sentence, "The patient presented with acute headache and hypertension. Examination revealed stenosis of the left carotid artery. The doctor recommended the use of metoprolol and ACE inhibitors, and performed an ultrasound examination and blood pressure monitoring."

[0162] For example, the entities and categories to be identified are the following seven: headache, hypertension, carotid artery stenosis, metoprolol, ACE inhibitor, ultrasound examination, and blood pressure measurement; please refer to [reference needed] for the final prediction results. Figure 11 As shown, the EntMBERT-CRF model can recognize all entities; although the EntMBERT-Softmax model failed to recognize all entities, its recognition performance is still better than that of the BERT-Softmax model and the RanDMBERT-Softmax model, which also use the Softmax layer.

[0163] It is evident that when performing named entity recognition and prediction masking tasks while sharing parameters, constructing an entity masking model and masking entities allows us to focus on the entity and its context, thereby learning entity-related features more effectively and achieving the goal of medical named entity recognition.

[0164] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0165] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0166] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0167] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0168] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A mask-enhanced named entity recognition method based on a large model, characterized in that, include: Step S1: Collect the text data to be recognized; Step S2: Preprocess the text data to be recognized to obtain the input sequence; Step S3: Input the input sequence into the trained recognition model to obtain the recognition result; The training process of the recognition model includes: Collect training text data; preprocess the training text data to obtain a training input sequence; perform masking processing on the training input sequence based on a set masking strategy to obtain a masked input sequence; The training input sequence and the masked input sequence are fed into the BERT model to obtain entity context representation features and masked context representation features. The system performs named entity recognition and prediction mask tasks with shared parameters. In the named entity recognition task, NER-Classifier is used to identify and classify the entity context representation features to obtain entity prediction values. In the prediction mask task, MLM-Classifier is used to perform token prediction on the mask context representation features to obtain mask prediction values. A first loss function is calculated based on the entity context representation features and the entity prediction value, and a second loss function is calculated based on the mask prediction value; gradient calculation is performed based on the first loss function and the second loss function in each training cycle to update the parameters of the recognition model; At the end of each training cycle, a performance evaluation is performed based on the entity prediction value and the mask prediction value; training is repeated until the performance of the recognition model meets the set requirements.

2. The mask-enhanced named entity recognition method based on a large model according to claim 1, characterized in that, The method for masking the training input sequence based on a set masking strategy to obtain the masked input sequence includes: Construct an entity masking model, and perform masking processing on a portion of the tokens in the training input sequence based on a set masking strategy, to construct a mask matrix and the masked input sequence: First, the entities in the training input sequence x for: ； Then, for The masking strategy is used for the token spanned by each entity in the data: ； Next, regarding All entities in the sequence are subjected to the masking strategy described above to obtain the masked input sequence. Accordingly, the mask matrix value corresponding to the spanning position contained in each entity should be set to 1, indicating that the part has been masked. ； Where b represents the starting position of entity u, and q represents the span length of entity u; Represents the entity u after masking, random token It is a token randomly selected from the vocabulary, [MASK] represents the mask token, and p represents the selection probability of the masking strategy; M mask,i The mask matrix value represents the position i in the sequence.

3. The mask-enhanced named entity recognition method based on a large model according to claim 1, characterized in that, The method for calculating the first loss function based on the entity context representation features and the entity prediction value includes: Using a conditional random field model, the first loss function is... Perform the calculation: ； ； ； ； ； Where emissions(t,y) represents the score for selecting label y at sequence position t, W y The weights h of the NER-Classifier when using the conditional random field model are described above. t b is the BERT output representation at position t of the sequence. y This refers to the bias of the fully connected layer of the NER-Classifier when using the conditional random field model; T yt-1，yt Indicates from y t-1 Transfer to y t The score, y t-1 It is the label at sequence position t-1, y t Here, t is the label of the sequence position, trans() represents the transition matrix, P(y│x) represents the conditional probability, Z(x) represents the normalization factor, exp() represents the exponential function, Σ represents the summation function, n is the sequence length, and log() represents taking the logarithm.

4. The mask-enhanced named entity recognition method based on a large model according to claim 3, characterized in that, Based on the first loss function The method for calculating the gradient with the second loss function to update the parameters of the recognition model is as follows: The first step is to freeze the parameters of the MLM-Classifier based on the first loss function. First, perform gradient calculation to update the parameters of the BERT model, the NER-Classifier, and the Conditional Random Field model. Second, freeze the weights of the NER-Classifier and the Conditional Random Field model, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the MLM-Classifier.

5. The mask-enhanced named entity recognition method based on a large model according to claim 1, characterized in that, The method for calculating the first loss function based on the entity context representation features and the entity prediction value includes: Using a Softmax layer, the first loss function is... Perform the calculation: ； ； ； Among them, z t W represents the label score at sequence position t. s The weights of the NER-Classifier when using the Softmax layer, h t b is the BERT output representation at position t of the sequence. s This refers to the bias of the NER-Classifier when using the Softmax layer; p(y t =c│h t ) is the probability that the token at position t in the sequence corresponds to the tag c, z t,c This corresponds to the logit value of the label c, where exp() represents the exponential function, Σ represents the summation function, n is the sequence length, and L is the number of labels. It is an indicator function. When the actual label of the sequence position t is c, the indicator function has a value of 1, otherwise it is 0.

6. The mask-enhanced named entity recognition method based on a large model according to claim 5, characterized in that, Based on the first loss function The method for calculating the gradient with the second loss function to update the parameters of the recognition model is as follows: The first step is to freeze the parameters of the MLM-Classifier based on the first loss function. First, perform gradient calculation to update the parameters of the BERT model and the NER-Classifier. Second, freeze the weights of the NER-Classifier, unfreeze the parameters of the MLM-Classifier, perform gradient calculation based on the second loss function, and update the parameters of the BERT model and the MLM-Classifier.

7. The mask-enhanced named entity recognition method based on a large model according to claim 1, characterized in that, Calculate the second loss function based on the mask prediction value. The method is as follows: ； ； ； Among them, z i m W represents the predicted mask value. m b is the weight of the MLM-Classifier. m h represents the deviation of the MLM-Classifier. i m The mask context represents the features; This represents the mask prediction value z. i m The probability distribution, This is for predicting labels, where exp() represents the exponential function, Σ represents the summation function, V is the vocabulary size, and z is the z-value. i,j m It is the predicted score of the masked token at sequence position i; |M mask | is the mask matrix M mask The number of non-zero elements in the sequence, where n is the sequence length; It is an indicator function; its value is 1 when the actual label of sequence position m is j, and 0 otherwise; M mask,i It is the mask value of the sequence position i. If the sequence position i is masked, M mask,i The value is 1 if it is not 0 otherwise; log() means taking the logarithm.

8. The mask-enhanced named entity recognition method based on a large model according to claim 1, characterized in that, The masking strategy is as follows: there is an 80% probability of replacing the input at the target location with a mask [MASK], a 10% probability of replacing the token at the target location with a random token, and a 10% probability of not making any changes.

9. The mask-enhanced named entity recognition method based on a large model according to claim 1, characterized in that, When using NER-Classifier to identify and classify the entity context representation features to obtain entity prediction values, a label sequence as the entity prediction value is found based on the Viterbi algorithm.

10. The mask-enhanced named entity recognition method based on a large model according to claim 1, characterized in that, At the end of each training cycle, a performance evaluation is performed based on the entity prediction value and the mask prediction value. The evaluation metrics for the performance evaluation include weighted precision, weighted recall, and weighted score.

Images