Common disease named entity recognition method fusing BERT and BiLSTM

Abstract

The invention discloses a common disease named entity recognition method fusing BERT and BiLSTM, and belongs to the technical field of natural language processing and medical information processing. The method comprises the steps that firstly, dynamic word vector representation of an input text is obtained through a BERT pre-training model, and deep semantic features of the text are captured through a bidirectional Transform structure of the dynamic word vector representation; inputting a BERT output vector into a BiLSTM network for context feature enhancement, and extracting a forward and backward long-distance dependency relationship of the sequence through a bidirectional long-short-term memory network; and finally, introducing a CRF layer to carry out tag sequence constraint on features output by the BiLSTM, and optimizing entity boundary identification by using a transition probability matrix. Aiming at the characteristics of Chinese medical texts, a word and word combination mode is adopted to process the problem of unregistered words, and a labeling corpus containing entities such as symptoms, diseases and drugs is constructed for model training, so that the method remarkably improves the recognition accuracy of the medical entity boundary.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing (NLP), and in particular to a method for recognizing named entities for common diseases that integrates BERT (Bidirectional Encoder Representations from Transformers) and BiLSTM (Bidirectional Long Short-Term Memory). The method is suitable for scenarios such as Chinese medical text structuring, intelligent medical question-and-answering, clinical decision support, and medical knowledge graph construction. Background Art

[0002] With the widespread adoption and intelligent transformation of electronic medical record systems, clinical data platforms, and medical question-and-answer systems, the medical field is rapidly accumulating massive amounts of unstructured text data. According to the "White Paper on the Development of Medical Informatization in China," tertiary hospitals alone generate over 50,000 text items per day, including outpatient medical records, inpatient records, and imaging reports. Key medical entity information accounts for 83% of this, forming the underlying framework of medical knowledge graphs and the core basis for clinical decision-making. Accurately extracting entities such as disease names, symptom descriptions, drug ingredients, and treatments is a core challenge in promoting the implementation of applications such as intelligent diagnosis, rational drug use monitoring, and disease risk prediction. Named entity recognition (NER) technology is a key component of natural language processing, responsible for transforming free text into structured knowledge. Its recognition accuracy directly determines the reliability of subsequent medical knowledge reasoning.

[0003] From the 1990s to the early 2000s, named entity recognition (NER) relied primarily on rule-based approaches. Medical experts manually compiled terminology dictionaries and designed regular expression templates and context-constrained rules. A typical example is the MedLEE system, which achieved an 85% accuracy rate in radiology reports. However, it had limitations: weak generalization of term variants (the probability of "myocardial infarction" matching "myocardial infarction" was less than 40%), a high rate of mis-splitting when handling nested entities (for example, "type 2 diabetic nephropathy stage IV" was misclassified as two separate entities), and expansion to new departments required the reconstruction of 70% of the rule base, resulting in development costs exceeding 200 man-days, making it difficult to cope with the complexity and diversity of clinical texts.

[0004] To overcome the bottleneck of rule-based methods, researchers from the early 2000s to the 2010s turned to statistical learning-based approaches. Hidden Markov models (HMMs) model entity boundaries using state transition probabilities. For example, the transition probability of "cough → cough" (P=0.93) is significantly higher than that of "cough → blood" (P=0.21). Maximum entropy models (MEs) fuse multi-source features to predict label distributions. Conditional random fields (CRFs) have become a mainstream framework, jointly optimizing label sequences by defining feature functions, such as:

[0005]

[0006]

[0007] This type of method achieved an F1 value of 85.7% on the CoNLL2003 English dataset, but it exposed two major flaws when applied to Chinese medical scenarios: First, the ambiguity of Chinese word segmentation makes it difficult to identify entity boundaries. For example, "cough with chest pain" may be incorrectly segmented, and the recall rate of complex symptoms is only 74.3%; second, the model relies on manually designed feature engineering and needs to be reconfigured when migrating across departments. The F1 value of pediatric text is 19.7 percentage points lower than that of internal medicine text.

[0008] In recent years, deep learning has driven the development of NER technology. BERT uses the Transformer architecture to generate dynamic word embeddings, addressing polysemy. BiLSTM captures long-range contextual dependencies. CRF, used in the decoding layer, constrains label logic. The BERT-BiLSTM-CRF architecture achieved an F1 score of 89.5% on the CMeEE Chinese medical dataset, a significant breakthrough.

[0009] However, existing models still face three challenges in Chinese medical scenarios: First, over 30% of medical entities exhibit complex nested structures, resulting in a 32% error rate in boundary recognition for traditional models; second, generalization of abbreviations and terminology variants is difficult, and static word vectors cannot establish semantic equivalence; and third, cross-domain transferability is weak, with the F1 score dropping 15-20 percentage points when migrating models from internal medicine to pediatrics. These bottlenecks restrict the effectiveness of medical AI applications in real-world scenarios, necessitating innovative technical solutions. Summary of the Invention

[0010] The present invention aims to solve the problems existing in the existing technology in the Chinese medical named entity recognition task, such as difficulty in recognizing nested structures, weak ability to handle term ambiguity, and poor cross-domain generalization ability of the model. A common disease named entity recognition method that integrates BERT and BiLSTM is proposed.

[0011] The main technical solutions of the present invention include:

[0012] 1. Architecture: The entity extractor uses a sequence annotation structure (such as the BIOES annotation system) and a pre-trained language model (such as BERT) as the encoder. Given an input sequence X = [x1, x2, …, xn], the encoder outputs a latent vector:

[0013] in is the sequence length, is the hidden layer dimension. A single-layer linear network will Mapped to label probability matrix ( is the number of boundary label categories).

[0014] 2. Two-stage training mechanism - Coarse-grained training stage: Data re-labeling algorithm, constructing entity candidate sets for unlabeled corpora such as Wikipedia Through the hash table Record Entity The frequency of anchor text and the total frequency The rules for filtering low-quality entities are as follows: in ( Through empirical optimization, it can filter out non-entity high-frequency words such as "the".

[0015] Noun-aware loss function: reduces the training weight of noun-based non-entities to alleviate the noise caused by unrecalled entities:

[0016] 3. Multivariate Gaussian distribution modeling:

[0017] For each type of entity Establishing feature-independent multivariate Gaussian distribution .

[0018] Calculation Category The mean vector of and variance vector :

[0019]

[0020]

[0021] in For category The sample set, For words The discriminator encoding.

[0022] 4. Probability density and loss function: word In category The probability density of :

[0023] The discriminator loss is in the form of cross entropy:

[0024] 5. Joint reasoning mechanism - entity boundary extraction: The entity extractor outputs a set of candidate entities:

[0025] 6. Type voting decision:

[0026] Step 1: Count each word The most confident type label:

[0027] Step 2: Entity All Voting, the highest vote type set .

[0028] like , direct output type;

[0029] like , calculate the cumulative score and take the maximum value:

[0030]

[0031] 7. Cross-domain generalization: After the entity extractor strips away the type information, the grammatical knowledge learned in the source domain (such as nominal entity patterns) can be directly transferred to the target domain.

[0032] On the FEW-NERD INTRA dataset, the F1 value under the 5-way / 10-way setting reached 55.50 / 49.35, an increase of 6.02 / 6.51 percentage points over the baseline DecomposedMetaNER.

[0033] 8. Distribution Modeling Robustness:

[0034] The multivariate Gaussian distribution accurately describes the distribution of category features and avoids the simplification bias of the mean representation of the traditional prototype network.

[0035] The voting-accumulation two-stage decision-making alleviates local confidence anomalies (for example, the probability of the correct type at a certain location is 0), and the error tolerance rate is improved by 40%.

[0036] 9. Continuous learning ability:

[0037] The two-stage training framework supports incremental updates: the coarse-grained stage absorbs general knowledge from Wikipedia (covering more than 5 million entities), and the fine-grained stage requires only 1,000 domain-annotated samples for fine-tuning, reducing annotation costs by 90%.

[0038] 10. Entity extractor output:

[0039] Taking the medical text "a child has fever and cough, suspected pneumonia" as an example, boundary recognition: ["suffer", "O"; "child", "O"; "rash", "B-SYM"; "fever", "E-SYM"; "accompanied", "O"; "cough", "B-SYM"; "cough", "E-SYM"; "suspected", "O"; "diagnosis", "O"; "lung", "B-DIS"; "inflammation", "E-DIS"];

[0040] Candidate entity set: .

[0041] 11. Type discriminator calculation:

[0042] Pair : Location Voting: Lung → Disease (confidence 0.92), Inflammation → Disease (confidence 0.88) → Direct output type: Disease (DIS)

[0043] This solution addresses the core issues of fuzzy entity boundaries and type confusion in few-shot scenarios through task decoupling, distributed modeling optimization, and joint reasoning mechanisms, providing an extensible solution for entity recognition in cross-domain medical, financial, and other professional scenarios. The present invention has the following advantages compared with the prior art:

[0044] 1. It can effectively identify nested entities and compound terms;

[0045] 2. It has a strong semantic consistency judgment ability for medical abbreviations;

[0046] 3. It enhances the model's syntax awareness through multi-task joint optimization;

[0047] 4. It has good robustness and scalability in cross-domain transfer tasks;

[0048] 5. The model architecture supports lightweight transformation and is suitable for actual clinical system deployment. Brief Description of the Drawings

[0049] Figure 1 It is the overall system flowchart, showing the four-step process of text preprocessing, highlighting character-level word segmentation and WordPiece conversion.

[0050] Figure 2 It is the structural schematic diagram of the method of the present invention, showing the overall processing flow of medical text after BERT dynamic encoding, BiLSTM modeling, and CRF label decoding, including the input layer, BERT encoding layer, BiLSTM modeling layer, CRF structure layer, and output layer. Detailed Embodiments

[0051] 1. Text preprocessing:<​​​​​​​​​​​​​WordPiece segmentation: Using the BERT vocabulary (containing 21,128 Chinese characters and subwords), the formula is:

[0056]

[0057] The processing example is as follows Figure 1 shown.

[0058] 2. BERT encoding module:

[0059] Use the BERT-Base Chinese pre-training model (12-layer Transformer, 768 hidden layers):

[0060]

[0061]

[0062]

[0063]

[0064] .

[0065] Whole-word masking strategy: mask compound medical terms as a whole, for example:

[0066] Original: "Acute exacerbation of chronic obstructive pulmonary disease";

[0067] Mask: “[MASK][MASK][MASK][MASK]Acute exacerbation”;

[0068] This strategy enables the model to learn term-level semantic representations and improve the ability to recognize abbreviations such as "COPD".

[0069] 3. BiLSTM modeling module:

[0070] The bidirectional LSTM receives the BERT output and extracts contextual features:

[0071]

[0072]

[0073]

[0074] Parameter configuration: Hidden unit: 256 dimensions (512 dimensions in two directions);

[0075] Dropout rate: 0.3;

[0076] Maximum sequence length: 512 tokens.

[0077] This module is specifically optimized for compound entity recognition:

[0078] Example input: "Cough with expectoration for 3 days".

[0079] BiLSTM output:

[0080] "Cough" → symptom start, "Sore throat" → symptom continuation,

[0081] "With" → conjunction, "Expectoration" → symptom start, "Phlegm" → symptom end.

[0082] 4. CRF decoding module:

[0083] Conditional random field constraints the label transition logic:

[0084]

[0085]

[0086]

[0087] .

[0088] The label set adopts the BIOES scheme:

[0089] .

[0090] Illegal transfer constraints (partial):

[0091]

[0092] 5. Multi-task joint optimization:

[0093]

[0094]

[0095]

[0096]

[0097] .

[0098] Part-of-speech tagging set:

[0099]

[0100] An example of the multi-task joint architecture processing is shown in the appendix Figure 1 as follows.

[0101] 6. Transfer learning and domain adaptation

[0102] Adaptation process in the field of pediatrics:

[0103]

[0104]

[0105] .

[0106]

[0107]

[0108]

[0109]

[0110] .

[0111] Pediatric adaptation effect:

[0112] Initial F1: 76.3% (without adaptation)

[0113] After adaptation, F1: 81.2% (+4.9%).

[0114] 7. Experimental results:

[0115] Performance comparison on three Chinese medical datasets:

[0116] Table 1 Model performance comparison (F1 value / %)

[0117] .

[0118] 8. Lightweight deployment:

[0119] Model compression technology:

[0120] Distillation loss:

[0121]

[0122] Deployment plan comparison:

[0123] .

[0124] Detailed description of embodiments

[0125] Implementation Case 1: Respiratory Medical Record Analysis

[0126] Enter text:

[0127] "The patient is a 65-year-old male who complained of recurrent coughing, sputum production and shortness of breath for 5 years, which worsened for 1 week. Physical examination: scattered moist rales in both lungs. Diagnosis: acute exacerbation of COPD."

[0128] Processing flow:

[0129] 1. Participle: ["patient", "patient", "male", "sex", ",","65", "years old", ",","main", "complaint", "repeated", "cough", "cough", "cough", "sputum", "accompanied by", "short breath", "short breath"...]

[0130] 2. BERT encoding: The semantic similarity between the dynamic vector of "COPD" and "chronic obstructive pulmonary disease" is 0.92.

[0131] 3. BiLSTM output:

[0132] “Cough” → B-SYM, I-SYM

[0133] "Coughing up phlegm" → B-SYM, E-SYM

[0134] “Acute exacerbation of COPD” → B-DIS, I-DIS, I-DIS, E-DIS.

[0135] 4. CRF decoding: output structured entities

[0136] “Symptoms”: [“cough”, “sputum”, “shortness of breath”],

[0137] “Disease”: [“Acute exacerbation of COPD”].

[0138] Implementation Case 2: Pediatric Transfer Learning

[0139] Source domain: Internal medicine electronic medical records (annotated data)

[0140] Target domain: Pediatric clinic records (unlabeled).

[0141] Adaptation steps:

[0142] 1. Load CMeEE pre-trained model

[0143] 2. Continue pre-training with pediatric data:

[0144] Learning rate: 1e-5

[0145] Batch size: 32

[0146] Masking ratio: 15%.

[0147] 3. Feature distribution matching:

[0148] The pseudo code is as follows:

[0149] for epoch in range(10):

[0150] #Train the discriminator

[0151] D_loss = bce(D(H_src), 1) + bce(D(H_tgt), 0)

[0152] #Training generator

[0153] G_loss = bce(D(H_tgt), 1) + kl_div(H_src,H_tgt).

[0154] 4. After fine-tuning on 500 annotated pediatric data, the F1 score increased from 76.8% to 81.2%.

[0155] Technical effect verification

[0156] 1. Ablation Experiment (CMeEE Dataset)

[0157] .

[0158] 2. Nested Entity Recognition

[0159] Test sample: "Diabetic nephropathy stage III"

[0160] This invention identifies: a single disease entity (DIS)

[0161] Baseline model mis-segmentation: [“diabetes”, “stage III kidney disease”].

[0162] 3. Cross-domain generalization

[0163]

[0164] Implementation Notes

[0165] 1. Hardware configuration recommendations:

[0166] Training: NVIDIA Tesla V100×4 (32GB video memory)

[0167] Inference: NVIDIA T4 GPU (16GB VRAM) or Intel Xeon 6346 CPU.

[0168] 2. Parameter Tuning Guide

[0169]

[0170]

[0171] .

[0172] 3. Error handling mechanism

[0173] Abbreviation analysis: built-in medical dictionary (e.g.: "ASP" → "aspirin")

[0174] Boundary correction: rule engine correction (e.g.: "lung #inflammation" → "pneumonia").

[0175] This paper achieves high-precision entity recognition in medical texts by using a BERT-BiLSTM-CRF fusion architecture, combined with dynamic semantic encoding, bidirectional sequence modeling, and label transfer constraints. The innovation is reflected in:

[0176] 1. Full-word masking strategy to enhance medical terminology representation

[0177] 2. Multi-task joint optimization improves grammatical perception

[0178] 3. Hierarchical transfer learning solves domain adaptation problems

[0179] 4. Distillation and compression technology enables efficient deployment.

[0180] Experimental verification shows that this method achieves SOTA performance on multiple medical datasets, providing a reliable technical foundation for applications such as electronic medical record structuring and clinical decision support.

Claims

1. A method for common disease named entity recognition integrating BERT and BiLSTM, characterized in that: The following steps are involved: S1: Preprocess medical text and generate dynamic word vectors using the BERT model; S2: Input the dynamic word vector into the BiLSTM network to extract the context feature sequence of the text; S3: Based on the feature sequence output by BiLSTM, the label sequence is optimized through the CRF layer to output the named entity recognition result; The BERT model is pre-trained using a full-word masking strategy and fine-tuned using a medical terminology library. The BiLSTM network includes a bidirectional LSTM structure to capture long-term dependencies in both the forward and reverse directions. The CRF layer constrains illegal label combinations through the label transfer matrix and uses the Viterbi algorithm to search for the global optimal label sequence.

2. A method for common disease named entity recognition integrating BERT and BiLSTM according to claim 1, characterized in that: The S1 step specifically includes: Perform WordPiece segmentation on the input text and add [CLS] and [SEP] tags; Generate word vectors through BERT's Token Embeddings, Segment Embeddings, and Position Embeddings; The word vector is input into the multi-layer Transformer encoder and the dynamic semantic encoding vector is output.

3. The method for common disease named entity recognition integrating BERT and BiLSTM according to claim 1, characterized in that: The calculation process of the BiLSTM network in step S2 includes: Calculating the forget gate ,in is the weight matrix, is the bias term; Computational memory gate ; Update cell status ; Compute output gate ; Output the current hidden state , and concatenate the bidirectional hidden states to obtain the context feature sequence.

4. The method for common disease named entity recognition integrating BERT and BiLSTM according to claim 1, characterized in that: The global score function of the CRF layer in the S3 step is defined as: ,in, is the label probability output by BiLSTM, A is the label transfer matrix, and is the sequence boundary label.

5. The method for common disease named entity recognition integrating BERT and BiLSTM according to claim 1, characterized in that: Also includes multi-task training strategies: In the coarse-grained training phase, Wikipedia corpus is used for pre-training of entity boundary recognition; In the fine-grained training phase, the entity type classification is fine-tuned by combining the CMeEE and IMCS-NER medical datasets; Adopting noun-aware loss function , giving dynamic weights to noun words.

6. The method for common disease named entity recognition integrating BERT and BiLSTM according to claim 1, characterized in that: The described method supports cross-domain transfer learning: Based on the source domain pre-training model, the word vector distribution of BERT is adjusted through the domain adaptation strategy; A meta-learning framework is used to generate few-shot training tasks from unlabeled data on pediatric diseases. Enhance the recognition of nested entity boundaries based on the attention mechanism and calculate the entity fragment weights: in, is the learnable parameter matrix, is the hidden state output by BiLSTM.

7. A system for implementing the method for common disease named entity recognition integrating BERT and BiLSTM as described in any one of claims 1 to 6, characterized in that: include: Text preprocessing module, used to generate dynamic word vectors; The context modeling module includes a BERT-BiLSTM dual-channel structure for extracting semantic and sequence features; The label optimization module outputs the final entity label based on the CRF transfer constraint; The domain adaptation module supports incremental training of the model on unlabeled medical data.

8. The method for common disease named entity recognition integrating BERT and BiLSTM according to claim 7, characterized in that: The domain adaptation module includes: Entity normalization unit, aligning the recognition results with the standard entities in the UMLS medical knowledge base; The abbreviation disambiguation unit calculates the similarity between the abbreviation and the candidate full name based on contextual attention: ,in, , The entity encoding vector output by BERT.

Images