A Chinese named entity recognition method in the medical field

By combining the BBCPR model with BERT, BiLSTM, and CRF, and introducing a POS fusion layer and READ regularization method, the problems of poor model performance and overfitting in Chinese medical named entity recognition are solved, and the robustness and accuracy of the model are improved.

CN114638214BActive Publication Date: 2024-11-29NAT UNIV OF DEFENSE TECH

Patent Information

Application Number
CN202210268640.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-18
Publication Date
2024-11-29
Estimated Expiration
2042-03-18

AI Technical Summary

Technical Problem

Chinese named entity recognition in the medical field suffers from poor model performance, overfitting, and small data scale, especially when using external expertise, the model lacks generality and robustness.

Method used

We employ the BBCPR model, combining BERT, BiLSTM, and CRF, and introduce a POS fusion layer and READ regularization method. We also utilize adversarial training and dropout techniques to improve the robustness and accuracy of the model.

Benefits of technology

It significantly improves the accuracy and robustness of Chinese medical named entity recognition, especially its performance on small datasets, improving F1 score and recall.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114638214B_ABST
    Figure CN114638214B_ABST
Patent Text Reader

Abstract

The invention discloses a method for identifying Chinese named entities in the medical field. The method uses a BBCPR model to identify Chinese named entities in the medical field. The BBCPR model is composed of a word embedding layer, a BERT embedding layer, a POS fusion layer, a BiLSTM layer and a CRF layer. The word embedding layer converts a given sentence into a word embedding, which is input into the BERT embedding layer; the BERT embedding layer adopts an MCBERT encoder to obtain a BERT output embedding; the POS fusion layer connects the BERT output embedding and the POS embedding to obtain a fusion embedding; the BiLSTM layer encodes the fusion embedding to obtain the final implicit representation of the input sequence; the CRF layer decodes the output of the BiLSTM layer to obtain a label sequence and outputs it. The method can explicitly learn word boundary information, and can solve the overfitting problem and enhance the robustness of the model on small data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and digital medical technology, and specifically relates to a Chinese named entity recognition method in the medical field. Background Art

[0002] Named Entity Recognition (NER) is a core task in Natural Language Processing (NLP), which aims to identify potential entities and their categories from unstructured text. As an important component of many NLP downstream tasks (such as relation extraction and information retrieval), NER has always been a hot issue in the NLP community and has attracted a lot of attention. In general, most previous works focus on the NER task in English and have achieved good performance by integrating character-level features.

[0003] Compared to NER in English, East Asian languages ​​such as Chinese usually lack clear word boundaries and have complex composition forms, which are more challenging for NER models. For example, the performance of the current best performing models (SOTAs) in Chinese NER tasks is much lower than that of the SOTAs in English NER tasks, with a gap of nearly 10% in F1 evaluation indicators. In addition, recent research has paid more attention to NER in specific fields, such as medicine, which is complex and requires expertise in external fields.

[0004] Chinese named entity recognition (CNER) in the medical field is considered as a character-level sequence labeling problem, while English is a word-level problem. Recently, deep learning methods have been widely used for CNER tasks due to their excellent ability in automatically extracting features from massive data. For example, previous works usually utilize bidirectional long short-term memory (BiLSTM) networks to capture sequence information and achieve comparable results. In addition, due to the superior ability of language models in extracting contextual information, transformer-based models such as BERT are becoming a new paradigm for CNER.

[0005] Especially in the medical field, external expertise is beneficial for the model to understand technical terms and recognize word boundaries, which has prompted recent studies to add dictionary knowledge on top of traditional BiLSTM-CRF or BERT architectures. However, high-quality dictionary construction usually requires a lot of time and expertise, which is very expensive and labor-intensive. In addition, these dictionary-based methods may reduce the versatility and robustness of NER models. At the same time, due to privacy, ethical and highly specialized restrictions, annotated Chinese medical NER data is difficult to obtain and is usually small in scale. The small scale can easily lead to the overfitting problem of the model. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a method for improving Chinese named entity recognition in the medical field using part-of-speech information and a new regularization method. The method uses the BBCPR (BERT-BiLSTM-CRF with POS and Regularization) model to identify Chinese named entities in the medical field. The BBCPR model uses a POS (discourse) fusion layer to incorporate external grammatical knowledge, and introduces a novel READ (REgularization and Adversarial training and Dropout) method to improve the robustness of the model. The BBCPR model consists of a word embedding layer, a BERT embedding layer, a POS fusion layer, a BiLSTM layer, and a CRF layer.

[0007] The word embedding layer converts the given sentence into a word embedding E and inputs it into the BERT embedding layer.

[0008] The BERT embedding layer uses the MCBERT encoder to get the BERT output embedding, which is used as the input of the POS fusion layer.

[0009] The POS fusion layer concatenates the BERT output embedding and the POS embedding to obtain a fused embedding, where the POS embedding is obtained by passing the POS tag of a given sentence through the POS embedding layer.

[0010] The BiLSTM layer encodes the fused embedding to obtain the final implicit representation of the input sequence.

[0011] The CRF layer decodes the output of the BiLSTM layer to obtain a label sequence and output it.

[0012] The BBCPR model uses a regularization module with adversarial training and abandonment. The regularization module uses FGSM to generate adversarial word embedding E' through adversarial perturbations on word embedding E. The word embedding E and adversarial word embedding E' are respectively generated into two different sub-modules through the Dropout mechanism, and two different model prediction distributions are output. Then, the bidirectional KL divergence between the two prediction distributions is minimized to reduce the prediction difference between the two sub-models.

[0013] The bidirectional KL divergence between the two prediction distributions is minimized to reduce the prediction difference between the two sub-models. The training goal is to make the loss function of the data (X, Y) Minimize:

[0014]

[0015]

[0016]

[0017] Where λ is the coefficient weight, P(Y|X) is the probability distribution of word embedding E, P′(Y|X) is the probability distribution of adversarial word embedding E′, and D KL is the KL divergence, X is the predicted sentence, and Y is the label sequence output by the BBCPR model.

[0018] The POS fusion layer concatenates the BERT output embedding and the POS embedding, specifically: where v i is the concatenation of BERT output embedding and POS embedding, h i is the output embedding of BERT, p i is the POS embedding of the i-th token in the sentence.

[0019] Furthermore, the given sentence is obtained through the LAC (Lexical Analysis of Chinese) tool to obtain POS tags, and the POS tags are sent to the POS embedding layer to obtain POS embedding.

[0020] Furthermore, each token in the sentence is predicted using BIO tags (Begin, Inside, Outside).

[0021] Furthermore, MCBERT consists of a stack of L identical layers, each layer contains two sub-layers, where the first sub-layer is a multi-head self-attention mechanism, and the second sub-layer is a fully connected feed-forward neural network, and the two sub-layers are connected in sequence through residual connections and layer normalization.

[0022] Furthermore, the calculation process of BiLSTM is:

[0023]

[0024]

[0025]

[0026] in and They represent forward LSTM and backward LSTM respectively.

[0027] The present invention designs a POS fusion module based on MCBERT combined with BiLSTM-CRF, and incorporates POS information into the model, which can be used to guide the model to explicitly learn the word boundary information of CNER tasks in the medical field. At the same time, a new regularization method is proposed to regularize the model output on adversarial training and dropout perturbation samples to solve the overfitting problem and enhance the robustness of the model on small data. The present invention conducts comprehensive experiments on the cMedQANER and cEHRNER datasets. The results show that the BBCPR model outperforms several competitive baselines, and the ablation study proves the effectiveness of the proposed method. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0029] Figure 1 This is the overall architecture diagram of the BBCPR model.

[0030] Figure 2 This is the architecture diagram of the READ regularization method.

[0031] Figure 3 Schematic diagram of word embedding.

[0032] Figure 4 The impact of the BBCPR model on the cMedQANER and cEHRNER test sets.

[0033] Figure 5 Effect of POS embedding size on the test set of cMedQANER and cEHRNER.

[0034] Figure 6 The effect of the regularized loss function weight λ on the cMedQANER and cEHRNER test sets.

[0035] Figure 7 The effect of dropout rate on the cMedQANER and cEHRNER test sets. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] like Figure 1 As shown in the figure, the BBCPR model consists of a word embedding layer, a BERT embedding layer, a POS fusion layer, a BiLSTM layer, and a CRF layer.

[0038] The word embedding layer converts the given sentence into a word embedding E and inputs it into the BERT embedding layer.

[0039] The BERT embedding layer uses the MCBERT encoder to get the BERT output embedding, which is used as the input of the POS fusion layer.

[0040] The POS fusion layer concatenates the BERT output embedding and the POS embedding to obtain a fused embedding, where the POS embedding is obtained by passing the POS tag of a given sentence through the POS embedding layer.

[0041] The BiLSTM layer encodes the fused embedding to obtain the final implicit representation of the input sequence.

[0042] The CRF layer decodes the output of the BiLSTM layer to obtain a label sequence and output it.

[0043] The CNER task in the medical field is designed to identify and predict entities in text (such as diseases, symptoms, drugs, etc.). This paper regards the CNER task in the medical field as a sequence labeling problem. Given a sentence X = {x1, x2, ..., x n}, the goal is to predict each token x in sentence X using BIO tags (Begin, Inside, Outside) i , and obtain a label sequence Y = {y1,y2,...,y n} as output. Examples of label entities in sentences are shown in Table 1, where Bs represents the beginning of entity symptom, Is represents the inside of entity symptom, Bd represents the beginning of entity disease, Id represents the inside of entity disease, Bp represents the beginning of entity person, Ip represents the inside of entity person, and O represents the external entity.

[0044] Table 1 Entity tags in sentences

[0045]

[0046] For clarity, the main symbols used are summarized in Table 2.

[0047] Table 2 Main symbols used in this paper

[0048]

[0049]

[0050] BERT is a pre-trained model with multiple layers of bidirectional transformer encoders that obtain deep language representations by jointly learning each layer with the context. MCBERT is a further training of BERT on the Chinese Medical Corpus and has the same structure as BERT. It consists of a stack of N identical layers. Each layer contains two sub-layers, where the first sub-layer is a multi-head self-attention mechanism and the second sub-layer is a fully connected feed-forward neural network. The two sub-layers are connected in sequence through residual connections and layer normalization.

[0051] The word embedding layer first converts the given sentence X into a series of word embeddings E = {e1, e2, ..., e n}, simply embedding tags, segments, and positions, such as Figure 2 shown.

[0052] The BERT embedding layer uses the MCBERT encoder, and the input of the MCBERT encoder is E. For convenience, the input of the first layer is recorded as H0, and the output of the lth layer is recorded as H l The output of the previous layer represents H l-1 is fed into the multi-head self-attention (MSA) sub-layer to obtain the context-level representation. The output representation of the network (FFN) sub-layer. These operations are expressed as:

[0053] H′ l =LayerNorm(MSA(H l-1 )+H i l-1 )

[0054] H l =LayerNorm(H′ l +FFN(H′ l ))

[0055] The final BERT output embedding {h1,h2,...,h n} is sent to the POS fusion layer.

[0056] The POS fusion layer incorporates the features of the speech part into the model. It connects the BERT output embedding and the POS embedding to obtain a fused embedding. The POS embedding is obtained by passing the POS tag of the given sentence through the POS embedding layer. Since BERT processes text as tags and generates tag-level embeddings. But in Chinese, words are usually used as the smallest unit to express semantics. Although the tag-based model can avoid segmentation errors, it loses some semantics and also increases the difficulty of entity boundary extraction. POS refers to word features that serve as the basis for word classification, including nouns, verbs, adjectives, modal particles, etc. POS can greatly improve NER annotations.

[0057] For a given sentence X = {x1, x2, ..., x n}, use Baidu LAC tool to obtain POS tags T = {t1, t2, ..., t n Then T is fed into the POS embedding layer to obtain POS embedding P = {p1, p2, ..., p n},in d p represents the size of the POS embedding layer. The POS fusion layer embeds the BERT output into h i and POS embedding p i Connect in

[0058] The BiLSTM layer can capture more comprehensive contextual information. This application introduces BiLSTM to encode the fusion embedding. It consists of a forward LSTM and a backward LSTM. LSTM can more accurately capture long-distance dependencies and avoid the gradient vanishing or gradient exploding problems caused by standard RNN. Therefore, BiLSTM can better capture long-distance and bidirectional semantic dependencies by learning contextual information in two directions. An LSTM unit at time t can be defined as follows:

[0059] i t =σ(W i [h t-1 ,v t ]+b i )

[0060] f t =σ(W f [h t-1 ,v t ]+b f )

[0061]

[0062] o t =σ(W o [h t-1 ,v t ]+b o )

[0063]

[0064] h t =o t ⊙tanh(C t )

[0065] where v t and h t W represents the input vector and hidden state at time t.i , W f , W c , W o and b i 、b f 、b c 、b o is a learnable parameter of LSTM. σ is the sigmoid function. Represents the dot product function. i t represents the input gate, which determines the information storage at time t. t Represents the forget gate, which determines the information to be discarded at the previous moment. t Represents the output gate, which determines the output of information. and C t Represent the candidate cell state and the cell state at time t respectively. The calculation process of BiLSTM is as follows:

[0066]

[0067]

[0068]

[0069] in and denote the forward LSTM and the backward LSTM respectively. The final implicit representation of the input sequence is expressed as where d LSTM Represents the hidden size of the LSTM.

[0070] In the named entity recognition task, there are sequential relationships and some constraints between adjacent labels, for example, the I-symptom label must appear after the B-symptom label. BiLSTM focuses on long-term contextual information rather than the dependency between labels, while CRF can simulate the sequential relationship between labels by learning adjacent relationships. In the present invention, the standard CRF layer is used to decode the final sequence label.

[0071] First, the output H of BiLSTM is converted into the input matrix P of CRF through a linear function.

[0072] P=W p H+b p

[0073] in is a learnable parameter, and k is the number of label types. The purpose of CRF is to calculate the random variable sequence Y = {y1, y2, ..., y n}. In a given sentence X = {x1, x2, ..., xn}, the probability Y of the final optimal label sequence can be calculated as:

[0074]

[0075]

[0076] in Y represents the ground truth label sequence. X Represents the set of all possible tag sequences. Indicates that from label y i to label y i+1 The transition probability matrix A is a learnable model parameter. Indicates that the i-th token is mapped to the named entity label y i The unnormalized probability of . During model training, the loss function is defined as:

[0077]

[0078] After decoding, the output label sequence Y with the highest score * The calculation method is:

[0079]

[0080] Due to privacy, ethical and highly specialized restrictions, annotated Chinese medical NER data is difficult to obtain and is usually small in scale. To this end, the BBCPR model introduces a novel REgularization and Adversarial training and Dropout (READ) method to improve the robustness of the model. The READ method uses a regularization module with adversarial training and dropout. The regularization module uses FGSM to generate adversarial word embedding E' through adversarial perturbations on word embedding E. The word embedding E and adversarial word embedding E' are respectively generated into two different sub-modules through the dropout mechanism, and two different model prediction distributions are output. Then, the bidirectional KL divergence between the two prediction distributions is minimized to reduce the prediction difference between the two sub-models. The structure of READ is as follows Figure 3 shown.

[0081] For a given sentence, the predicted sentence X = {x1, x2, ..., x n} and the output sequence Y = {y1,y2,...,y n}, first use FGSM to generate adversarial word embedding E' by generating adversarial perturbations on the original word embedding E, and use the dropout mechanism to get two different sub-models. Then the original word embedding E and adversarial word embedding E' are respectively fed into the above two sub-models to output two different model prediction distributions, denoted as P(Y|X) and P'(Y|X).

[0082] Since adversarial perturbations and dropout noise will make the representation far away from the representation of the original sentence. In this training step, the READ method focuses on reducing the prediction difference between the two sub-models by minimizing the bidirectional KL divergence between the two output distributions. Formally, this process is expressed as:

[0083]

[0084] Basic learning objective function with two forward passes It can be expressed as:

[0085]

[0086] The final training goal is to make the data (X, Y) Minimize:

[0087]

[0088] Among them, λ is the coefficient weight used to balance the two training losses, D KL is the KL divergence.

[0089] When a neural network is trained on a small training set, it usually performs poorly on the test data. Dropout is widely used to regularize fully connected neural network layers due to its simplicity and efficiency.

[0090] In terms of model robustness, the Fast Gradient Signed Method (FGSM) is a popular method for generating adversarial samples to make neural network models robust to perturbations. The basic principle is to add interference during model training to construct adversarial samples, thereby improving the robustness of the model when encountering adversarial samples.

[0091] The present invention designs a new regularization module, combines R-Drop and FGSM to deal with the overfitting problem and improve the robustness of the model. On the basis of generating adversarial samples, two drops are performed, and the distance between the two sub-models is shortened by KL clustering. Clustering is used to shorten the distance between the two sub-models.

[0092] At the same time, in order to better identify named entities, many works have introduced the integration of external knowledge. This application uses a pre-trained BERT model to dynamically generate word vector sequences in a large-scale Chinese media corpus, and designs a POS fusion layer to incorporate the POS tags of words, which can be used as a supervisory signal to solve the boundary problem of entity annotations. Then, a BiLSTM layer is used to obtain the positional features of each word. Finally, CRF is used as a decoder to obtain the final predicted label.

[0093] In order to demonstrate the advantages of the model proposed in the present invention in CNER in the medical field, the present invention conducted comprehensive experiments on two datasets, cMedQANER and cEHRNER.

[0094] The cMedQANER and cEHRNER datasets are derived from Chinese community question answering and annotations of Chinese electronic health records, respectively. The cMedQANER dataset contains 2,063 annotated instances, including 11 types of medical named entities, including body, population, department, disease, drug, feature, physiology, symptom, test, time, and treatment. The annotated instances have been divided into 1,673 training instances, 175 training instances, and 215 test instances. The cEHRNER dataset contains 999 annotated samples with seven types of medical named entities, including disease and diagnosis, operation, anatomical part, drug, symptom, imaging examination, and laboratory test. The annotated samples have been divided into 914 training samples, 44 training samples, and 41 test samples. Table 3 lists the statistics of the cMedQANER and cEHRNER datasets. Tables 4 and 5 list the statistics of different types of entities on the cMedQANER and cEHRNER datasets, respectively. To measure the performance of CNER, we use precision (P), recall (R), and F1 score, which are widely used indicators for sequence labeling tasks. The specific formula is as follows:

[0095]

[0096]

[0097]

[0098] Table 3 Statistics of cMedQANER and cEHRNER datasets

[0099]

[0100] Table 4 Statistics of different types of entities in the cMedQANER dataset

[0101]

[0102] Table 5 Statistics of different types of entities on the cEHRNER dataset

[0103]

[0104] Baidu LAC tool was used to obtain POS tags for cMedQANER and cEHRNER datasets. The pre-trained language model MCBERT was selected as the context embedding layer in the experiment, with 12 layers, 12 self-attention heads, and a hidden size of 768. The feedforward network dimension and BiLSTM hidden size were set to 1024 and 256, respectively. POS embeddings were randomly initialized from a standard normal distribution, and the embedding size was set to 512. All datasets were trained with a batch size of 32 and a maximum sequence length of 256. During training, the AdamW optimizer was used with β1=0.9, β2=0.998, a linear learning rate decay schedule, and a weight decay of 0.01. The learning rate of MCBERT and BiLSTM was set to 7e-5. The learning rate of CRF was set to 5e-3. The dropout rate and regularization loss weight λ were set to 0.2 and 2.0, respectively. The models were trained 50 times, and the best model was used to predict the test dataset. Its hyperparameter configuration is shown in Table 6.

[0105] Table 6 Hyperparameter configuration of the method of the present invention

[0106]

[0107] The proposed model is compared with several CMNER methods, HMM, BiLSTM without CRF layer, BiLSTM-CRF (neural architecture for named entity recognition), MCBERT, MC-BERT-CRF. In addition, word2vec embedding is used for BiLSTM and BiLSTM-CRF models. MCBERT is a pre-trained language model that is further trained on the Chinese Biomedical Corpus. Table 7 shows the performance of these models in terms of precision, recall, and F1 score on the cMedQANER and cEHRNER datasets. The traditional statistical learning method HMM achieves poor results. In addition, the additional CRF layer can significantly improve the performance of the model because CRF can model the dependencies between label sequences. In addition, the results show that MC-BERT can capture better contextual representations compared to word2vec. Moreover, we can see that our method achieves the best results, with an F1 score improvement of 2.08 and 2.35 on the two datasets, respectively. This proves the effectiveness of our method. The improvement of our method on the smaller dataset cEHRNER exceeds the improvement on the large dataset cMedQANER. This shows that our method can improve the robustness of Chinese medical named entity recognition models.

[0108] Table 7. Comparison of model performance on cMedQANER and cEHRNER datasets.

[0109]

[0110] In order to evaluate the effects of two important components in our model, namely the POS fusion layer and the regularization method, an ablation study is performed on the cMedQANER and cEHRNER test sets. The results are shown in Table 8. [AP] and READ [DP]Denote adversarial perturbations and drop-off perturbations in READ, respectively. In general, we can see that when certain modules are removed, the performance of the model decreases significantly in all indicators, which can verify the effectiveness of our proposed method. For example, on the cMedQANER and cEHRNER datasets, after removing the READ method, the model performance decreases the most, with an absolute decrease of 1.10% and 1.50% in F1 score, 1.72% and 3.10% in P score, and 0.47% in R score. This is because READ can improve the robustness of the model by regularizing two distributions of the same sample, namely the original distribution and the distribution interfered by adversarial perturbations (AP) and drop-off perturbations (DP). In addition to exploring the effectiveness of READ, we also investigate the contribution of AP and DP alone. On the cMedQANER dataset, removing AP and DP leads to an absolute decrease of 0.61% and 0.72% in F1 score, respectively. Similar results can be found on the cEHRNER dataset, where the model presents an absolute decrease of 0.52% and 0.78%. As for P and R, removing AP results in an absolute drop of 0.22%-0.80% and 0.24%-1.00%, while removing DP results in an absolute drop of 1.11%-1.51% and 0.31%. The reason why DP performs better may be that it is applied to the entire model, while AP only works on BERT. In addition, we can also observe that the combination of adversarial perturbations and dropout perturbations outperforms either one alone. Because each perturbation is relatively simple, adopting only one method will cause a small degree of perturbation. Diverse perturbations can increase the dissimilarity of representations of the same sample. As for the POS fusion layer, on the cMedQANER and cEHRNER datasets, absolutely, after removing this module, the F1 score dropped by 0.70% and 0.66%, the P score dropped by 0.12% and 1.22%, and the R score dropped by 1.28% and 0.07%. This shows that incorporating POS tag features into deep neural networks can significantly improve the performance of Chinese-MNER models because POS tag features add additional latent entity boundary information.

[0111] Table 8 Performance of P, R and F1 without different modules on cMedQANER and cEHRNER datasets

[0112]

[0113] We further conduct comparative experiments to analyze the impact of different pre-trained models. BERT-wwm uses a full-word masking strategy for Chinese text. RoBERTa is a robust and optimized BERT pre-training method. MacBERT masks the word with similar words in Chinese text. MCBERT is a pre-trained language model that is further trained on a Chinese biomedical corpus. Figure 4 As shown in the figure, (a) is the cMedQANER set and (b) is the cEHRNER set. MC-BERT outperforms other pre-trained models. Since MC-BERT adapts to the entire entity masking strategy and the entire span masking strategy, it injects medical domain knowledge to obtain better contextual representation of Chinese biomedical text. We choose the MCBERT model as the context embedding layer in the following experiments.

[0114] We further conduct experiments on the cMedQANER and cEHRNER test sets to explore the relationship between the performance of our model and the size of POS embedding. In our experiments, the size of POS embedding is set to 128, 256, 384, 512, 640, and 768 respectively. The results are shown in Figure 5 In the figure, (a) is the cMedQANER set and (b) is the cEHRNER set. When the size of POS embedding increases, the performance of our model initially increases because increasing the size of the neural network can increase the complexity of the model to obtain more powerful representation capabilities. However, when the scale increases further, the performance deteriorates due to model overfitting.

[0115] We further study the effect of the regularized loss weight λ. Here we vary λ in {1, 2, 3, 4, 5, 10} and conduct experiments. Figure 6 As shown in the figure, (a) is the cMedQANER set, (b) is the cEHRNER set. Too small or too large λ will make our model perform poorly. When λ = 2, the model achieves the best performance. We choose 2 as the regularization loss weight in the experiment.

[0116] During training, the same value (0.1) is set for both distributions. Therefore, the effect of the dropout rate value is further investigated. We vary the dropout rate in 0.05, 0.1, 0.2, 0.3, 0.4, 0.5 and conduct experiments. Figure 7 As shown in the figure, (a) is the cMedQANER set and (b) is the cEHRNER set. It can be found that the same dropout rate (0.3, 0.3) is the best choice.

[0117] According to the embodiments of the present invention as described above, these embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. According to the above description, many modifications and changes can be made. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can make good use of the present invention and the modified use based on the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. A method for recognizing Chinese named entities in the medical field, characterized in that: The method uses the BBCPR model to identify Chinese named entities in the medical field. The BBCPR model consists of a word embedding layer, a BERT embedding layer, a POS fusion layer, a BiLSTM layer, and a CRF layer. The word embedding layer converts the given sentence into a word embedding E and inputs it into the BERT embedding layer; The BERT embedding layer uses the MCBERT encoder to get the BERT output embedding, which is used as the input of the POS fusion layer; The POS fusion layer concatenates the BERT output embedding and the POS embedding to obtain a fused embedding, where the POS embedding is obtained by passing the POS tag of a given sentence through the POS embedding layer; The BiLSTM layer encodes the fused embedding to obtain the final implicit representation of the input sequence; The CRF layer decodes the output of the BiLSTM layer to obtain a label sequence and output it; The BBCPR model uses a regularization module with adversarial training and abandonment. The regularization module uses FGSM to generate adversarial word embedding E' through adversarial perturbations on word embedding E. The word embedding E and adversarial word embedding E' are respectively generated into two different sub-modules through the Dropout mechanism, and two different model prediction distributions are output. Then, the bidirectional KL divergence between the two prediction distributions is minimized to reduce the prediction difference between the two sub-models.

2. The identification method according to claim 1, characterized in that: The bidirectional KL divergence between the two prediction distributions is minimized to reduce the prediction difference between the two sub-models, specifically: The training goal is to make the loss function of the data (X, Y) Minimize: Where λ is the coefficient weight, P(Y|X) is the probability distribution of word embedding E, P′(Y|X) is the probability distribution of adversarial word embedding E′, and D KL is the KL divergence, X is the predicted sentence, and Y is the label sequence output by the BBCPR model.

3. The identification method according to claim 1, characterized in that: The POS fusion layer concatenates the BERT output embedding and the POS embedding, specifically: where v i is the concatenation of BERT output embedding and POS embedding, h i is the output embedding of BERT, p i is the POS embedding of the i-th token in the sentence.

4. The identification method according to claim 3, characterized in that: The given sentence obtains POS tags through the LAC tool, and the POS tags are sent to the POS embedding layer to obtain POS embedding.

5. The identification method according to any one of claims 1 to 4, characterized in that: Predict each token in the sentence using BIO tags.

6. The identification method according to any one of claims 1 to 4, characterized in that: MCBERT consists of a stack of L identical layers, each of which contains two sub-layers, where the first sub-layer is a multi-head self-attention mechanism and the second sub-layer is a fully connected feed-forward neural network, and the two sub-layers are connected in sequence through residual connections and layer normalization.

7. The identification method according to any one of claims 1 to 4, characterized in that: The calculation process of BiLSTM is: in and They represent forward LSTM and backward LSTM respectively.

Citation Information

Patent Citations

  • Chinese named entity recognition model and method based on double neural network fusion

    CN111783462A

  • Coreference-aware representation learning for neural named entity recognition

    US20210034701A1

Cited By

  • Method of training POS tagging model, computer-readable recording medium and POS tagging method

    US12585876B2