Medical text named entity recognition method and system

Abstract

The invention belongs to the technical field of electronic medical records, and particularly discloses a medical text named entity recognition method and system.The method comprises the steps that an original medical text is collected and serves as input of a DLBC, a pre-training language model serves as an embedded layer of the DLBC to conduct word segmentation processing on the original medical text, the text is converted into a character sequence, and the character sequence is stored in a database; sending the character sequence into a pre-training language model to generate a corresponding character vector representation; extracting medical nouns and categories by using a corpus, and constructing a domain dictionary containing multiple categories of medical nouns; using a multi-head attention mechanism to obtain a feature sequence; performing feature extraction and context information modeling on the feature sequence by using a bidirectional long short-term memory network Bi LSTM; and the output of the Bi LSTM is converted into NER label probability distribution corresponding to each character by using a linear layer, and a final recognition result is obtained by applying a joint decoding strategy CRF. By adopting the technical scheme, entity embedding enhancement is carried out by utilizing the external information provided by the domain dictionary, and the medical entity recognition accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of electronic medical records and relates to a method and system for identifying named entities in medical texts. Background Art

[0002] Electronic medical records (EMRs) are an important source of information in the healthcare field, designed to provide patients' medical information to support medical decision-making and treatment processes. They contain a wealth of medical expertise, but most of them exist in the form of unstructured data, which is difficult to extract and use automatically.

[0003] Information extraction is a key step in natural language processing, which mainly includes named entity recognition and relation extraction. The main task of named entity recognition is to identify entities with specific meanings in sentences and classify them according to given categories.

[0004] Under the current background of hospital informatization and rapid development of medical information, a huge amount of electronic medical records will be generated every day. These medical records contain a lot of potential knowledge (including demographics, clinical data, laboratory and examination data, follow-up and prognosis, etc.), making electronic medical records an important source for analyzing and studying health information. Information extraction from these data can be better applied to downstream tasks such as the construction of medical knowledge graphs and the development of evidence-based medicine. It can provide accurate decision support for clinical medical staff and provide patients with better quality and higher level of medical services. However, most of these data are currently stored in the form of semi-structured or unstructured text, and computers cannot directly process and use them. How to efficiently and accurately extract medical knowledge from electronic medical records is a problem that needs to be solved at present.

[0005] The named entity recognition task has gradually evolved from early rule-based and template-based development to shallow machine learning, and then to the now widely used deep learning-based methods. Commonly used named entity recognition models based on deep learning methods can already achieve good recognition results in general fields.

[0006] Unlike general fields, medical texts contain a large number of professional terms and abbreviations, and there are also the effects of nested entities. Named entity recognition requires a high degree of reliance on context, and previous rule-based and template-based, statistical, and machine learning-based named entity recognition models have difficulty capturing long-range dependencies in texts. Summary of the invention

[0007] The purpose of the present invention is to provide a method and system for named entity recognition in medical texts, so as to improve the recognition accuracy of named entity recognition tasks in medical texts.

[0008] In order to achieve the above object, the basic scheme of the present invention is: a method for named entity recognition in medical text, comprising the following steps:

[0009] Collecting original medical texts;

[0010] Set up a medical text named entity recognition model DLBC, and use the original medical text as the input of DLBC. Use the Chinese-BERT-wwm-ext pre-trained language model as the embedding layer of DLBC. First, perform word segmentation on the original medical text, convert the text into a character sequence, and then send the character sequence to the Chinese-BERT-wwm-ext pre-trained language model to generate the corresponding character vector representation.

[0011] Use the corpus to extract medical terms and categories, remove duplicates and classify the extracted medical terms, and build a domain dictionary containing multi-category medical terms; use the multi-head attention mechanism to dynamically adjust the attention of the corresponding dictionary embedding vector and the character vector generated by the pre-trained language model according to their importance, realize dynamic feature fusion, and obtain a feature sequence;

[0012] Use the bidirectional long short-term memory network BiLSTM to extract features from feature sequences and model context information;

[0013] A linear layer is used to convert the output of BiLSTM into the NER label probability distribution corresponding to each character through the weight matrix, and the joint decoding strategy CRF is applied to obtain the final recognition result.

[0014] The working principle and beneficial effects of this basic solution are as follows: This technical solution reduces the adverse effects of unregistered words on model recognition performance by constructing a rich domain dictionary, which can not only ensure accurate recognition of known words, but also effectively deal with the negative effects of unregistered words.

[0015] Pre-trained language models that use characters as segmentation units perform well in entity recognition tasks, but they usually ignore coarse-grained domain vocabulary as a representation of the overall semantic unit, and use a multi-head attention mechanism to fuse entity vectors instead of simply concatenating or adding them. The multi-head attention mechanism can flexibly combine the contextual information provided by the pre-trained language model and the entity knowledge in the domain dictionary, making the model more accurate and efficient when processing complex medical texts.

[0016] When an entity in the dictionary is identified, ensure that the corresponding category label in the dictionary is used for the final prediction, thereby improving the recognition accuracy and consistency of the model.

[0017] Furthermore, the method of dynamically adjusting the attention of the corresponding dictionary embedding vector and the character vector generated by the pre-trained language model according to their importance is:

[0018] For each character vector, the multi-head attention mechanism calculates its similarity with the dictionary embedding vector using cosine similarity;

[0019] Generate attention weights based on similarity. The higher the similarity, the greater the attention weight. Then normalize the attention weights to get probability distribution.

[0020] The probability distribution is applied to the dictionary embedding vector to dynamically adjust the emphasis on different features and generate a fused feature sequence.

[0021] The multi-head attention mechanism dynamically adjusts the weights according to the importance of different features to achieve effective feature fusion.

[0022] Furthermore, the bidirectional long short-term memory network BiLSTM is used to extract features from feature sequences and model context information. The specific method is as follows:

[0023] By combining two LSTMs, the forward and reverse sequence data are processed separately, and then the outputs of the two directions are spliced:

[0024]

[0025]

[0026]

[0027] Among them, t represents the current time; Represents the calculation process of forward LSTM, Represents the calculation process of backward LSTM, is the hidden state of the forward LSTM at the previous moment, is the hidden state of the backward LSTM at the previous moment; Indicates the splicing operation; x t represents the input of the current time step, h t Used to calculate the hidden state at the current moment.

[0028] By combining two LSTMs, the forward and reverse sequence data are processed separately, and the outputs in the two directions are concatenated, so that the output of each time step contains both the forward and reverse hidden states of the current point.

[0029] Furthermore, the conditional random field CRF is introduced after the linear layer. Suppose the input sequence X = (x 1 ,x 2 ,…,x n ) and the corresponding label sequence Y = (y 1 ,y2 ,…,y n ), the goal of CRF is to maximize the conditional probability P(Y|X) of the label sequence Y given the input sequence X:

[0030]

[0031] Where Z(X) is a normalization factor to ensure that the sum of probabilities is 1; t j (y i-1 ,y i ,X,i) is the transfer feature function, describing the transition from label y i-1 to label y i The transition probability depends on the current position i and the input sequence X; j is the transfer characteristic function t j The weight of k (y i ,X,i) is the state feature function, describing the label y i The state probability at position i also depends on the input sequence X; μ k is the state characteristic function s k The weight of .

[0032] Conditional Random Field (CRF) is a sequence labeling model that is widely used in named entity tasks. It can accurately model the transition probability between input labels after fully considering contextual information and output the conditional distribution of entity category labels.

[0033] The present invention also provides a medical text named entity recognition system, comprising a data acquisition unit and a processing unit, wherein the data acquisition unit is used to adopt the original medical text, and the output end of the data acquisition unit is connected to the input end of the processing unit;

[0034] The processing unit executes the method of the present invention to perform named entity recognition in medical texts.

[0035] The system utilizes a data acquisition unit and a processing unit to perform named entity recognition in medical texts, thereby improving the recognition accuracy of named entity recognition tasks in medical texts.

[0036] Furthermore, a medical text named entity recognition model DLBC is provided in the processing unit, and the medical text named entity recognition model DLBC includes an input layer, a Chinese-BERT-wwm-ext pre-trained language module, a multi-head attention mechanism module, a bidirectional long short-term memory network BiLSTM module, a linear layer and a decoding layer connected in sequence;

[0037] The input layer receives the original medical text and generates a character sequence after word segmentation;

[0038] The linear layer maps the output of the bidirectional long short-term memory network BiLSTM module to the label space to generate a label probability distribution for each character;

[0039] The decoding layer uses the conditional random field (CRF) to jointly decode the label probability distribution and output the final recognition result.

[0040] The medical text named entity recognition model DLBC uses domain dictionaries to enhance entity embedding and integrates word information into the character-based named entity recognition model, which can effectively improve the model's ability to deeply understand medical semantics. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the process of the medical text named entity recognition method of the present invention;

[0042] Figure 2 It is a structural schematic diagram of a medical text named entity recognition model DLBC of the medical text named entity recognition system of the present invention. DETAILED DESCRIPTION

[0043] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0044] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0045] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0046] The present invention discloses a method for named entity recognition in medical texts, which uses a domain dictionary to perform entity embedding enhancement and improve the recognition accuracy of named entity recognition tasks in medical texts. Figure 1 As shown, the medical text named entity recognition method includes the following steps:

[0047] Collecting original medical texts;

[0048] A medical text named entity recognition model DLBC (DomainLexicon-Chinese-BERT-wwm-ext-BiLSTM-CRF) is set up, and the original medical text is used as the input of DLBC. The Chinese-BERT-wwm-ext pre-trained language model is used as the embedding layer of DLBC. The original medical text is first segmented and converted into a character sequence. The character sequence is then sent to the Chinese-BERT-wwm-ext pre-trained language model to generate the corresponding character vector representation. The pre-trained language model Chinese-BERT-wwm-ext is introduced into the model as a feature extractor. By using its ability to understand and capture text context information, it can provide more meaningful input for BiLSTM.

[0049] Use the corpus to extract medical terms and categories, remove duplicates and classify the extracted medical terms, and build a domain dictionary containing multi-category medical terms; use the multi-head attention mechanism to dynamically adjust the attention of the corresponding dictionary embedding vector and the character vector generated by the pre-trained language model according to their importance, realize dynamic feature fusion, and obtain a feature sequence;

[0050] Use the bidirectional long short-term memory network BiLSTM to extract features from feature sequences and model context information;

[0051] Use a linear layer to convert the output of BiLSTM into the NER label probability distribution corresponding to each character through the weight matrix (in the named entity recognition task, BiLSTM is first used to extract context features, and then the output of BiLSTM is mapped to the label space through the weight matrix to generate the label probability distribution of each character. Finally, CRF is used to jointly decode these labels, and the dependencies between the labels are considered to maximize the conditional probability of the entire sequence to obtain the final prediction result.), and the joint decoding strategy CRF is applied to obtain the final recognition result.

[0052] Preferably, Chinese-BERT-wwm-ext is a BERT model that supports Chinese. It uses more extended data for pre-training based on the Chinese-BERT-wwm model. In training, the Whole Word Masking method is used to enable the model to better capture the contextual information of the whole word rather than a single character during training. By expanding the corpus and training with a stratified sampling method, the differences in sentence length and difficulty are effectively considered, and the model performance is further improved by combining knowledge distillation. At the same time, the multi-window mechanism and self-attention mechanism are used in the model structure to further improve the performance of the model.

[0053] Chinese-BERT (Bidirectional Encoder Representation from Transformers)-wwm-ext uses extended data for pre-training based on the Chinese-BERT-wwm model. In the training of the pre-trained language model, the whole word masking method is used to mask the entire word to ensure that the model can capture the complete vocabulary context during the learning process, enhance the pre-trained language model's ability to understand complete vocabulary, and improve the processing effect of polysemous words and phrases; by expanding the corpus and adopting a stratified sampling method for training, considering the differences in sentence length and difficulty, combined with knowledge distillation, the performance of the DLBC model is improved.

[0054] Since the pre-trained language model has been unsupervisedly pre-trained on a large-scale corpus and has learned rich semantic information and contextual relationships, it can reduce the need for a large amount of domain-specific corpus, save training time and resources, and enhance the generalization ability of the model.

[0055] In a preferred embodiment of the present invention, a method for dynamically adjusting the attention of the corresponding dictionary embedding vector and the character vector generated by the pre-trained language model according to their importance is:

[0056] For each character vector, the multi-head attention mechanism calculates its similarity with the dictionary embedding vector using cosine similarity;

[0057] Generate attention weights based on similarity. Similarity and attention weights are positively correlated: the higher the similarity, the greater the attention weight. Then, the attention weights are normalized to obtain a probability distribution, so that the model can focus more on features that better match the input information, thereby improving the context understanding effect.

[0058] The probability distribution is applied to the dictionary embedding vector to dynamically adjust the emphasis on different features and generate a fused feature sequence.

[0059] In a preferred embodiment of the present invention, a bidirectional long short-term memory network BiLSTM (Bi-directional Long Short-Term Memory, BiLSTM) is used to extract features from feature sequences and model context information. The specific method is as follows:

[0060] By combining two LSTMs, the forward and reverse sequence data are processed separately, and then the outputs of the two directions are spliced:

[0061]

[0062]

[0063]

[0064] Among them, t represents the current time; Represents the calculation process of forward LSTM, Represents the calculation process of backward LSTM, is the hidden state of the forward LSTM at the previous moment, is the hidden state of the backward LSTM at the previous moment; Indicates the splicing operation; x t represents the input of the current time step, h t Used to calculate the hidden state at the current moment.

[0065] The output of each time step contains both the forward and reverse hidden states of the current point. BiLSTM can capture the forward and backward information of sequence data by extracting features and modeling context information of the input sequence, and has a strong ability to model context information in text data.

[0066] In a preferred embodiment of the present invention, the conditional random field (CRF) is a sequence labeling model that is widely used in named entity tasks. It can accurately model the transition probability between input labels and output the conditional distribution of entity category labels after fully considering the context information.

[0067] After the linear layer, the conditional random field CRF (CRF) is introduced. Suppose the input sequence X = (x 1 ,x 2 ,…,x n ) and the corresponding label sequence Y = (y 1 ,y 2 ,…,y n ), the goal of CRF is to maximize the conditional probability P(Y|X) of the label sequence Y given the input sequence X:

[0068]

[0069] Where Z(X) is a normalization factor to ensure that the sum of probabilities is 1; t j (y i-1 ,y i ,X,i) is the transfer feature function, describing the transition from label y i-1 to label y i The transition probability depends on the current position i and the input sequence X; j is the transfer characteristic function t j The weight of k (y i ,X,i) is the state feature function, describing the label y i The state probability at position i also depends on the input sequence X; μ k is the state characteristic function s k The weight of .

[0070] CRF learns the weights of these feature functions through training data, so that the model can output the most likely label sequence under a given input sequence. At the same time, CRF also effectively constrains the annotations, taking into account the relationship between labels to ensure that the output label sequence is reasonable and consistent. For example, an entity can only start with B-type or O, but not with I-type. Compared with the Softmax function that predicts each label independently, CRF can solve the dependency between entity labels through global optimization and achieve more accurate predictions.

[0071] The present invention also provides a medical text named entity recognition system, including a data acquisition unit and a processing unit. The data acquisition unit is used to collect original medical texts. The output end of the data acquisition unit is electrically connected to the input end of the processing unit for data transmission.

[0072] The processing unit executes the method of the present invention to perform named entity recognition in medical texts. The system utilizes the data acquisition unit and the processing unit to perform named entity recognition in medical texts, thereby improving the recognition accuracy of named entity recognition tasks in medical texts.

[0073] Preferably, Figure 2 As shown, a medical text named entity recognition model DLBC is provided in the processing unit, and the medical text named entity recognition model DLBC includes an input layer, a Chinese-BERT-wwm-ext pre-trained language module, a multi-head attention mechanism module, a bidirectional long short-term memory network BiLSTM module, a linear layer and a decoding layer connected in sequence.

[0074] The input layer receives the original medical text and generates character sequences after word segmentation.

[0075] The Chinese-BERT-wwm-ext pre-trained language module and the multi-head attention mechanism module, namely the embedding layer, use the Chinese-BERT-wwm-ext pre-trained language model to generate character vectors, and the domain dictionary generates domain vectors through dictionary embedding as the initial representation of the text. Through the multi-head attention mechanism, the dictionary embedding vector is fused with the character vector generated by the pre-trained language model. The multi-head attention mechanism dynamically adjusts the weights according to the importance of different features to achieve effective feature fusion.

[0076] The BiLSTM module, i.e., the feature extraction layer, uses a BiLSTM network to model feature sequences and extract contextual information.

[0077] The linear layer maps the output of the BiLSTM module to the label space and generates a label probability distribution for each character.

[0078] The decoding layer uses the conditional random field CRF to jointly decode the label probability distribution and output the final recognition result.

[0079] By constructing a domain dictionary as an external feature, word vector conversion is performed after the model is introduced. The output vector of the pre-trained language model is spliced ​​through the attention mechanism, and the model's attention is dynamically adjusted according to the importance of the word, realizing the fusion of semantic information of different granularities, that is, the fusion of character information and word information in the text. Integrating word information into the character-based named entity recognition model can effectively improve the model's ability to deeply understand medical semantics. The comparative experimental results show that the proposed method has better recognition effect than the baseline model.

[0080] The present invention reduces the adverse effects of unknown words on model recognition performance by constructing a rich domain dictionary, which can ensure accurate recognition of known words and effectively deal with the negative effects of unknown words.

[0081] Pre-trained language models that use characters as segmentation units perform well in entity recognition tasks, but they usually ignore coarse-grained domain vocabulary as a representation of the overall semantic unit. To solve this problem, the present invention uses a multi-head attention mechanism to fuse entity vectors instead of simply concatenating or adding them. The multi-head attention mechanism can flexibly combine the contextual information provided by the pre-trained language model and the entity knowledge in the domain dictionary, making the model more accurate and efficient when processing complex medical texts.

[0082] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0083] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for named entity recognition in medical text, characterized in that: The steps include: Collecting original medical texts; Set up a medical text named entity recognition model DLBC, and use the original medical text as the input of DLBC. Use the Chinese-BERT-wwm-ext pre-trained language model as the embedding layer of DLBC. First, perform word segmentation on the original medical text, convert the text into a character sequence, and then send the character sequence to the Chinese-BERT-wwm-ext pre-trained language model to generate the corresponding character vector representation. Use the corpus to extract medical terms and categories, remove duplicates and classify the extracted medical terms, and build a domain dictionary containing multi-category medical terms; Using the multi-head attention mechanism, the corresponding dictionary embedding vector and the character vector generated by the pre-trained language model are dynamically adjusted according to their importance to achieve dynamic feature fusion and obtain a feature sequence; Use the bidirectional long short-term memory network BiLSTM to extract features from feature sequences and model context information; A linear layer is used to convert the output of BiLSTM into the NER label probability distribution corresponding to each character through the weight matrix, and the joint decoding strategy CRF is applied to obtain the final recognition result.

2. The method for named entity recognition in medical text according to claim 1, characterized in that: The method of dynamically adjusting the attention of the corresponding dictionary embedding vector and the character vector generated by the pre-trained language model according to their importance is: For each character vector, the multi-head attention mechanism calculates its similarity with the dictionary embedding vector using cosine similarity; Generate attention weights based on similarity. The higher the similarity, the greater the attention weight. Then normalize the attention weights to get probability distribution. The probability distribution is applied to the dictionary embedding vector to dynamically adjust the emphasis on different features and generate a fused feature sequence.

3. The method for named entity recognition in medical text according to claim 1, characterized in that: The bidirectional long short-term memory network BiLSTM is used to extract features from feature sequences and model context information. The specific method is as follows: By combining two LSTMs, the forward and reverse sequence data are processed separately, and then the outputs of the two directions are spliced: Among them, t represents the current time; Represents the calculation process of forward LSTM, Represents the calculation process of backward LSTM, is the hidden state of the forward LSTM at the previous moment, is the hidden state of the backward LSTM at the previous moment; Indicates the splicing operation; x t represents the input of the current time step, h t Used to calculate the hidden state at the current moment.

4. The method for named entity recognition in medical text according to claim 1, characterized in that: Conditional random field CRF is introduced after the linear layer. Suppose the input sequence X = (x1, x2, ..., x n ) and the corresponding label sequence Y = (y1, y2, ..., y n ), the goal of CRF is to maximize the conditional probability P(Y|X) of the label sequence Y given the input sequence X: Where Z(X) is a normalization factor to ensure that the sum of probabilities is 1; t j (y i-1 ,y i ,X,i) is the transfer feature function, describing the transition from label y i-1 to label y i The transition probability depends on the current position i and the input sequence X; j is the transfer characteristic function t j The weight of k (y i ,X,i) is the state feature function, describing the label y i The state probability at position i also depends on the input sequence X; μ k is the state characteristic function s k The weight of .

5. A medical text named entity recognition system, characterized in that: It includes a data acquisition unit and a processing unit, wherein the data acquisition unit is used to adopt the original medical text, and the output end of the data acquisition unit is connected to the input end of the processing unit; The processing unit executes the method described in any one of claims 1-4 to perform named entity recognition in medical texts.

6. The medical text named entity recognition system according to claim 5, characterized in that: The processing unit is provided with a medical text named entity recognition model DLBC, and the medical text named entity recognition model DLBC includes an input layer, a Chinese-BERT-wwm-ext pre-trained language module, a multi-head attention mechanism module, a bidirectional long short-term memory network BiLSTM module, a linear layer and a decoding layer connected in sequence; The input layer receives the original medical text and generates a character sequence after word segmentation; The linear layer maps the output of the bidirectional long short-term memory network BiLSTM module to the label space to generate a label probability distribution for each character; The decoding layer uses the conditional random field (CRF) to jointly decode the label probability distribution and output the final recognition result.

Images