Chinese electronic medical record named entity identification method based on multiple attention mechanisms and SNN network

Abstract

The invention provides a Chinese electronic medical record named entity recognition method based on a multi-head attention mechanism and a pulse neural network (SNN), and the method comprises the steps: obtaining a dynamic word vector through a pre-training BERT model, obtaining a text code through a bidirectional long short-term memory network (BiLSTM) model, and screening the features of the code through the SNN network for a named entity in an electronic medical record; the multi-head attention mechanism calculates the dependency relationship between characters, enhances the relevance to the context, and finally outputs a final sequence through a conditional random field (CRF) module to obtain an entity recognition result. According to the method, the time cost is remarkably reduced, the efficiency and precision of identifying the named entities in the medical text are improved, and a more efficient and accurate technical solution is provided for the field of medical information processing.

Description

Technical Field

[0001] The present invention relates to the technical field of computer natural language processing, and in particular to a Chinese electronic medical record named entity recognition method based on a multi-head attention mechanism and an SNN network. Background Art

[0002] In the field of natural language processing, named entity recognition (NER) is an important technology for information extraction. Its main goal is to identify specific types of entities from text, such as time, place, name, date, and organization name. The key challenge of this task is to accurately determine the boundaries of entities in the text and accurately classify their types. The development of NER technology is not only of great significance to subsequent tasks such as information extraction, question-answering systems, and machine translation, but also plays a key role in academic research and commercial applications. Accurately identifying entities can not only help the system better understand the text, but also provide important semantic information for various natural language processing tasks. Therefore, the continuous improvement and application of NER technology will bring more innovation and progress to the field of natural language processing.

[0003] In the medical field, named entity recognition plays a vital role. The main goal of medical named entity recognition is to extract pre-defined professional medical terms such as diseases, drugs, diagnosis types, and imaging examinations from unstructured medical texts. Accurately identifying and extracting these entities is of great significance for downstream work, because high-accuracy entity recognition can improve the efficiency and quality of information processing in the medical field. In the medical field, accurate named entity recognition can not only help doctors quickly obtain key information, but also provide researchers with valuable medical data support. Therefore, the continuous optimization and application of medical named entity recognition technology will have a positive impact on medical information extraction, diagnostic assistance systems, and medical data analysis.

[0004] Although many models have emerged in current research and have achieved some results, most of them have certain deficiencies in detecting entity boundaries. Their word granularity is relatively coarse, and they tend to ignore the local features of entities and have poor robustness. To address this problem, a new model for named entity recognition in Chinese medical texts is proposed, which combines a multi-head attention mechanism and a spiking neural network (SNN). By deeply and meticulously mining the semantic features of the text, it aims to improve the accuracy and robustness of entity recognition. This method aims to fill the gaps in existing models and provide new ideas and methods for in-depth research and application of named entity recognition tasks in Chinese medical texts. Through this model, entity information can be captured more comprehensively, thereby improving the effectiveness of medical text entity recognition and laying a foundation for further research and application of entity recognition. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for named entity recognition in Chinese electronic medical records based on a multi-head attention mechanism and an SNN network, which solves the problems of the prior art in the process of named entity recognition in electronic medical records, such as insufficient detection of entity boundaries, coarse character granularity, ignoring local features of entities and poor robustness, and has a faster and more accurate entity recognition effect.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a Chinese electronic medical record named entity recognition method based on a multi-head attention mechanism and an SNN network, comprising the following steps: Step 1, data processing: obtain the electronic medical record dataset and preprocess the dataset to obtain sequence labels corresponding to the text; Step 2: Use BERT to obtain dynamic word vectors: Input the sentences in the processed dataset into the pre-trained BERT model to obtain dynamic word vectors containing contextual information. ; Step 3: Construct the BiLSTM model: In the input model, after being processed by the BiLSTM model, the original input vector will be converted into a new representation vector ; The new vector It contains richer contextual information and sequence features, which can more effectively capture the temporal relationships and long-term dependencies in the data, thereby improving the performance of subsequent tasks; Step 4: Construct the SNN network: transform the vector processed by the BiLSTM model into The SNN network simulates the pulse emission mechanism of the biological neural system and transmits the input vector Processing to generate a new vector ; Step 5: Construct a multi-head attention mechanism: vector processed by the SNN network Passed as input to the attention mechanism; the multi-head attention mechanism passes the input vector Segmentation and processing are performed in multiple dimensions to achieve multi-angle feature attention and information integration to generate new vectors ; Step 6: Construct the CRF module: transform the vector processed by the multi-head attention mechanism It is passed to the CRF module to obtain the final output sequence, which is converted into a sequence annotation in the form of BIO (Begin, Inside, Outside) by decoding, and used as the final annotation result of medical text entity recognition.

[0007] Preferably, the step 1 comprises the following steps: Step 1.1: Select a suitable public dataset and perform systematic data cleaning and normalization operations on the selected dataset to remove noise, missing values ​​and outliers in the data, thereby ensuring the accuracy and consistency of the data; at the same time, normalizing the text format will help eliminate unnecessary differences in the text, making the data easier to process and analyze later; Step 1.2: Perform entity annotation on the optimized data to identify key information in the text; Step 1.3: Divide the labeled data set into training set, test set and validation set in the ratio of 8:1:1 to ensure the effectiveness of model training and evaluation.

[0008] Preferably, step 2 comprises the following steps: Input the dataset obtained in step 1 into the pre-trained BERT model to generate dynamic word vectors with global context information , ,in Indicates the token representation of the nth character of the output sentence; the dynamic word vector , expressed as These word vectors not only reflect the local semantics of a single word, but also integrate the multiple meanings of words in different contexts, providing richer semantic information.

[0009] Preferably, step 3 comprises the following steps: Step 3.1: Build the BiLSTM model and fine-tune the model's hyperparameters, including learning rate, batch size, and other training parameters; the choice of these parameters has an important impact on the model's training results and final performance.

[0010] Step 3.2: Convert the vector generated in step 2 Passed as input to the constructed BiLSTM model; in this model, the input vector It will go through a two-way propagation process, namely forward propagation and backward propagation. This two-way processing mechanism enables BiLSTM to capture information from both the front and back directions of the sequence at the same time, thereby extracting semantic features more comprehensively. The specific formula is as follows: ; ; ; Where f and g are the activation functions of sigmoid and tanh respectively. is the weight parameter, is the text vector input at time t, and Respectively represent the forward and reverse acquisition of context sequence information, Represents the final output, and this output vector is used as .

[0011] Preferably, step 4 comprises the following steps: Create an SNN network and convert the output vector obtained in step 3 As the input of this network, the SNN network introduces time information. Neurons are only active when they receive or generate peak signals, and time is used as an additional input factor. Sparse pulse signals are used to record meaningful information. Finally, the vector is obtained after screening by the SNN network. .

[0012] Preferably, the step 5 comprises the following steps: Step 5.1: Convert the output vector into three matrices of the same dimension, namely the value matrix (V), key matrix (K) and query matrix (Q); Step 5.2: Input these matrices into the attention and multi-head attention mechanisms in turn to get the final output vector ; The specific formula is: ; ; Perform h times of information joint representation to obtain the text matrix feature MHA (Q, K, V) at time t, where Represents the latitude value, represents the result of a single calculation, C is the matrix link function, is the transformation matrix.

[0013] Preferably, step 6 comprises the following steps: Step 6.1: The resulting vector Passed into the CRF model; Step 6.2: Convert the vector A sequence of text as output , the predicted text label sequence is , the formula of the prediction function S is as follows: ; Where A is the transfer score matrix, For Transfer to The probability of the label. Expressed as Words are marked as The probability of Step 6.3: Use the Softmax function to calculate the probability of the label sequence Y. The formula is as follows: ; Step 6.4: Use the Viterbi algorithm to obtain the final text sequence label The formula is as follows: ; After passing through the CRF layer as the final output sequence, it is converted into a sequence annotation in the form of BIO using a decoding method, and this is used as the final annotation result of medical text entity recognition.

[0014] The present invention provides a method for Chinese electronic medical record named entity recognition based on a multi-head attention mechanism and an SNN network, which has the following beneficial effects: 1. Multi-level semantic feature extraction: Through the combination of BERT, BiLSTM, SNN and multi-head attention mechanism (MHA), the model can deeply explore the semantic features of the text, including local and global semantics. This multi-layer feature extraction enables the model to perform well in understanding complex medical language.

[0015] 2. Feature enhancement and improved robustness: Through the combination of SNN and MHA, the model effectively captures the semantic associations between long-distance and short-distance, improves the ability to cope with changes in different contexts, and enhances the robustness and generalization ability of the model.

[0016] 3. More accurate entity boundary recognition: Traditional NER models tend to ignore entity boundary information. This model reduces the error rate of entity boundary recognition and improves the accuracy of entity recognition through the collaboration of multi-head attention mechanism and BiLSTM.

[0017] 4. Effectively solve the overfitting problem: BiLSTM output is prone to overfitting, especially on small data sets. After the introduction of the SNN module, the model retains meaningful information through sparse pulse signals, reducing overfitting during training and improving the stability of the model.

[0018] 5. Higher training efficiency and time optimization: Compared with other models, the SNN-MHA module shortens the training time. This optimization reduces the computing cost and improves the practicality of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 is a flow chart of the method of the present invention; Figure 2 This is a model architecture diagram of the present invention. DETAILED DESCRIPTION

[0020] like Figure 1 As shown in the figure, the Chinese electronic medical record named entity recognition method based on multi-head attention mechanism and SNN network includes the following steps: Step 1, data processing: obtain the electronic medical record dataset and preprocess the dataset to obtain sequence labels corresponding to the text; Step 2: Use BERT to obtain dynamic word vectors: Input the sentences in the processed dataset into the pre-trained BERT model to obtain dynamic word vectors containing contextual information. ; Step 3: Construct the BiLSTM model: In the input model, after being processed by the BiLSTM model, the original input vector will be converted into a new representation vector ; The new vector It contains richer contextual information and sequence features, which can more effectively capture the temporal relationships and long-term dependencies in the data, thereby improving the performance of subsequent tasks; Step 4: Construct the SNN network: transform the vector processed by the BiLSTM model into The SNN network simulates the pulse emission mechanism of the biological neural system and transmits the input vector Processing to generate a new vector ; Step 5: Construct a multi-head attention mechanism: vector processed by the SNN network Passed as input to the attention mechanism; the multi-head attention mechanism passes the input vector Segmentation and processing are performed in multiple dimensions to achieve multi-angle feature attention and information integration to generate new vectors ; Step 6: Construct the CRF module: transform the vector processed by the multi-head attention mechanism It is passed to the CRF module to obtain the final output sequence, which is converted into a sequence annotation in the form of BIO (Begin, Inside, Outside) by decoding, and used as the final annotation result of medical text entity recognition.

[0021] Preferably, the step 1 comprises the following steps: Step 1.1: Select a suitable public dataset and perform systematic data cleaning and normalization operations on the selected dataset to remove noise, missing values ​​and outliers in the data, thereby ensuring the accuracy and consistency of the data; at the same time, normalizing the text format will help eliminate unnecessary differences in the text, making the data easier to process and analyze later; Step 1.2: Perform entity annotation on the optimized data to identify key information in the text; Step 1.3: Divide the labeled data set into training set, test set and validation set in the ratio of 8:1:1 to ensure the effectiveness of model training and evaluation.

[0022] Preferably, step 2 comprises the following steps: Input the dataset obtained in step 1 into the pre-trained BERT model to generate dynamic word vectors with global context information , ,in Indicates the token representation of the nth character of the output sentence; the dynamic word vector , expressed as These word vectors not only reflect the local semantics of a single word, but also integrate the multiple meanings of words in different contexts, providing richer semantic information.

[0023] Preferably, step 3 comprises the following steps: Step 3.1: Build the BiLSTM model and fine-tune the model's hyperparameters, including learning rate, batch size, and other training parameters; the choice of these parameters has an important impact on the model's training results and final performance.

[0024] Step 3.2: Convert the vector generated in step 2 Passed as input to the constructed BiLSTM model; in this model, the input vector It will go through a two-way propagation process, namely forward propagation and backward propagation. This two-way processing mechanism enables BiLSTM to capture information from both the front and back directions of the sequence at the same time, thereby extracting semantic features more comprehensively. The specific formula is as follows: ; ; ; Where f and g are the activation functions of sigmoid and tanh respectively. is the weight parameter, is the text vector input at time t, and Respectively represent the forward and reverse acquisition of context sequence information, Represents the final output, and this output vector is used as .

[0025] Preferably, step 4 comprises the following steps: Create an SNN network and convert the output vector obtained in step 3 As the input of this network, the SNN network introduces time information. Neurons are only active when they receive or generate peak signals, and time is used as an additional input factor. Sparse pulse signals are used to record meaningful information. Finally, the vector is obtained after screening by the SNN network. .

[0026] Preferably, the step 5 comprises the following steps: Step 5.1: Convert the output vector into three matrices of the same dimension, namely the value matrix (V), key matrix (K) and query matrix (Q); Step 5.2: Input these matrices into the attention and multi-head attention mechanisms in turn to get the final output vector ; The specific formula is: ; ; Perform h times of information joint representation to obtain the text matrix feature MHA (Q, K, V) at time t, where Represents the latitude value, represents the result of a single calculation, C is the matrix link function, is the transformation matrix.

[0027] Preferably, step 6 comprises the following steps: Step 6.1: The resulting vector Passed into the CRF model; Step 6.2: Convert the vector A sequence of text as output , the predicted text label sequence is , the formula of the prediction function S is as follows: ; Where A is the transfer score matrix, For Transfer to The probability of the label. Expressed as Words are marked as probability; Step 6.3: Use the Softmax function to calculate the probability of the label sequence Y. The formula is as follows: ; Step 6.4: Use the Viterbi algorithm to obtain the final text sequence label The formula is as follows: ; After passing through the CRF layer as the final output sequence, it is converted into a sequence annotation in the form of BIO using a decoding method, and this is used as the final annotation result of medical text entity recognition.

[0028] like Figure 2 As shown in the figure, it is a model architecture diagram of the present invention, taking "detecting lesser curvature lymph nodes" as an example to process the input text. In this process, the input text is processed by the pre-trained model BERT, and each word is represented as its corresponding dynamic word vector; the BiLSTM layer is used to capture the context information of the text; the SNN-MHA layer is used to obtain the local features of the entity; finally, the CRF layer applies the conditional random field technology to learn the sequence label to output the correct sequence label.

[0029] In order to verify the effectiveness of the model experiment, this study selected several commonly used word fusion-based models for comparative experiments. The experiment was verified on two datasets, namely the CMeEE dataset and the CCKS2020 dataset. The experimental results are listed in Tables 1 and 2.

[0030] Table 1 CMeEE comparative experimental data (%)

[0031] Table 2 CCKS2020 comparative experimental data (%)

[0032] From the data in Table 1 and Table 2 above, we can get: (1) In the comparison between the BERT-BiLSTM-CRF model and the BiLSTM-CRF model, it was found that the model indicators were improved after the introduction of the BERT model. This may be due to the fact that the BiLSTM-CRF model uses the embedding of word2vec static word vectors, while the BERT model is output through multiple layers of transformers, which can comprehensively consider the semantic information in different contexts and generate dynamic word vectors, thereby effectively solving the problem of polysemy. Therefore, the pre-trained BERT model can capture richer semantic information features.

[0033] (2) Compared with the BERT-BiLSTM-CRF model, the F1 value of our model increased by 1.04% and 2.82% on the CMeEE dataset and CCKS2020 dataset, respectively. This difference may be due to the introduction of the time series multi-head attention mechanism SNN-MHA network module, which can more comprehensively extract semantic features based on the context, thereby enhancing the entity recognition performance of the model.

[0034] The present invention provides a method for named entity recognition of Chinese electronic medical records based on a multi-head attention mechanism and a spiking neural network (SNN). For named entities in electronic medical records, a dynamic word vector is obtained through a pre-trained BERT model, a bidirectional long short-term memory network (BiLSTM) model is used to obtain text encoding, and the SNN network screens the features of the encoding. The multi-head attention mechanism calculates the dependency between characters and strengthens the relevance to the context. Finally, the conditional random field (CRF) module outputs the final sequence to obtain the entity recognition result. The present invention significantly reduces the time cost and improves the efficiency and accuracy of named entity recognition in medical texts, providing a more efficient and accurate technical solution for the field of medical information processing.

[0035] The above embodiments are only preferred technical solutions of the present invention and should not be regarded as limiting the present invention. The protection scope of the present invention shall be the technical solutions recorded in the claims, including equivalent replacement solutions of the technical features in the technical solutions recorded in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present invention.

Claims

1. A Chinese electronic medical record named entity recognition method based on multi-head attention mechanism and SNN network, characterized by: The following steps are involved: Step 1, data processing: obtain the electronic medical record dataset and preprocess the dataset to obtain sequence labels corresponding to the text; Step 2: Use BERT to obtain dynamic word vectors: Input the sentences in the processed dataset into the pre-trained BERT model to obtain dynamic word vectors containing contextual information. ; Step 3: Construct the BiLSTM model: In the input model, after being processed by the BiLSTM model, the original input vector will be converted into a new representation vector ; The new vector It contains richer contextual information and sequence features, which can more effectively capture the temporal relationships and long-term dependencies in the data, thereby improving the performance of subsequent tasks; Step 4: Construct the SNN network: transform the vector processed by the BiLSTM model into The SNN network simulates the pulse emission mechanism of the biological neural system and transmits the input vector Processing to generate a new vector ; Step 5: Construct a multi-head attention mechanism: vector processed by the SNN network Passed as input to the attention mechanism; The multi-head attention mechanism takes the input vector Segmentation and processing are performed in multiple dimensions to achieve multi-angle feature attention and information integration to generate new vectors ; Step 6: Construct the CRF module: transform the vector processed by the multi-head attention mechanism It is passed to the CRF module to obtain the final output sequence, which is converted into a sequence annotation in BIO form by decoding, and used as the final annotation result of medical text entity recognition.

2. According to claim 1, the method for named entity recognition of Chinese electronic medical records based on multi-head attention mechanism and SNN network is characterized in that: The step 1 comprises the following steps: Step 1.1: Select a public dataset and perform systematic data cleaning and normalization operations on the selected dataset to remove noise, missing values ​​and outliers in the data, thereby ensuring the accuracy and consistency of the data; at the same time, normalizing the text format will help eliminate unnecessary differences in the text, making the data easier to process and analyze later; Step 1.2: Perform entity annotation on the optimized data to identify key information in the text; Step 1.3: Divide the labeled data set into training set, test set and validation set in the ratio of 8:1:1 to ensure the effectiveness of model training and evaluation.

3. According to claim 1, the Chinese electronic medical record named entity recognition method based on multi-head attention mechanism and SNN network is characterized in that: The step 2 comprises the following steps: Input the dataset obtained in step 1 into the pre-trained BERT model to generate dynamic word vectors with global context information , ,in Indicates the token representation of the nth character of the output sentence; the dynamic word vector , expressed as .

4. According to claim 1, the method for named entity recognition of Chinese electronic medical records based on multi-head attention mechanism and SNN network is characterized in that: The step 3 comprises the following steps: Step 3.1: Build the BiLSTM model and fine-tune the model's hyperparameters, including learning rate, batch size, and other training parameters; Step 3.2: Convert the vector generated in step 2 Passed as input to the constructed BiLSTM model; in this model, the input vector It will go through a two-way propagation process, namely forward propagation and backward propagation. This two-way processing mechanism enables BiLSTM to capture information from both the front and back directions of the sequence at the same time, thereby extracting semantic features more comprehensively. The specific formula is as follows: ； ； ； Where f and g are the activation functions of sigmoid and tanh respectively. is the weight parameter, is the text vector input at time t, and Respectively represent the forward and reverse acquisition of context sequence information, Represents the final output, and this output vector is used as .

5. According to claim 1, the method for named entity recognition of Chinese electronic medical records based on multi-head attention mechanism and SNN network is characterized in that: The step 4 comprises the following steps: Create an SNN network and convert the output vector obtained in step 3 As the input of this network, the SNN network introduces time information. Neurons are only active when they receive or generate peak signals, and time is used as an additional input factor. Sparse pulse signals are used to record meaningful information. Finally, the vector is obtained after screening by the SNN network. .

6. According to claim 1, the method for named entity recognition of Chinese electronic medical records based on multi-head attention mechanism and SNN network is characterized in that: The step 5 comprises the following steps: Step 5.1: Convert the output vector into three matrices of the same dimension, namely the value matrix (V), key matrix (K) and query matrix (Q); Step 5.2: Input these matrices into the attention and multi-head attention mechanisms in turn to get the final output vector ; The specific formula is: ； ； Perform h times of information joint representation to obtain the text matrix feature MHA (Q, K, V) at time t, where Represents the latitude value, represents the result of a single calculation, C is the matrix link function, is the transformation matrix.

7. According to claim 1, the method for named entity recognition of Chinese electronic medical records based on multi-head attention mechanism and SNN network is characterized in that: The step 6 comprises the following steps: Step 6.1: The resulting vector Passed into the CRF model; Step 6.2: Convert the vector A sequence of text as output , the predicted text label sequence is , the formula of the prediction function S is as follows: ； Where A is the transfer score matrix, For Transfer to The probability of the label. Expressed as Words are marked as The probability of Step 6.3: Use the Softmax function to calculate the probability of the label sequence Y. The formula is as follows: ； Step 6.4: Use the Viterbi algorithm to obtain the final text sequence label The formula is as follows: ； After passing through the CRF layer as the final output sequence, it is converted into a sequence annotation in the form of BIO using a decoding method, and this is used as the final annotation result of medical text entity recognition.

Images