Chinese electronic medical record named entity recognition method based on multi-head attention mechanism and SNN network

Abstract

This paper provides a method for named entity recognition in Chinese electronic medical records based on a multi-head attention mechanism and a spiking neural network (SNN). For named entities in electronic medical records, a pre-trained BERT model is used to obtain dynamic word vectors. A bidirectional long short-term memory (BiLSTM) model is used to obtain text encodings. The SNN network then filters the encoding features. A multi-head attention mechanism calculates dependencies between characters and strengthens contextual relevance. Finally, a conditional random field (CRF) module is used to output the final sequence, resulting in entity recognition results. This method significantly reduces time costs while improving the efficiency and accuracy of named entity recognition in medical texts, providing a more efficient and accurate technical solution for the field of medical information processing.

Description

Technical Field

[0001] The present invention relates to the field of computer natural language processing technology, and in particular to a method for recognizing named entities in Chinese electronic medical records based on a multi-head attention mechanism and an SNN network. Background Art

[0002] In the field of natural language processing, named entity recognition (NER), as a key technology for information extraction, aims to identify specific types of entities from text, such as time, place, personal names, dates, and organization names. The key challenge of this task lies in accurately determining the boundaries of entities within the text and accurately classifying their types. The development of NER technology is not only of great significance to subsequent tasks such as information extraction, question-answering systems, and machine translation, but also plays a key role in academic research and commercial applications. Accurately identifying entities not only helps the system better understand the text but also provides important semantic information for various natural language processing tasks. Therefore, the continuous improvement and application of NER technology will bring more innovation and progress to the field of natural language processing.

[0003] Named entity recognition (NER) plays a vital role in the medical field. The primary goal of medical NER is to extract predefined professional medical terms, such as diseases, medications, diagnostic types, and imaging examinations, from unstructured medical text. Accurately identifying and extracting these entities is crucial for downstream tasks, as highly accurate entity recognition can improve the efficiency and quality of information processing in the medical field. In the medical field, accurate NER not only helps doctors quickly access critical information but also provides researchers with valuable medical data support. Therefore, the continued optimization and application of NER technology will have a positive impact on medical information extraction, diagnostic assistance systems, and medical data analysis.

[0004] Although numerous models have emerged in current research and achieved some success, most models suffer from limitations in detecting entity boundaries. Their coarse character granularity tends to overlook local entity features and exhibits poor robustness. To address this issue, a new model for named entity recognition in Chinese medical texts is proposed, integrating a multi-head attention mechanism with a spiking neural network (SNN). By deeply and meticulously mining the semantic features of the text, it aims to improve the accuracy and robustness of entity recognition. This approach aims to address the shortcomings of existing models and provide new insights and methods for in-depth research and application of named entity recognition in Chinese medical texts. This model can more comprehensively capture entity information, thereby improving the effectiveness of entity recognition in medical texts and laying a foundation for further research and application of entity recognition. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a Chinese electronic medical record named entity recognition method based on a multi-head attention mechanism and an SNN network, which solves the problems of the existing technology in the process of electronic medical record named entity recognition, such as insufficient detection of entity boundaries, rough word granularity, ignoring local features of entities and poor robustness, and has a faster and more accurate entity recognition effect.

[0006] To solve the above technical problems, the technical solution adopted by the present invention is: a method for named entity recognition of Chinese electronic medical records based on a multi-head attention mechanism and an SNN network, comprising the following steps:

[0007] Step 1, data processing: obtain the electronic medical record dataset and preprocess the dataset to obtain sequence labels corresponding to the text;

[0008] Step 2: Use BERT to obtain dynamic word vectors: Input the sentences in the processed dataset into the pre-trained BERT model to obtain dynamic word vectors containing contextual information. ;

[0009] Step 3, construct the BiLSTM model: transform the processed vector In the input model, after being processed by the BiLSTM model, the original input vector will be converted into a new representation vector ; The new vector It contains richer contextual information and sequence features, which can more effectively capture the temporal relationships and long-term dependencies in the data, thereby improving the performance of subsequent tasks;

[0010] Step 4: Construct the SNN network: transform the vector processed by the BiLSTM model into As input, it is passed to the SNN network; the SNN network simulates the pulse emission mechanism of the biological neural system to process the input vector Processing to generate a new vector ;

[0011] Step 5: Construct a multi-head attention mechanism: vector processed by the SNN network Passed as input to the attention mechanism; the multi-head attention mechanism takes the input vector Segmentation and processing are performed in multiple dimensions to achieve multi-angle feature attention and information integration to generate new vectors ;

[0012] Step 6: Construct CRF module: The vector processed by the multi-head attention mechanism The final output sequence is passed to the CRF module and converted into a sequence annotation in the form of BIO (Begin, Inside, Outside) using a decoding method, which is used as the final annotation result of medical text entity recognition.

[0013] Preferably, the step 1 comprises the following steps:

[0014] Step 1.1: Select a suitable public dataset and perform systematic data cleaning and normalization to remove noise, missing values, and outliers to ensure data accuracy and consistency. Normalizing the text format will help eliminate unnecessary discrepancies in the text, making the data easier to process and analyze.

[0015] Step 1.2: Perform entity annotation on the optimized data to identify key information in the text;

[0016] Step 1.3: Divide the labeled dataset into training set, test set, and validation set in the ratio of 8:1:1 to ensure the effectiveness of model training and evaluation.

[0017] Preferably, the step 2 comprises the following steps:

[0018] Input the dataset obtained in step 1 into the pre-trained BERT model to generate dynamic word vectors with global context information , ,in Indicates the token representation of the nth character of the output sentence; the dynamic word vector , expressed as These word vectors not only reflect the local semantics of a single word, but also integrate the multiple meanings of words in different contexts, providing richer semantic information.

[0019] Preferably, step 3 comprises the following steps:

[0020] Step 3.1: Build the BiLSTM model and fine-tune the model's hyperparameters, including the learning rate, batch size, and other training parameters. The choice of these parameters has a significant impact on the model's training results and final performance.

[0021] Step 3.2: Convert the vector generated in step 2 Passed as input to the constructed BiLSTM model; in this model, the input vector The process goes through a two-way propagation process, namely forward propagation and backward propagation. This two-way processing mechanism enables BiLSTM to capture information from both the forward and backward directions of the sequence, thereby extracting semantic features more comprehensively.

[0022] The specific formula is as follows:

[0023] ;

[0024] ;

[0025] ;

[0026] Where f and g are the activation functions of sigmoid and tanh respectively, is the weight parameter, is the text vector input at time t, and Respectively represent forward and reverse acquisition of context sequence information, Represents the final output, and this output vector is used as .

[0027] Preferably, step 4 comprises the following steps:

[0028] Create an SNN network and convert the output vector obtained in step 3 As the input of this network, the SNN network introduces time information. Neurons are only active when they receive or generate peak signals, and time is used as an additional input factor. Sparse pulse signals are used to record meaningful information, and finally the vector is obtained after screening by the SNN network. .

[0029] Preferably, the step 5 comprises the following steps:

[0030] Step 5.1: Convert the output vector into three matrices of the same dimension: value matrix (V), key matrix (K), and query matrix (Q).

[0031] Step 5.2: Input these matrices into the attention and multi-head attention mechanisms in turn to obtain the final output vector ;

[0032] The specific formula is:

[0033] ;

[0034] ;

[0035] Perform h times of information joint representation to obtain the text matrix feature MHA (Q, K, V) at time t, where Indicates the latitude value, Represents the result of a single calculation, C is the matrix link function, is the transformation matrix.

[0036] Preferably, the step 6 comprises the following steps:

[0037] Step 6.1: The obtained vector Passed into the CRF model;

[0038] Step 6.2: Convert the vector A text sequence as output , the predicted text label sequence is , the formula of the prediction function S is as follows:

[0039] ;

[0040] Where A is the transfer score matrix, For Transfer to The probability of the label. Expressed as Words are marked as probability;

[0041] Step 6.3: Use the Softmax function to calculate the probability of the label sequence Y. The formula is as follows:

[0042] ;

[0043] Step 6.4: Use the Viterbi algorithm to obtain the final text sequence label The formula is as follows:

[0044] ;

[0045] After passing through the CRF layer as the final output sequence, it is converted into a sequence annotation in the form of BIO using a decoding method, and this is used as the final annotation result of medical text entity recognition.

[0046] The present invention provides a method for named entity recognition in Chinese electronic medical records based on a multi-head attention mechanism and an SNN network, which has the following beneficial effects:

[0047] 1. Multi-layer semantic feature extraction: By combining BERT, BiLSTM, SNN, and the multi-head attention mechanism (MHA), the model can deeply mine the semantic features of text, including both local and global semantics. This multi-layer feature extraction enables the model to excel in understanding complex medical language.

[0048] 2. Feature enhancement and improved robustness: Through the combination of SNN and MHA, the model effectively captures the semantic associations between long-distance and short-distance, improves the ability to cope with changes in different contexts, and enhances the robustness and generalization ability of the model.

[0049] 3. More accurate entity boundary recognition: Traditional NER models tend to ignore entity boundary information. However, this model reduces the error rate of entity boundary recognition and improves the accuracy of entity recognition through the collaboration of multi-head attention mechanism and BiLSTM.

[0050] 4. Effectively address overfitting: BiLSTM outputs are prone to overfitting, especially on small datasets. The introduction of the SNN module allows the model to retain meaningful information through sparse spike signals, reducing overfitting during training and improving model stability.

[0051] 5. Higher training efficiency and time optimization: Compared with other models, the SNN-MHA module shortens the training time. This optimization reduces the computational cost and improves the practicality of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention will be further described below with reference to the accompanying drawings and examples:

[0053] Figure 1 is a flow chart of the method of the present invention;

[0054] Figure 2 This is a model architecture diagram of the present invention. DETAILED DESCRIPTION

[0055] like Figure 1 As shown in the figure, the Chinese electronic medical record named entity recognition method based on the multi-head attention mechanism and SNN network includes the following steps:

[0056] Step 1, data processing: obtain the electronic medical record dataset and preprocess the dataset to obtain sequence labels corresponding to the text;

[0057] Step 2: Use BERT to obtain dynamic word vectors: Input the sentences in the processed dataset into the pre-trained BERT model to obtain dynamic word vectors containing contextual information. ;

[0058] Step 3, construct the BiLSTM model: transform the processed vector In the input model, after being processed by the BiLSTM model, the original input vector will be converted into a new representation vector ; The new vector It contains richer contextual information and sequence features, which can more effectively capture the temporal relationships and long-term dependencies in the data, thereby improving the performance of subsequent tasks;

[0059] Step 4: Construct the SNN network: transform the vector processed by the BiLSTM model into As input, it is passed to the SNN network; the SNN network simulates the pulse emission mechanism of the biological neural system to process the input vector Processing to generate a new vector ;

[0060] Step 5: Construct a multi-head attention mechanism: vector processed by the SNN network Passed as input to the attention mechanism; the multi-head attention mechanism takes the input vector Segmentation and processing are performed in multiple dimensions to achieve multi-angle feature attention and information integration to generate new vectors ;

[0061] Step 6: Construct CRF module: The vector processed by the multi-head attention mechanism The final output sequence is passed to the CRF module and converted into a sequence annotation in the form of BIO (Begin, Inside, Outside) using a decoding method, which is used as the final annotation result of medical text entity recognition.

[0062] Preferably, the step 1 comprises the following steps:

[0063] Step 1.1: Select a suitable public dataset and perform systematic data cleaning and normalization to remove noise, missing values, and outliers to ensure data accuracy and consistency. Normalizing the text format will help eliminate unnecessary discrepancies in the text, making the data easier to process and analyze.

[0064] Step 1.2: Perform entity annotation on the optimized data to identify key information in the text;

[0065] Step 1.3: Divide the labeled dataset into training set, test set, and validation set in the ratio of 8:1:1 to ensure the effectiveness of model training and evaluation.

[0066] Preferably, the step 2 comprises the following steps:

[0067] Input the dataset obtained in step 1 into the pre-trained BERT model to generate dynamic word vectors with global context information , ,in Indicates the token representation of the nth character of the output sentence; the dynamic word vector , expressed as These word vectors not only reflect the local semantics of a single word, but also integrate the multiple meanings of words in different contexts, providing richer semantic information.

[0068] Preferably, step 3 comprises the following steps:

[0069] Step 3.1: Build the BiLSTM model and fine-tune the model's hyperparameters, including the learning rate, batch size, and other training parameters. The choice of these parameters has a significant impact on the model's training results and final performance.

[0070] Step 3.2: Convert the vector generated in step 2 Passed as input to the constructed BiLSTM model; in this model, the input vector The process goes through a two-way propagation process, namely forward propagation and backward propagation. This two-way processing mechanism enables BiLSTM to capture information from both the forward and backward directions of the sequence, thereby extracting semantic features more comprehensively.

[0071] The specific formula is as follows:

[0072] ;

[0073] ;

[0074] ;

[0075] Where f and g are the activation functions of sigmoid and tanh respectively, is the weight parameter, is the text vector input at time t, and Respectively represent forward and reverse acquisition of context sequence information, Represents the final output, and this output vector is used as .

[0076] Preferably, step 4 comprises the following steps:

[0077] Create an SNN network and convert the output vector obtained in step 3 As the input of this network, the SNN network introduces time information. Neurons are only active when they receive or generate peak signals, and time is used as an additional input factor. Sparse pulse signals are used to record meaningful information, and finally the vector is obtained after screening by the SNN network. .

[0078] Preferably, the step 5 comprises the following steps:

[0079] Step 5.1: Convert the output vector into three matrices of the same dimension: value matrix (V), key matrix (K), and query matrix (Q).

[0080] Step 5.2: Input these matrices into the attention and multi-head attention mechanisms in turn to obtain the final output vector ;

[0081] The specific formula is:

[0082] ;

[0083] ;

[0084] Perform h times of information joint representation to obtain the text matrix feature MHA (Q, K, V) at time t, where Indicates the latitude value, Represents the result of a single calculation, C is the matrix link function, is the transformation matrix.

[0085] Preferably, the step 6 comprises the following steps:

[0086] Step 6.1: The obtained vector Passed into the CRF model;

[0087] Step 6.2: Convert the vector A text sequence as output , the predicted text label sequence is , the formula of the prediction function S is as follows:

[0088] ;

[0089] Where A is the transfer score matrix, For Transfer to The probability of the label. Expressed as Words are marked as probability;

[0090] Step 6.3: Use the Softmax function to calculate the probability of the label sequence Y. The formula is as follows:

[0091] ;

[0092] Step 6.4: Use the Viterbi algorithm to obtain the final text sequence label The formula is as follows:

[0093] ;

[0094] After passing through the CRF layer as the final output sequence, it is converted into a sequence annotation in the form of BIO using a decoding method, and this is used as the final annotation result of medical text entity recognition.

[0095] like Figure 2 The figure below shows the model architecture of the present invention, using the example of "detecting lesser curvature lymph nodes" to process input text. In this process, the input text is processed by the pre-trained BERT model, representing each word as its corresponding dynamic word vector. The BiLSTM layer captures the contextual information of the text; the SNN-MHA layer extracts local features of entities; and finally, the CRF layer uses conditional random field technology to learn sequence labels and output the correct sequence labels.

[0096] To verify the effectiveness of the model experiments, this study selected several commonly used word fusion-based models for comparative experiments. The experiments were conducted on two datasets: the CMeEE dataset and the CCKS2020 dataset. The experimental results are detailed in Tables 1 and 2.

[0097] Table 1 CMeEE comparative experimental data (%)

[0098]

[0099] Table 2 CCKS2020 comparative experimental data (%)

[0100]

[0101] From the data in Table 1 and Table 2 above, we can get:

[0102] (1) In the comparison between the BERT-BiLSTM-CRF model and the BiLSTM-CRF model, the model indicators were found to be improved after the introduction of the BERT model. This may be due to the fact that the BiLSTM-CRF model uses the word2vec static word vector embedding, while the BERT model, through the multi-layer transformer output, can comprehensively consider the semantic information in different contexts and generate dynamic word vectors, thereby effectively solving the problem of polysemy. Therefore, the use of the pre-trained BERT model can capture richer semantic information features.

[0103] (2) Compared with the BERT-BiLSTM-CRF model, the F1 value of our model increased by 1.04% and 2.82% on the CMeEE dataset and CCKS2020 dataset, respectively. This difference may be due to the introduction of the time series multi-head attention mechanism SNN-MHA network module, which can more comprehensively extract semantic features based on the context, thereby enhancing the entity recognition performance of the model.

[0104] This paper provides a method for named entity recognition in Chinese electronic medical records based on a multi-head attention mechanism and a spiking neural network (SNN). For named entities in electronic medical records, a pre-trained BERT model is used to obtain dynamic word vectors. A bidirectional long short-term memory (BiLSTM) model is used to obtain text encodings. The SNN network then filters the encoding features. A multi-head attention mechanism calculates dependencies between characters and strengthens contextual relevance. Finally, a conditional random field (CRF) module is used to output the final sequence, resulting in entity recognition results. This method significantly reduces time costs while improving the efficiency and accuracy of named entity recognition in medical texts, providing a more efficient and accurate technical solution for the field of medical information processing.

[0105] The above embodiments are merely preferred technical solutions of the present invention and should not be construed as limiting the present invention. The scope of protection of the present invention shall be the technical solutions set forth in the claims, including equivalent alternatives to the technical features of the technical solutions set forth in the claims. In other words, equivalent alternatives and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. A Chinese electronic medical record named entity recognition method based on a multi-head attention mechanism and SNN network, characterized by: The following steps are involved: Step 1, data processing: obtain the electronic medical record dataset and preprocess the dataset to obtain sequence labels corresponding to the text; Step 2: Use BERT to obtain dynamic word vectors: Input the sentences in the processed dataset into the pre-trained BERT model to obtain dynamic word vectors containing contextual information. ; Step 3, construct the BiLSTM model: transform the processed vector In the input model, after being processed by the BiLSTM model, the original input vector will be converted into a new representation vector ; The new vector It contains richer contextual information and sequence features, which can more effectively capture the temporal relationships and long-term dependencies in the data, thereby improving the performance of subsequent tasks; Step 4: Construct the SNN network: transform the vector processed by the BiLSTM model into As input, it is passed to the SNN network; the SNN network simulates the pulse emission mechanism of the biological neural system to process the input vector Processing to generate a new vector ; Step 5: Construct a multi-head attention mechanism: vector processed by the SNN network Passed as input to the attention mechanism; The multi-head attention mechanism is implemented by taking the input vector Segmentation and processing are performed in multiple dimensions to achieve multi-angle feature attention and information integration to generate new vectors ; Step 6: Construct CRF module: The vector processed by the multi-head attention mechanism The final output sequence is passed to the CRF module and converted into a sequence annotation in BIO format using decoding method, which is used as the final annotation result of medical text entity recognition.

2. The method for named entity recognition in Chinese electronic medical records based on a multi-head attention mechanism and SNN network according to claim 1 is characterized in that: The step 1 comprises the following steps: Step 1.1: Select a public dataset and perform systematic data cleaning and normalization to remove noise, missing values, and outliers to ensure data accuracy and consistency. Normalizing the text format will help eliminate unnecessary discrepancies in the text, making the data easier to process and analyze. Step 1.2: Perform entity annotation on the optimized data to identify key information in the text; Step 1.3: Divide the labeled dataset into training set, test set, and validation set in the ratio of 8:1:1 to ensure the effectiveness of model training and evaluation.

3. The method for named entity recognition in Chinese electronic medical records based on a multi-head attention mechanism and SNN network according to claim 1 is characterized in that: The step 2 comprises the following steps: Input the dataset obtained in step 1 into the pre-trained BERT model to generate dynamic word vectors with global context information , ,in Indicates the token representation of the nth character of the output sentence; the dynamic word vector , expressed as .

4. The method for named entity recognition in Chinese electronic medical records based on a multi-head attention mechanism and SNN network according to claim 1 is characterized in that: The step 3 comprises the following steps: Step 3.1: Build the BiLSTM model and fine-tune the model's hyperparameters, including learning rate, batch size, and other training parameters. Step 3.2: Convert the vector generated in step 2 Passed as input to the constructed BiLSTM model; in this model, the input vector The process goes through a two-way propagation process, namely forward propagation and backward propagation. This two-way processing mechanism enables BiLSTM to capture information from both the forward and backward directions of the sequence, thereby extracting semantic features more comprehensively. The specific formula is as follows: ； ； ； Where f and g are the activation functions of sigmoid and tanh respectively, is the weight parameter, is the text vector input at time t, and Respectively represent forward and reverse acquisition of context sequence information, Represents the final output, and this output vector is used as .

5. The method for named entity recognition in Chinese electronic medical records based on a multi-head attention mechanism and SNN network according to claim 1 is characterized in that: The step 4 comprises the following steps: Create an SNN network and convert the output vector obtained in step 3 As the input of this network, the SNN network introduces time information. Neurons are only active when they receive or generate peak signals, and time is used as an additional input factor. Sparse pulse signals are used to record meaningful information, and finally the vector is obtained after screening by the SNN network. .

6. The method for named entity recognition in Chinese electronic medical records based on a multi-head attention mechanism and SNN network according to claim 1 is characterized in that: The step 5 comprises the following steps: Step 5.1: Convert the output vector into three matrices of the same dimension: value matrix (V), key matrix (K), and query matrix (Q). Step 5.2: Input these matrices into the attention and multi-head attention mechanisms in turn to obtain the final output vector ; The specific formula is: ； ； Perform h times of information joint representation to obtain the text matrix feature MHA (Q, K, V) at time t, where Indicates the latitude value, Represents the result of a single calculation, C is the matrix link function, is the transformation matrix.

7. The method for named entity recognition in Chinese electronic medical records based on a multi-head attention mechanism and SNN network according to claim 1 is characterized in that: The step 6 comprises the following steps: Step 6.1: The resulting vector Passed into the CRF model; Step 6.2: Convert the vector A text sequence as output , the predicted text label sequence is , the formula of the prediction function S is as follows: ； Where A is the transfer score matrix, For Transfer to The probability of the label, Expressed as Words are marked as probability; Step 6.3: Use the Softmax function to calculate the probability of the label sequence Y. The formula is as follows: ； Step 6.4: Use the Viterbi algorithm to obtain the final text sequence label The formula is as follows: ； After passing through the CRF layer as the final output sequence, it is converted into a sequence annotation in the form of BIO using a decoding method, and this is used as the final annotation result of medical text entity recognition.

Images