A method for named entity recognition in the medical field

Abstract

The present invention provides a method for named entity recognition in the medical field, which is characterized by comprising the following steps: S1, obtaining relevant data in the medical field and annotating the medical data; S2, using an EDA method to process the annotated data, including synonym replacement, random insertion, random exchange and random deletion; S3, constructing a Bert (which has been pre-trained on a large medical data set, and has used a hierarchical decomposition method to expand position coding) + Bi-GRU (integrated with an attention mechanism) + CRF model; S4, using 10-fold cross validation to reasonably adjust and optimize the model; the present invention can effectively solve the problem of data scarcity, can effectively extract complex medical entities in ultra-long texts, and uses K-fold cross validation to reasonably adjust and optimize the model.

Description

Technical Field

[0001] The present invention relates to an implementation of artificial intelligence technology in the medical field, and specifically to a method for named entity recognition in the medical field. Background Art

[0002] Medical named entity recognition refers to identifying the boundaries of medical entities from medical texts and determining the categories of medical entities. Common medical entity categories include disease names, body parts, drug information, examinations or examination items, and symptoms. The accuracy of medical named entity recognition affects the results of tasks such as event extraction and relationship extraction. It is a key task in medical text data mining and provides a key foundation for building medical ICD coding systems, health care systems, intelligent medical question-and-answer systems, and medical knowledge graphs. A good method for medical named entity recognition is of far-reaching significance.

[0003] Many existing medical named entity recognition technologies have ignored the problem of data scarcity, and the deep learning methods used are too simple. They often cannot produce good results when faced with complex medical entities and extremely long text attributes in electronic medical records, and the models lack optimized processes. Summary of the invention

[0004] The purpose of the present invention is to propose a method for named entity recognition in the medical field to solve the problem that many existing medical named entity recognition technologies proposed in the above background technology ignore the problem of data scarcity, the deep learning method used is too simple, and it often cannot produce good results when faced with complex medical entities and ultra-long text attributes of electronic medical records, and the model lacks optimized processes.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for named entity recognition in the medical field comprises the following steps:

[0007] S1. Obtain relevant data in the medical field and label the medical data;

[0008] S2. Use EDA (a data enhancement technique for text classification) to process the labeled data (mainly the processing of unlabeled words), including synonym replacement, random insertion, random exchange and random deletion;

[0009] S3. Build a model of Bert (which has been pre-trained on a large medical data set and uses a hierarchical decomposition method to expand the position encoding) + Bi-GRU (integrated with the attention mechanism) + CRF;

[0010] S4. Use 10-fold cross validation to reasonably adjust and optimize the model.

[0011] The acquisition of medical field related data in step S1 specifically includes the following steps:

[0012] S1.1. Connect with medical institutions to obtain electronic medical record data;

[0013] S1.2. Web crawling for medical data; specifically including:

[0014] S1.2.1. Obtain the URL of the target medical data;

[0015] S1.2.2. Submit an HTTP request to the corresponding URL;

[0016] S1.2.3. Parse HTTP response;

[0017] S1.2.4. Storage of analysis results

[0018] In the step S1, the acquired medical data is labeled, and the main labeling types include: diagnosis, surgery, treatment, examination, medicine, and location.

[0019] The processing of the marked data in step S2 specifically includes the following contents:

[0020] ① Synonym replacement: Randomly select 1-10 non-stop words in the sentence. For each selected word, replace it with its randomly selected synonym;

[0021] ② Random insertion: Find a non-stop word in the sentence, randomly select a synonym for it, and insert it into any position in the sentence. Repeat randomly 1-10 times;

[0022] ③ Random swap: randomly select two words in a sentence and swap their positions. Repeat randomly 1-10 times;

[0023] ④ Random deletion: For each word with a probability greater than 0.1 in the sentence, randomly delete it or not delete it.

[0024] The step S3 specifically includes the following contents:

[0025] S3.1. Obtain the Bert model that has been pre-trained on a large-scale medical dataset from the Internet;

[0026] S3.2, construct the Bert layer of hierarchical decomposition containing position encoding;

[0027] S3.3. Construct a Bi-GRU layer with attention mechanism.

[0028] The step S3.2 specifically includes the following contents:

[0029] Specifically, let the maximum position encoding length that Bert can train by default be n, and the corresponding position encoding vectors are p1, p2, ···, p n , the new encoding vectors that can be constructed by this method are q1, q2, ···, q m , where m = n 2 ,

[0030] q (i-1)×n+j =au i +(1+a)u j

[0031]

[0032] Where i is the position index of the first layer, j is the position index of the second layer, n is the Bert layer length, a∈(0,1) and a≠0.5.

[0033] The step S3.3 specifically includes the following contents:

[0034] For any time step i, given a mini-batch of input data X i ∈R n×d , where n is the batch length, d is the vector length, and the hidden layer activation function is The forward and backward hidden states at this time step are divided into l i ∈R n×h and r i ∈R n×h , where h is the number of hidden units. The updates of the forward and backward hidden states are as follows:

[0035]

[0036]

[0037] in is the weight term, is the bias term, Xi is formed according to the self-attention mechanism;

[0038] where r i Corresponding to the i-th data after embedding, with r i As a query, all the original input data r1, r2, ... n As keys and values, the attention score function is f, then

[0039]

[0040] Among them, concat is used to concatenate two vectors, β(r i ,r j) is to map the query and key vectors into scalars through the attention score function (here the scaled dot product attention score function is used), and then pass it through the softmax function to get:

[0041]

[0042] The step S4 specifically includes the following contents:

[0043] S4.1. Divide the data into K parts (K is set to 10 here), take one of them as the test set, and the rest as the training set. Here we can get 10 training sets and validation sets. Use these data to train the model in turn, and take the average of 10 errors.

[0044] S4.2. Reasonably adjust the model hyperparameters and neural network structure, repeat the steps in S4.1 to find the model with the best error result, and then use all the data to train this optimal model.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention can be used to extract named entities from medical texts by performing EDA data augmentation on the original data; the model uses Bert pre-trained on a large-scale medical data set as the Embedding layer, and hierarchically decomposes the position encoding of the Bert layer to construct a Bi-GRU+CRF layer with a self-attention mechanism; a 10-fold cross-validation method is used to train and verify the model, and the hyperparameters of the model and the structure of the neural network are reasonably adjusted. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is the overall flow chart of the present invention;

[0048] Figure 2 A flowchart for obtaining medical field related data and annotating medical data for the present invention;

[0049] Figure 3 A schematic diagram of hierarchical decomposition of position coding when constructing a Bert layer containing hierarchical decomposition of position coding in the present invention;

[0050] Figure 4 A schematic diagram of a method for constructing a Bi-GRU layer with an attention mechanism for the present invention;

[0051] Figure 5 X in step S3.2 of the present invention i The specific flow chart formed;

[0052] Figure 6 The specific process of model construction in step S3 of the present invention;

[0053] Figure 7 This is a specific flow chart of step S4 of the present invention. DETAILED DESCRIPTION

[0054] In order to clarify the technical problems, technical solutions, implementation processes and performance demonstrations, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are for explanation only. The present invention is not intended to limit the present invention. Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0055] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0056] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0057] Example 1

[0058] like Figure 1 As shown, a method for named entity recognition in the medical field includes the following steps:

[0059] S1. Obtain relevant data in the medical field and label the medical data;

[0060] S2. Use EDA (a data enhancement technique for text classification) to process the labeled data (mainly the processing of unlabeled words), including synonym replacement, random insertion, random exchange and random deletion;

[0061] S3. Build a model of Bert (which has been pre-trained on a large medical data set and uses a hierarchical decomposition method to expand the position encoding) + Bi-GRU (integrated with the attention mechanism) + CRF;

[0062] S4. Use 10-fold cross validation to reasonably adjust and optimize the model.

[0063] like Figure 2 As shown, the acquisition of medical field related data in step S1 specifically includes the following steps:

[0064] S1.1. Connect with medical institutions to obtain electronic medical record data;

[0065] S1.2. Crawling the Internet to obtain medical data; specifically including:

[0066] S1.2.1. Obtain the URL of the target medical data;

[0067] S1.2.2. Submit an HTTP request to the corresponding URL;

[0068] S1.2.3. Parse HTTP response;

[0069] S1.2.4. Storage of analysis results

[0070] In step S1, the acquired medical data is labeled, and the main labeling types include: diagnosis, surgery, treatment, examination, medicine, and location.

[0071] The processing of the marked data in step S2 specifically includes the following contents:

[0072] ① Synonym replacement: Randomly select 1-10 non-stop words in the sentence. For each selected word, replace it with its randomly selected synonym;

[0073] ② Random insertion: Find a non-stop word in the sentence, randomly select a synonym for it, and insert it into any position in the sentence. Repeat randomly 1-10 times;

[0074] ③ Random swap: randomly select two words in a sentence and swap their positions. Repeat randomly 1-10 times;

[0075] ④ Random deletion: For each word with a probability greater than 0.1 in the sentence, randomly delete it or not delete it.

[0076] The step S3 specifically includes the following contents:

[0077] S3.1. Obtain the Bert model that has been pre-trained on a large-scale medical dataset from the Internet;

[0078] S3.2, construct the Bert layer of hierarchical decomposition containing position encoding;

[0079] S3.3. Construct a Bi-GRU layer with attention mechanism.

[0080] like Figure 3 As shown, the step S3.2 specifically includes the following contents:

[0081] Specifically, let the maximum position encoding length that Bert can train by default be n, and the corresponding position encoding vectors are p1, p2, ···, p n , the new encoding vectors that can be constructed by this method are q1, q2, ···, q m, where m = n 2 ,

[0082] q (i-1)×n+j =au i +(1+a)u j

[0083]

[0084] Where i is the position index of the first layer, j is the position index of the second layer, n is the Bert layer length, a∈(0,1) and a≠0.5.

[0085] like Figure 4 As shown, the step S3.3 specifically includes the following contents:

[0086] For any time step i, given a mini-batch of input data X i ∈R n×d , where n is the batch length, d is the vector length, and the hidden layer activation function is The forward and reverse hidden states of this time step are divided into l i ∈R n×h and r i ∈R n ×h , where h is the number of hidden units. The updates of the forward and backward hidden states are as follows:

[0087]

[0088]

[0089] in is the weight term, is a bias term, Xi is formed according to the self-attention mechanism, and the specific process is as follows Figure 5 As shown;

[0090] where r i Corresponding to the i-th data after embedding, with r i As a query, all the original input data r1, r2, ... n As keys and values, the attention score function is f, then

[0091]

[0092] Among them, concat is used to concatenate two vectors, β(r i ,r j) is to map the query and key vectors into scalars through the attention score function (here the scaled dot product attention score function is used), and then pass it through the softmax function to get:

[0093]

[0094] like Figure 6 As shown, step S4 specifically includes the following contents:

[0095] S4.1. Divide the data into K parts (K is set to 10 here), take one of them as the test set, and the rest as the training set. Here we can get 10 training sets and validation sets. Use these data to train the model in turn, and take the average of 10 errors.

[0096] S4.2. Reasonably adjust the model hyperparameters and neural network structure, repeat the steps in S4.1 to find the model with the best error result, and then use all the data to train this optimal model.

[0097] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for named entity recognition in the medical field, characterized in that: The following steps are involved: S1. Obtain relevant data in the medical field and label the medical data; S2. Use EDA methods to process the labeled data, including synonym replacement, random insertion, random exchange, and random deletion; S3. Build a Bert+Bi-GRU+CRF model. Bert has been pre-trained on a large medical dataset, and uses a hierarchical decomposition method to expand position encoding. Bi-GRU integrates the attention mechanism. S4. Use 10-fold cross validation to reasonably adjust and optimize the model; The acquisition of medical field related data in step S1 specifically includes the following steps: S1.

1. Connect with medical institutions to obtain electronic medical record data; S1.

2. Crawling the Internet to obtain medical data; specifically including: S1.2.

1. Obtain the URL of the target medical data; S1.2.

2. Submit an HTTP request to the corresponding URL; S1.2.

3. Parse HTTP response; S1.2.

4. Storage of analysis results In step S1, the acquired medical data is annotated, and the main annotation types include: diagnosis, surgery, treatment, examination, medicine, and location; The processing of the marked data in step S2 specifically includes the following contents: ① Synonym replacement: randomly select 1-10 non-stop words in the sentence; for each selected word, replace it with its randomly selected synonym; ② Random insertion: Find a non-stop word in the sentence, randomly select a synonym for it, and insert it into any position in the sentence; repeat randomly 1-10 times; ③ Random exchange: randomly select two words in a sentence and exchange their positions; repeat randomly 1-10 times; ④ Random deletion: For each word with a probability greater than 0.1 in the sentence, randomly delete or not delete it; Step S3 specifically includes the following contents: S3.

1. Obtain the Bert model that has been pre-trained on a large-scale medical dataset from the Internet; S3.2, construct the Bert layer of hierarchical decomposition containing position encoding; S3.

3. Construct a Bi-GRU layer with attention mechanism. Step S3.2 specifically includes the following contents: Assume that the maximum position encoding length that Bert can train by default is n, and the corresponding position encoding vectors are p1, p2, ···, p n , the new encoding vectors constructed based on the following two formulas are q1, q2, ···, q m , where m = n 2 , q (k-1)×n+j to the k +(1+a)u j Where k is the position index of the first layer, j is the position index of the second layer, a∈(0,1) and a≠0.5; Step S3.3 specifically includes the following contents: For any time step i, given a mini-batch of input data X i ∈R n×d , let the hidden layer activation function be The forward and reverse hidden states of this time step are l i ∈R n×h and r i ∈R n×h , where h is the number of hidden units, the forward and backward hidden states are updated as follows: in is the weight item; is the bias term, Xi is formed according to the self-attention mechanism; Among them, i Corresponding to the i-th data after embedding, Er i As query Query, all input data E1, E2...E n As keys Keys and values ​​Values, the attention score function is f, then Among them, concat is used to concatenate two vectors, β(Eri,r j ) is to map the query and key vectors into scalars through the attention score function, and then pass them through the softmax function to get: Step S4 specifically includes the following contents: S4.

1. Divide the data into K parts, K=10, take one of them as the test set, and the rest as the training set. Here we can get 10 training sets and validation sets. Use these data to train the model in turn, and get the average of 10 errors. S4.

2. Reasonably adjust the model hyperparameters and neural network structure, repeat the steps in S4.1 to find the model with the best error result, and then use all the data to train this optimal model.

Images