A clinical named entity recognition method based on graph neural network multi-embedding combination
Through the multi-embedding joint method based on graph neural network, combined with feature extraction of FT-Bert and convolutional neural network, the problem of low accuracy and recall of clinical named entity recognition in massive medical text data is solved, and a higher Entity recognition accuracy and context understanding ability.
Patent Information
- Application Number
- CN202210540397.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-17
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2042-05-17
AI Technical Summary
The prior art is difficult to effectively utilize massive medical text data, especially in clinical naming entity recognition, which lacks effective context understanding and feature extraction capabilities, resulting in low accuracy and recall of entity recognition.
The multi-embedding joint method based on graph neural network is adopted, and initial weight is assigned through the FT-Bert model, combined with pinyin, glyph and radical features are used to enhance data, and features are extracted using convolutional neural network, and information is transmitted through multi-head attention mechanism. Finally, the features are input to BiLSTM and CRF for entity recognition.
It improves the contextual understanding ability of entity prediction, significantly improves F1-score, achieves a higher harmonized average of accuracy and recall, and enhances the model's ability to recognize clinical medical texts.
Smart Images
Figure CN114841167B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a clinical medical named entity recognition method that combines deep learning, artificial intelligence, and natural language processing, especially multi-angle data enhancement technology based on graph neural networks. Background Art
[0002] With the development of informatization in the medical field, modern medical information systems have a huge amount of data. Effectively utilizing massive data to turn it into a more valuable academic resource is an urgent problem for researchers. Natural language processing technology can help researchers mine medical text information data. Among them, named entity recognition technology can effectively identify important entities, such as disease, surgery, treatment, and drug types. The relationship between these entities can be divided into time periods, thereby providing data support for clinical medical decision-making and improving the overall quality of hospital medical care, which is of great significance. Summary of the invention
[0003] The present invention aims to solve the above problems of the prior art. A clinical named entity recognition method based on graph neural network multi-embedding combination is proposed. The present invention specifically includes the following steps:
[0004] 101. Perform word segmentation on each word in the input text corresponding to the vocabulary in the pre-training result, and assign initial weights through FT-Bert;
[0005] 102. Divide the input text into batches and truncate it by MAX_LEN;
[0006] 103. Fill each sentence of the input text to the same length in batches, and the length is the maximum length of the sentences in the current batch;
[0007] 104. Connect all sentence pairs by using [CLS] at the beginning of the sentence and [SEP] at the end of the sentence;
[0008] 105. Add a pinyin feature vector to the input text data, including pinyin and tone, add glyph features and radical features, and concatenate them as data enhancement vectors. Further extract features by using a convolutional neural network to obtain data enhancement multi-embedded joint features.
[0009] 106. Construct a graph neural network, in which the information between nodes and edges is transmitted through a multi-head attention mechanism;
[0010] 107. Construct word group information about clinical fields;
[0011] 108. Combine FT-Bert, data-enhanced multi-embedded joint features, and the aggregation results based on graph neural network as the input of BiLSTM to further extract text features;
[0012] 109. Use CRF to decode the output data of BiLSTM to achieve entity recognition.
[0013] The advantages and beneficial effects of the present invention are as follows:
[0014] The present invention benefits from strengthening text features from multiple angles and deeply mining to associate phrase information with word information, so that it has stronger context understanding ability in the entity prediction process, and thus can obtain a higher F1-score, which is the harmonic mean of precision and recall.
[0015] The following is a detailed analysis of the advantages of steps 105-108: Step 105 obtains the glyph features, radical features and pinyin features based on the data after text segmentation, and further refines the features through a convolutional neural network to obtain a more representative feature representation. In this step, a character representation with Chinese characteristics is obtained, allowing the model to further assist in understanding the characters; Step 106 amplifies the important influential parts of the data through a graph attention network, and ultimately makes the node information contain the information of adjacent nodes through information transmission, thereby obtaining more representative node information data. In this step, the graph neural network enables characters to obtain potential vocabulary information, thereby locating potential character boundaries and eliminating ambiguity; among them, the information transmission between nodes and edges is realized through the multi-head attention mechanism, so that the node information contains edge information and the edge information also contains node information; step 107 is based on the ctb.50d.vec word vector, and a simple 2-gram method is used to construct clinically related phrases for the edge-connected vocabulary in the graph neural network. Because the constructed vocabulary is more inclined to the medical field, the vocabulary enhancement of Chinese medical NER is more targeted; step 108 is to connect the output of the pre-trained model FT-Bert, the joint feature embeddings and the output of the graph neural network as the final feature representation, and input it into the BiLSTM neural network. This step enables the model to achieve vocabulary enhancement while also achieving character-level enhancement, improving the model's understanding ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 It is an overall step diagram of a clinical named entity recognition method based on graph neural network multi-embedding combination of the present invention;
[0017] Figure 2 It is a flow chart of a clinical named entity recognition method based on graph neural network multi-embedding combination of the present invention;
[0018] Figure 3 It is a schematic diagram of the construction of the relationship between words and phrases by the graph neural network;
[0019] Figure 4This is the overall structure diagram of FTBert-GNN-BiLSTM-CRF adopted in the present invention. DETAILED DESCRIPTION
[0020] like Figure 1 and Figure 2 As shown, the present invention specifically comprises the following steps:
[0021] 101. Segment the input text, specifically, segment the tokens in the text through the pre-trained model vocabulary and obtain the initial weights through the Embedding layer of FT-Bert. Specifically, the FT-Bert model is further pre-trained based on Bert according to the medical text corpus data. In the process of extracting features, the self-attention mechanism is used to extract weights, and 15% of the characters are randomly masked in this process. Segment the words through Bert's tokenizer and vocabulary data, and obtain the weight initialization information through the Embedding layer of the pre-trained model.
[0022] 102.Truncate the text to the maximum length MAX_LEN, build batches, and build a dataset for training in batches.
[0023] 103. For the truncated data, since the length of each data in each batch is different, in order to make the length of each text data in a batch consistent, the longest length of the text in each batch is dynamically obtained, and the length of each text information is padded to the same length, which is the maximum text length of the current batch.
[0024] 104. The [CLS] tag indicates the beginning of a sentence, and the [SEP] tag indicates the end of a sentence, connecting text sentence pairs.
[0025] 105. Perform data enhancement on the text data and obtain the corresponding feature embedding, by adding the pinyin information (including pinyin and tone) and glyph information (glyph features and radical features) of each token in the text data to enhance the text data, specifically including extracting the text corresponding to the radical information, the text corresponding to the various glyph image information and the text corresponding to the pinyin and intonation information; extract the above features through the convolutional neural network, specifically implementing:
[0026]
[0027] Where p is the number of zero-padding layers, s is the stride, k is the convolution kernel size, w is the input matrix size, and w′ represents the feature map size.
[0028] 106. Construct a graph neural network for text, construct nodes, edges and global information, transfer information based on global reasoning relationships and local dependencies, and obtain clear entity connection relationships to learn the relationship between characters and phrase elements; Figure 3 As shown in the figure, the graph neural network is constructed by taking characters as node information, phrases as edge information, and nodes and the connection relationship between nodes as global information. The information transmission between characters and phrases is realized through the multi-head attention mechanism. The multi-head attention mechanism is used to transmit information and aggregate and update it to the node information, thereby mining the potential information between characters and words. In the process of information aggregation in the graph neural network, the weights extracted by the multi-head attention mechanism are expressed as:
[0029]
[0030]
[0031]
[0032] Attention (Q, K, V) is the weight of the self-attention mechanism; q, K, and V are the weighted word vectors obtained by multiplying the input word vector with three different weight matrices respectively; d k Indicates the dimension of the input vector. head i represents the calculation result of the i-th attention, W i q ,W i K ,W i V and W O express and The parameter mapping vector of Represents the weight of the multi-head attention mechanism.
[0033] 107. The base vocabulary set corresponding to the training set is constructed through the ctb.50d.vec word vector, and the 2-gram idea is used to construct phrases from the set B representing the beginning of each entity and the set I representing the content of the entity, so as to obtain the final phrase set.
[0034] Specifically, for dictionary construction, we first use the pre-trained word vector ctb.50d.vec as the initial text sequence representation; based on the phrase information in the word vector, we construct a corresponding phrase set for the characters in the text; on this basis, for each entity type, such as 'TREATMENT', we combine 'TREATMENT-B' with the characters in 'TREATMENT-I' through a 2-gram method and add them to the phrase set.
[0035] 108. Combine the embedding output of FT-Bert with the feature embeddings and the embedding obtained by graph neural network convergence update, and further extract text features through BiLSTM. BiLSTM can further strengthen the purpose of identifying entity boundaries, and can extract features based on the input text and the output of the previous LSTM, thereby avoiding the problem of long-term dependence. The single LSTM neuron in BiLSTM includes an input gate, an output gate, and a forget gate:
[0036] i t =σ(W xi x t +W hi h t-1 +b hi )
[0037] f t =σ(W xf x t +W hf h t-1 +b hf )
[0038] o t =σ(W xo x t +W ho h t-1 +b ho )
[0039] Where σ is the sigmoid function; W and b represent the weight matrix and bias vector parameters respectively, W xi ,W xf ,W xo Represent the weight parameters of the input gate, forget gate and output gate respectively, W hi ,W hf ,W ho They represent the weight parameters from the hidden layer to the input gate, forget gate, and output gate, respectively, and b hi ,b hf ,b ho They represent the bias parameters from the hidden layer to the input gate, forget gate, and output gate respectively; x t represents the input variable at time t; h t-1 represents the hidden state at time t-1; i t 、f t , o t Respectively represent the information of input gate, forget gate and output gate used to control the state of input memory cells; c t Represents the state of the memory cell at time t, which is calculated as follows:
[0040] c t =f tc t-1 +i c tanh(W xc x t +b ic +W hc h t-1 +b hc )
[0041] h t =o t tan(c t )
[0042] h t represents the hidden state at time t.
[0043] 109. Use CRF to decode the output of BiLSTM, correct the entity prediction, and obtain the final entity recognition result. Specifically, for example, if the current token is predicted to be 'CHECK-I', the next token will definitely not be 'CHECK-B'. CRF can consider the relationship between labels. In addition to outputting the maximum probability corresponding to each word, it can also avoid the situation where CHECK-I is later recognized as CHECK-B, thereby achieving the global optimal sequence. Input sentence X and the corresponding predicted label y, the decoding prediction process of CRF is as follows:
[0044]
[0045]
[0046] in, Represents t to label y t The score matrix, Represents the label y t to label y t+1 The transition score matrix; p(y|X) represents the probability of sequence X to label y; y* is the final output sequence of CRF. S(X,y) represents the CRF calculation score value, n represents the number of subsequences x, Y in the sequence x represents all possible tag sequences of sequence X, Indicates calculating predicted values.
[0047] Entity recognition process: Use the model file that has been retrained and fine-tuned in a specific field. Similar to the training process, input the unlabeled clinical medical text, and perform text processing such as word segmentation, batching, truncation, and sentence pairing in sequence. After passing through the multi-feature embedding layer, FT-BERT's transformer layer, GNN layer, BiLSTM, and output CRF layer, the final recognition result is obtained.
[0048] During the use of the model, it is mainly divided into the fine-tuning part and the entity recognition part. The fine-tuning part refers to the process of continuously modifying the network layer structure and adjusting the parameters of the neural units during the feature extraction and model construction through the training set data.
[0049] like Figure 4 In this example, a sentence "abdominal pain for 5 days with vomiting" is input, and a weight is assigned to each word through the Embedding layer of FT-Bert. The pinyin, glyph and radical enhancement information of the characters are obtained through CNN. The association and hidden information between characters and phrases are obtained through Attention-based GNN. The three are combined and input into the BiLSTM network for feature extraction, and finally the final entity recognition task is achieved through CRF. This example shows that the present invention can more effectively identify entities such as body parts, symptoms, diseases, examinations, and treatments in clinical medical texts.
[0050] The above is a detailed introduction to a clinical named entity recognition method based on graph neural network multi-embedding union provided by the present invention. This article explains the implementation methods of the principle set of the present invention, and the above description is used to help understand the core idea of the present invention. It should be pointed out that for ordinary researchers in this technical field, without departing from the principles of the present invention, the present invention can also be improved, modified and modified, and these improvements, modifications and variations also fall within the scope of protection of the claims of the present invention.
Claims
1. A clinical named entity recognition method based on graph neural network multi-embedding joint, characterized in that: The following steps are involved:
101. Perform word segmentation on each word in the input text corresponding to the vocabulary in the pre-training result, and assign initial weights through FT-Bert; 102. Divide the input text into batches and truncate it by MAX_LEN; 103. Fill each sentence of the input text to the same length in batches, and the length is the maximum length of the sentences in the current batch; 104. Connect all sentence pairs by using [CLS] at the beginning of the sentence and [SEP] at the end of the sentence; 105. Add a pinyin feature vector to the input text data, including pinyin and tone, add glyph features and radical features, and concatenate them as data enhancement vectors. Further extract features by using a convolutional neural network to obtain data enhancement multi-embedded joint features.
106. Construct a graph neural network, in which the information between nodes and edges is transmitted through a multi-head attention mechanism; 107. Construct word group information about clinical fields; 108. Combine FT-Bert, data-enhanced multi-embedded joint features, and the aggregation results based on graph neural network as the input of BiLSTM to further extract text features; 109. Use CRF to decode the output data of BiLSTM to achieve entity recognition.
2. According to claim 1, a clinical named entity recognition method based on graph neural network multi-embedding combination is characterized by: The FT-Bert model is further pre-trained based on Bert according to medical text corpus data. In the process of feature extraction, the self-attention mechanism is used to extract weights, and 15% of the characters are randomly masked in this process.
3. According to claim 1, a clinical named entity recognition method based on graph neural network multi-embedding combination is characterized by: The convolutional neural network is used to extract features, and the feature map calculation process is as follows: Where p is the number of zero-padding layers, s is the stride, k is the convolution kernel size, w is the input matrix size, and w′ represents the feature map size.
4. According to claim 1, a clinical named entity recognition method based on graph neural network multi-embedding combination is characterized by: In step 106, characters are used as node information, phrases are used as edge information, and nodes and the connection relationship between nodes are used as global information to construct a graph neural network. The information transmission between characters and phrases is realized through a multi-head attention mechanism. The information is transmitted through the multi-head attention mechanism and converged and updated to the node information to mine the potential information between characters and words.
5. A clinical named entity recognition method based on graph neural network multi-embedding combination according to claim 1 or 4, characterized in that: In the process of information aggregation by graph neural network, the weights extracted by multi-head attention mechanism are expressed as: head i =Attention(qW i q ,KW i K ,VW i V ) MultiAtt(q,K,V)=[head1;...;head k ]W O Attention(q,K,V) is the weight of the self-attention mechanism; q, K, and V are the weighted word vectors obtained by multiplying the input word vector with three different weight matrices respectively; d k and d v Respectively represent the dimensions of the input vectors q and V; head i represents the result of the i-th attention calculation, W i q , W i K , W i V and W O represents the parameter mapping vector, Represents the weight of the multi-head attention mechanism.
6. According to claim 1, a clinical named entity recognition method based on graph neural network multi-embedding combination is characterized by: Step 107 constructs word group information about the clinical field, constructs a base vocabulary set corresponding to the training set through the ctb.50d.vec word vector, and constructs phrases using 2-grams for the set B representing the beginning of each entity and the set I representing the content of the entity, thereby obtaining the final phrase set.
7. According to claim 1, a clinical named entity recognition method based on graph neural network multi-embedding combination, characterized in that: The step 108 uses BiLSTM to extract text features, and the process is as follows: i t =σ(W xi x t +b ii +W hi h t-1 +b hi ) f t =σ(W xf x t +b if +W hf h t-1 +b hf ) o t =σ(W xo x t +b io +W ho h t-1 +b ho ) c t =f t c t-1 +i c tanh(W xc x t +b ic +W hc h t-1 +b hc ) h t =o t tan(c t ) Where σ is the sigmoid function; W and b represent the weight matrix and bias vector parameters; x t represents the input variable at time t; h t-1 represents the hidden state at time t-1; i t 、f t , o t Respectively represent the information of input gate, forget gate and output gate used to control the state of input memory cells; c t represents the state of memory cells at time t; h t represents the hidden state at time t.
8. According to claim 1, a clinical named entity recognition method based on graph neural network multi-embedding combination is characterized by: In step 109, according to the input sentence X and the corresponding predicted label y, the decoding prediction process of CRF is as follows: in, represents the score matrix from t to label yt, Represents the transition score matrix from label yt to label yt+1; p(y|X) represents the probability of x to label y; y* is the final output sequence of CRF.