Biomedicine named entity recognition method based on cross Transform

Abstract

The invention belongs to the field of artificial intelligence and natural language processing, and particularly relates to a cross Transform-based biomedical named entity recognition method, which comprises the following steps of: constructing and training an entity recognition model, and inputting to-be-recognized data into the trained entity recognition model to obtain a recognition result; the entity recognition model comprises an input layer, a BioBERT embedding layer, a BiLSTM (Bilong Short Term Memory) feature extraction layer, an IDCNN (Inter-Decentralized Nearest Neighbor) feature extraction layer, a Cross-Transform feature fusion layer and a decoding layer; according to the invention, the entity identification accuracy can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and natural language processing, and in particular relates to a biomedical named entity recognition method based on a cross Transformer. Background Art

[0002] Named Entity Recognition (NER) refers to the process of automatically detecting and identifying entities with specific meanings from text data, such as names of people, places, and organization names, and annotating them. In recent years, the relevant literature data in the biomedical field has shown an exponential growth trend. This surge in data has provided a rich information resource for biomedical research. By using NER technology, researchers can automatically identify and extract named entities in text, such as disease names, drugs, genes, etc., so as to quickly extract key information involved in the literature. At the same time, it is also an indispensable key preprocessing step for performing subsequent more complex tasks such as biomedical literature retrieval and building biomedical question-answering systems.

[0003] Traditional named entity recognition methods are mainly based on rules, dictionaries or feature engineering. These methods usually require a lot of manual labeling work and the input of domain knowledge, which consumes a lot of manpower and time, and are difficult to cope with the challenges of diversity and large scale of biomedical literature. In recent years, researchers have turned to using deep learning methods to handle BioNER tasks. Deep learning models such as recurrent neural networks (RNN), long short-term memory networks (LSTM), bidirectional long short-term memory networks (BiLSTM), and models based on attention mechanisms such as Transformer can automatically learn feature representations and get rid of the reliance on manual feature engineering. These models can not only capture contextual information, but also effectively handle the complex relationships between words, thereby improving the accuracy of entity recognition. Compared with traditional machine learning methods, deep learning models have shown better generalization ability and higher entity recognition accuracy in the biomedical field. However, the feature expression of a single model is limited. Summary of the invention

[0004] In order to solve the problem that in the prior art, model features are usually fused by a simple splicing method, the correlation of feature information in the fusion process is ignored, resulting in low feature extraction accuracy, which in turn affects the accuracy of entity recognition, the present invention proposes a biomedical named entity recognition method based on a cross-Transformer, including constructing and training an entity recognition model, inputting the data to be recognized into the trained entity recognition model to obtain a recognition result; the entity recognition model includes an input layer, a BioBERT embedding layer, a BiLSTM feature extraction layer, an IDCNN feature extraction layer, a Cross-Transformer feature fusion layer, and a decoding layer.

[0005] The training process of the entity recognition model includes the following steps:

[0006] S1. Obtain a biomedical data set and use the BIO annotation method to annotate each text data in the biomedical data set;

[0007] S2. Send the annotated text data to the BioBERT embedding layer for word embedding to obtain a word vector sequence;

[0008] S3. Input the word vector sequence into the BiLSTM feature extraction layer to obtain the text temporal features;

[0009] S4. Input the word vector sequence into the IDCNN feature extraction layer to obtain the local features of the text;

[0010] S5. Input the text temporal features and the text local features into the Cross-Transformer layer for feature fusion to obtain a fused feature vector;

[0011] S6. Input the fused feature vector into the decoding layer to obtain the predicted label, calculate the loss value based on the predicted label and the true label, and back-propagate the training model until the model parameters converge.

[0012] Beneficial effects of the present invention:

[0013] This method constructs a named entity recognition model based on a cross Transformer, uses BiLSTM to capture the temporal features of the text, and uses IDCNN to extract the local features of the text. Through the feature fusion module, the temporal features are effectively combined with the local features to more comprehensively capture the semantic information of the text. This fusion method can effectively alleviate the existing methods that usually only fuse word features through splicing, reduce errors caused by missing context information or ignoring local features, and thus significantly improve the overall performance of named entity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of the method of the present invention;

[0015] Figure 2 It is a schematic diagram of the structure of the model of the present invention;

[0016] Figure 3 This is a schematic diagram of the LSTM unit structure of the present invention;

[0017] Figure 4 Schematic diagram of the Cross-Transformer feature fusion layer structure of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The present invention provides a biomedical named entity recognition method based on a cross-Transformer, comprising constructing and training an entity recognition model, and inputting data to be recognized into the trained entity recognition model to obtain a recognition result; the entity recognition model comprises an input layer, a BioBERT embedding layer, a BiLSTM feature extraction layer, an IDCNN feature extraction layer, a Cross-Transformer feature fusion layer, and a decoding layer.

[0020] like Figure 1 , Figure 2 As shown, the training process of the entity recognition model includes the following steps:

[0021] S1. Obtain a biomedical dataset and use the BIO annotation method to annotate each text data in the biomedical dataset.

[0022] Specifically, in step S1, the BIO annotation method is used to annotate each element of each text data in the biomedical data set, and BX is used to indicate that the element belongs to type X and is located at the beginning of the entity; IX is used to indicate that the element belongs to type X and is located in the middle or end of the entity; O is used to indicate that the element does not belong to any type.

[0023] S2. Send the annotated text data to the BioBERT embedding layer for word embedding to obtain a word vector sequence.

[0024] Specifically, the annotated text data is fed into the BioBERT embedding layer, and each character in the text data is mapped into a 768-dimensional feature vector through BioBERT's own encoding mechanism. The feature vector represents the meaning of the character in different contexts. Its core attention mechanism is:

[0025]

[0026] Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the Key vector; the Softmax function is a normalized exponential function.

[0027] S3. Input the word vector sequence into the BiLSTM feature extraction layer to obtain the text temporal features.

[0028] Specifically, the BiLSTM feature extraction layer is composed of multiple LSTM units, such as Figure 3 As shown, the processing of each LSTM unit includes:

[0029] i t =σ(W i [h t-1 ,x t ]+b i )

[0030]

[0031] o t =σ(W o [h t-1 ,x t ]+b o )

[0032] f t =σ(W f [h t-1 ,x t ]+b f )

[0033] Among them, i t represents the input gate, x t Indicates the current input, h t-1 represents the hidden layer state output by the previous LSTM unit, Indicates the temporary cell state, o t represents the output gate, f t represents the forget gate, σ represents the sigmoid function, tanh(·) represents the hyperbolic tangent function, W i , W C , W o , W f is the weight matrix, b i 、b C 、b o 、b f is the bias term.

[0034] Specific cell state updates and hidden layer state updates:

[0035]

[0036] in, and They represent the forward hidden layer state and backward hidden layer state of the current LSTM unit respectively. The two parts are concatenated to obtain the hidden layer state h of the current LSTM unit. t LSTM forward (·) represents the forward LSTM network, LSTM fbackward (·) represents a backward LSTM network.

[0037] S4. Input the word vector sequence into the IDCNN feature extraction layer to obtain the local features of the text.

[0038] Specifically, the IDCNN feature extraction layer is composed of multiple hole convolution layers stacked together, and the context information is obtained by increasing the convolution kernel layer by layer. The processing process of each hole convolution layer is expressed as

[0039] Y j =Conv(Y j-1 ,K,d j )

[0040] Among them, Y j represents the output of the jth hole convolution layer, K represents the convolution kernel, and d j represents the dilation size of the j-th atrous convolutional layer.

[0041] The outputs of multiple hole convolutional layers are concatenated and input into the activation function layer to obtain the final IDCNN output sequence, i.e., the local features of the text, which can be expressed as:

[0042] Y=W·f(Y1,Y2,...,Y N )

[0043] Among them, Y represents the local feature of the text, f represents the activation function, W represents the learnable parameter, and N represents the number of hole convolution layers.

[0044] S5. Input the text temporal features and text local features into the Cross-Transformer layer for feature fusion to obtain a fused feature vector.

[0045] Specifically, considering the limited expression of single model features, the model introduces a cross-transformer module in the feature fusion layer to dynamically integrate temporal and spatial semantic features, deepen the overall model's understanding of the multi-dimensional relationship of features, and effectively improve text processing performance. The Cross-Transformer layer includes two parallel transformer modules, each of which includes a multi-head attention layer, a first normalization layer, a feedforward network layer, and a second normalization layer, such as Figure 4 shown.

[0046] Step S5 specifically includes:

[0047] S51. Calculate the first query matrix Q according to the text temporal features b , the first bond matrix K b and the first value matrix V b , expressed as

[0048] K b =I·E b ,

[0049] Among them, E b represents the temporal features of the text, I represents the unit matrix, W b V is the weight matrix;

[0050] S52. Calculate the second query matrix Q based on the local features of the text r , the first bond matrix K r and the first value matrix V r , expressed as

[0051] Q r =W r Q ·E r , K r =I·E r , V r =W r V ·E r

[0052] Among them, E r represents the local features of the text, I represents the unit matrix, W r Q , W r V is the weight matrix;

[0053] S53. The second query matrix Q r , the first bond matrix K band the first value matrix V b Input a transformer module to get the first feature T b ; The first query matrix Q b , the first bond matrix K r and the first value matrix V r Input another transformer module to get the second feature T r .

[0054] In the transformer module of the present invention, in order to make the model pay more attention to key information, each attention head in the multi-head attention layer uses a differential attention mechanism. At the same time, in order to adapt to the problem of attention deviation of feature vectors of different models, a random attention matrix is ​​also introduced. The specific formula is as follows:

[0055]

[0056] Among them, Q r1 , Q r2 is the second query matrix Q r The two sets of feature vectors separated from b1 , K b2 is the first key matrix K b The two sets of feature vectors separated from , λ1 is a learnable scaling parameter used to balance the two attentions, and B1 is a randomly initialized parameter matrix that introduces a certain amount of randomness to enhance the robustness of the model.

[0057] Same reason

[0058]

[0059] Among them, Q b1 , Q b2 is the first query matrix Q b The two sets of feature vectors separated from r1 , K r2 is the second key matrix K r The two sets of feature vectors separated from , λ2 is a learnable scaling parameter used to balance the two attentions, B2 is a randomly initialized parameter matrix, d kb ,d kr Represents the vector dimension.

[0060] S54. Fuse the first feature and the second feature to obtain a fused feature vector T.

[0061] S6. Input the fused feature vector into the decoding layer to obtain the predicted label, calculate the loss value based on the predicted label and the true label, and back-propagate the training model until the model parameters converge.

[0062] Specifically, the decoding layer uses a CRF network. In order to reduce information loss, the first feature and the second feature are concatenated and input into the CRF network. The sequence with the largest output score is the predicted sequence of the label:

[0063]

[0064] A i,j =W i,j +b i,j

[0065]

[0066] in Indicates that word i corresponds to label y j Score, A i，j represents the transfer score from label i to label j. T represents the input feature, Represents the label y j The corresponding weight matrix is, Represents the label y j The corresponding bias, W i,j represents the weight transferred from label i to label j, b i,j It represents the bias of transferring from label i to label j, and M represents the sequence length. S represents the sum of the transfer score and the emission score. By comparing the sum of scores of different sequences, the label sequence with the highest score is found to complete the named entity recognition task.

[0067] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0068] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A biomedical named entity recognition method based on cross Transformer, characterized in that: Construct and train an entity recognition model, and input the data to be recognized into the trained entity recognition model to obtain a recognition result; the entity recognition model includes an input layer, a BioBERT embedding layer, a BiLSTM feature extraction layer, an IDCNN feature extraction layer, a Cross-Transformer feature fusion layer, and a decoding layer; The training process of the entity recognition model includes the following steps: S1. Obtain a biomedical data set and use the BIO annotation method to annotate each text data in the biomedical data set; S2. Send the annotated text data to the BioBERT embedding layer for word embedding to obtain a word vector sequence; S3. Input the word vector sequence into the BiLSTM feature extraction layer to obtain the text temporal features; S4. Input the word vector sequence into the IDCNN feature extraction layer to obtain the local features of the text; S5. Input the text temporal features and the text local features into the Cross-Transformer layer for feature fusion to obtain a fused feature vector; S6. Input the fused feature vector into the decoding layer to obtain the predicted label, calculate the loss value based on the predicted label and the true label, and back-propagate the training model until the model parameters converge.

2. According to claim 1, a biomedical named entity recognition method based on cross Transformer is characterized in that: The Cross-Transformer layer includes two parallel transformer modules, each of which includes a multi-head attention layer, a first normalization layer, a feedforward network layer, and a second normalization layer; step S5 specifically includes: S51. Calculate the first query matrix Q according to the text temporal features b , the first bond matrix K b and the first value matrix V b , expressed as Among them, E b represents the temporal features of the text, I represents the unit matrix, is the weight matrix; S52. Calculate the second query matrix Q based on the local features of the text r , the first bond matrix K r and the first value matrix V r , expressed as Among them, E r represents the local features of the text, I represents the unit matrix, is the weight matrix; S53. The second query matrix Q r , the first bond matrix K b and the first value matrix V b Input a transformer module to get the first feature; the first query matrix Q b , the first bond matrix K r and the first value matrix V r Input another transformer module to get the second feature; S54. Fuse the first feature and the second feature to obtain a fused feature vector.

3. According to claim 2, a biomedical named entity recognition method based on cross Transformer is characterized in that: The multi-head attention layer is processed using a differential attention mechanism that introduces a random parameter matrix, expressed as Among them, Q r1 , Q r2 is the second query matrix Q r The two sets of feature vectors separated from b1 , K b2 is the first key matrix K b The two sets of feature vectors separated from , λ1 is a learnable scaling parameter, B1 is a randomly initialized parameter matrix; Q b1 , Q b2 is the first query matrix Q b The two sets of feature vectors separated from r1 , K r2 is the second key matrix K r The two sets of feature vectors separated from , λ2 is a learnable scaling parameter, B2 is a randomly initialized parameter matrix, d kb d kr Represents the vector dimension.

4. According to claim 1, a biomedical named entity recognition method based on cross Transformer is characterized in that: In step S1, the BIO annotation method is used to annotate each element of each text data in the biomedical data set, and BX is used to indicate that the element belongs to type X and is located at the beginning of the entity; IX is used to indicate that the element belongs to type X and is located in the middle or end of the entity; O is used to indicate that the element does not belong to any type.

5. According to claim 1, a biomedical named entity recognition method based on cross Transformer is characterized in that: The BiLSTM feature extraction layer is composed of multiple LSTM units. The processing of each LSTM unit includes: i t =σ(W i [h t-1 ,x t ]+b i ) the t =σ(W o [h t-1 ,x t ]+b o ) f t =σ(W f [h t-1 ,x t ]+b f ) Among them, i t represents the input gate, x t Indicates the current input, h t-1 represents the hidden layer state, Indicates the temporary cell state, o t represents the output gate, f t represents the forget gate, σ represents the sigmoid function, tanh(·) represents the hyperbolic tangent function, W i , W C , W o , W f is the weight matrix, b i 、b C 、b o 、b f is the bias term.

6. A biomedical named entity recognition method based on cross Transformer according to claim 1, characterized in that: The IDCNN feature extraction layer is composed of multiple dilated convolutional layers stacked together. The processing process of each dilated convolutional layer is expressed as Y j =Conv(Y j-1 ,K,d j ) Among them, Y j represents the output of the jth hole convolution layer, K represents the convolution kernel, and d j represents the expansion size of the j-th hole convolutional layer; The outputs of multiple hole convolutional layers are concatenated and input into the activation function layer to obtain the local features of the text, which can be expressed as: Y=W·f(Y1,Y2,...,Y N ) Among them, Y represents the local feature of the text, f represents the activation function, W represents the learnable parameter, and N represents the number of hole convolution layers.

Images