Character convolution feature fusion biomedical entity recognition method and device

Abstract

The invention belongs to the field of natural language processing, and discloses a character convolution feature fusion biomedical entity recognition method and device. The method comprises the following steps: acquiring a training data set, extracting biomedical entity features in the training data set, capturing context features of a biomedical entity according to the extracted biomedical entity features, modeling biomedical entity character features by using a character convolutional neural network, and performing feature fusion to obtain biomedical entity character features. According to the technical scheme, modeling is carried out on the label context features through the conditional random field CRF, the labeling sequence is output, the accuracy of entity recognition in the biomedicine field is improved, and the problems that in the prior art, unstructured data is difficult to directly utilize, keyword retrieval is carried out through a search engine, and the difficulty of manual information screening is large are solved.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and in particular relates to a character convolution feature fusion biomedical entity recognition method and device. Background Art

[0002] In the past 20 years, research and technology in medicine, biology, pharmacy, nursing, biochemistry, biophysics, epidemiology and other fields related to biomedicine have developed by leaps and bounds, and a large number of results have been published in corresponding journals. The Medline database contains millions of academic documents related to biomedicine and life sciences. These documents come from journals, academic publications, conference papers and other credible medical literature sources around the world, and this growth trend is still increasing. The number of citations of documents in the Medline database from 2002 to 2022 is as follows: Figure 4 As shown in the figure, the number of citations in 2002 was 502,000, but in 2021 it exceeded 1 million, and even reached 1,369,600 in 2022. In 20 years, the number of citations has more than doubled. This also reflects the research enthusiasm in the biomedical field. Making full use of natural language processing technology to analyze and process massive amounts of text data can not only mine potential valuable information, but also be used to realize many intelligent scenarios. For example, gastric cancer, a serious malignant tumor, poses a great threat to the life and health of the people. Researchers, drug developers and clinical experts have conducted a lot of scientific research, and the research results have been published in various journals. As of September 2023, the search results for "stomach cancer" as the keyword in the PubMed search engine exceeded 130,000. It has become difficult to read, screen, and extract valuable information manually. At the same time, a large amount of data has been generated in the process of medical clinical development, mainly including a large number of researchers' scientific research data, hospital daily consultation records of patients, and experimental clinical data in the research and development process of related pharmaceutical companies. These data are almost all in unstructured form and are difficult to be used directly. Natural language processing technology must be used to more efficiently utilize biomedical knowledge, and to analyze and explore the potential relationships and laws between biomedical entities, so as to perform knowledge discovery. Therefore, the application of natural language processing technology in the field of biomedical text mining has become an important solution, among which the main tasks are biomedical entity recognition, relationship extraction, knowledge graph construction, etc. The present invention mainly focuses on the problem of biomedical entity recognition, which is the most basic and important problem among these tasks. The accuracy of recognition will directly affect the performance of subsequent tasks. Summary of the invention

[0003] In order to solve the problems of the prior art and improve the accuracy of entity recognition in the biomedical field, the present application provides a character convolution feature fusion biomedical entity recognition method and device.

[0004] In the first aspect, a method for character convolution feature fusion biomedical entity recognition is provided, which adopts the following technical solutions:

[0005] Get the training dataset:

[0006] De-duplicate the original corpus and create an index of words and characters;

[0007] Extract biomedical entity features from the training dataset:

[0008] The biomedical entity features are composed of word features and character features; the word features are vectorized using a word embedding model; and the character features are extracted via a character convolutional neural network;

[0009] Based on the extracted biomedical entity features, the contextual features of the biomedical entities are captured:

[0010] The context features are captured by a bidirectional long short-term memory (LSTM) network, which performs linear cyclic information transfer.

[0011] Extract label context features and output label sequence:

[0012] For the labels output by the bidirectional long short-term memory (LSTM) network, the label context features are modeled through the conditional random field (CRF) to output the label sequence.

[0013] Furthermore, the character convolutional neural network is formed by stacking convolutional networks; the convolutional network is divided into three layers, namely, a convolutional layer, a pooling layer, and a random dropout layer from bottom to top.

[0014] Furthermore, the character convolutional neural network is composed of three stacked convolutional networks, and the convolution kernel sizes of the corresponding convolutional layers are 2, 3, and 4 respectively.

[0015] Furthermore, feature selection is performed on the pooling layer to reduce the number of features; the specific method is:

[0016] Assume that the input feature map group of the pooling layer is X∈R M , the output is Y∈R M′ , for each feature map X, divide it into regions X m , 1≤m≤M′; downsample each region to get a value as a summary of the region; use maximum pooling to take the maximum value Y of all neurons in a region m ,

[0017]

[0018] where x i For region R m The activation value of each neuron in .

[0019] Furthermore, based on the extracted biomedical entity features, the method used to capture the contextual features of the biomedical entity is as follows:

[0020] The bidirectional long short-term memory LSTM network uses a new internal state c t Specializes in linear cyclic information transmission, and

[0021] Output information to the external state h of the hidden layer t ;

[0022]

[0023] h t =o t ⊙tanh(c t )

[0024] where f t ,i t , o t are three gates to control the path of information transmission; ⊙ is the product of vector elements; c t-1 is the memory unit of the previous moment; is the candidate state obtained by the hyperbolic tangent function tanh(), The value of the current word input x t And the input weight matrix W c , the external state h of the previous word hidden layer t-1 And the memory weight matrix U c , bias vector b c Sure;

[0025]

[0026] At each time t, the internal state c of the bidirectional long short-term memory LSTM network t Record the historical information up to the current time.

[0027] Build long dependencies.

[0028] Furthermore, the label context features are modeled through the conditional random field CRF, and the method used to output the label sequence is as follows:

[0029] Based on the output layer of the bidirectional long short-term memory LSTM network, the output layer of the bidirectional long short-term memory LSTM network is a fully connected layer. Each neuron is connected to all neurons in the previous layer and outputs the word category probability. The conditional random field CRF considers the dependency between labels based on the initial probability distribution of the labels generated by the fully connected layer to obtain a consistent label sequence.

[0030] Let X and Y be random variables, P(Y|X) is the conditional probability distribution of Y given X, if the random variable Y forms a Markov random field represented by an undirected graph G = (V, E), that is

[0031] P(Y v |X,Y w ; w≠v)=P(Y v |X,Y w ; w~v)

[0032] Where v represents any node, w represents all nodes except v, and w~v represents all nodes directly connected to v.

[0033] Then the conditional probability distribution P(Y|X) is a conditional random field;

[0034] The conditional random field is a linear conditional random field, X and Y have the same graph structure; X represents the input observation sequence, X = {x1, ..., x n}; Y represents the corresponding output tag sequence, Y = {y1,…,y n}, i is the number of each output sequence

[0035] positions;

[0036]

[0037] in

[0038]

[0039] Where exp() is the exponential function; t k is the transition feature function from one label to another, k is the number of the transition feature function, and λ k is the transfer characteristic function t k The weight of , if there is a connection from one tag to another in the training set, then t k =1, otherwise t k =0;s l is the state feature function of the current word input and the current tag, l is the number of the state feature function, μ l is the state characteristic function s l The weight of , if there is a training sample of the current word input to the corresponding label, then sl =1, otherwise s l =0; t k and l The value of is 1 or 0, so the conditional random field is completely determined by λ k and μ l Sure.

[0040] In a second aspect, a device for character convolution feature fusion biomedical entity recognition is provided, wherein the device is used to implement the method for character convolution feature fusion biomedical entity recognition described in the first aspect, and adopts the following technical solution:

[0041] The device for character convolution feature fusion biomedical entity recognition includes:

[0042] Encoding module: used to obtain the training data set, including deduplication of the original corpus and establishment of word and character indexes; used to extract biomedical entity features in the training data set, including word features and character features, where character features are extracted via a character convolutional neural network;

[0043] Context feature module: It consists of a bidirectional long short-term memory (LSTM) network to capture the context features of biomedical entities.

[0044] Decoding module: For the labels output by the bidirectional long short-term memory (LSTM) network, the label context features are modeled through the conditional random field (CRF) and the label sequence is output.

[0045] Furthermore, the character convolutional neural network is composed of three stacked convolutional networks. The character convolutional neural network consists of three layers in total, namely, convolutional layer, pooling layer, and random dropout layer from bottom to top.

[0046] Furthermore, the pooling layer performs feature selection to reduce the number of features.

[0047] Furthermore, when the decoding module uses the conditional random field CRF to model the label context features, it is based on the output layer of the bidirectional long short-term memory LSTM network; the output layer of the bidirectional long short-term memory LSTM network is a fully connected layer, each neuron is connected to all neurons in the previous layer, and outputs the word category probability; the conditional random field CRF extracts the label context features based on the initial probability distribution of the label generated by the fully connected layer.

[0048] The beneficial effects of the present invention are as follows:

[0049] In the existing technology, unstructured data is difficult to be used directly, and it is difficult to rely on search engines for keyword retrieval and manual screening of information.

[0050] The present invention adds a convolutional neural network to the basic neural network language model to extract the character structure features of biomedical entities, and continues to use the conditional random field CRF to further extract the contextual dependencies of labels in the decoding process, and finally outputs a more consistent label sequence, thereby improving the accuracy and efficiency of entity recognition in the biomedical field, making full use of biomedical knowledge, analyzing and exploring the potential relationships and laws between biomedical entities, and performing biomedical entity recognition, relationship extraction, knowledge graph construction, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 These are steps of the character convolution feature fusion biomedical entity recognition method according to an embodiment of the present invention;

[0052] Figure 2 It is a character convolution feature fusion biomedical entity recognition model of an embodiment of the present invention;

[0053] Figure 3 The character convolutional neural network of the embodiment of the present invention extracts character structure features;

[0054] Figure 4 It is the citation status of the literature in the Medline database in the past 20 years;

[0055] Figure 5 is the embedding vector of the word triglyceride “tryglycerid” in an embodiment of the present invention;

[0056] Figure 6 is the character convolution result of the word “Immunohistochemical” in an embodiment of the present invention;

[0057] Figure 7 Word length statistics of the experimental dataset word dictionary. DETAILED DESCRIPTION

[0058] The present invention is further described below.

[0059] The present invention provides a method for character convolution fusion biomedical entity recognition, the steps are as follows: Figure 1 shown.

[0060] 1. Get the training data set

[0061] 1-1. Create a word and character dictionary

[0062] The corpus is selected as the training set, the repeated words and characters appearing in the training set are removed, and the word dictionary and character dictionary are established.

[0063] This example is implemented in three Biocreative biomedical corpora BC2GM, BC4Chemd, BC5CDR and one NCBI corpus NCBI-Disease.

[0064] The number of words and characters in each corpus is shown in Table 1. The BC4Chemd corpus is the largest and has the largest number of words, 40,499, followed by the BC2GM corpus, which has 29,776 words. The other two disease-related corpora are of similar size in terms of word size, and the number of characters in the four corpora is comparable.

[0065] Table 1 Number of words and characters in different corpora

[0066] Number of words Number of characters BC2GM 29776 87 BC4Chemd 40499 94 BC5CDR 9983 82 NCBI-Disease 9285 86

[0067] 1-2. Create an index of words and characters

[0068] Look up the dictionary for each word and character in the training corpus, and convert the words and characters into the corresponding indexes in the dictionary. For example, the word and character indexes of the first sentence in the BC2GM training set are shown in Table 2. The words item in the table is the first sentence in the training set, and the word_ids item is the index of each word in the sentence in the word dictionary. For the convenience of display, the maximum length of the sequence is set to 50, so the last two 0s are padding strings. char_ids[0]

[0069] The item is the character index of the word "Immunohistochemical". The maximum length of the word is set to 10 here, so truncation is performed and only the first 10 characters are displayed. mask is the identifier of the sentence. There is only one sentence in this sample, so all except the last two padding strings are 1. If there are two sentences, the second sentence is identified as 2. label_ids is the label of the word. 0, 1, 2, 3, and 4 correspond to PAD, UNK, B, I, and O respectively. PAD is the abbreviation of the word padding, which means the padding category. Because in actual calculation, long sentences are truncated and short sentences are padded with some special words to the same length for calculation, there will be some padding-type word categories. UNK is the abbreviation of the word unknow, which means the unknown category. B is the abbreviation of the word begin, which means the beginning category of the entity category. Because entities often contain more than one word or character, I is the abbreviation of the word inside, which means the entity internal category. O is the abbreviation of the word outside, which means the entity external category.

[0070] Table 2 Word and character index of the first sentence of the BC2GM training set

[0071]

[0072] 2. Extracting Biomedical Entity Features from Training Datasets

[0073] Biomedical entity features are composed of word features and character features. Word features are represented by word vectors, and character features are represented by final character vectors, such as Figure 3 shown.

[0074] Among them, the word features use the word embedding model to realize word vectorization: In this embodiment, the word embedding directly calls the Word2Vec model. Considering that the vector dimension of each word embedding will increase after character convolution, the length of the vector embedded in the word itself is only set to 100. Google's word embedding model Word2Vec file word2vec_vi_words_100dims is a vector dictionary with words as keywords and values ​​of length 100. The number of words is 1587507. The trained model is used to map words to vectors of length 100. Call the Word2Vec model to assign a vector representation to each word in the word dictionary. For example, the embedding vector of the word triglyceride "tryglycerid" is as follows Figure 5 As shown in Figure 2, the character embedding in the character dictionary is done using random initialization.

[0075] Among them, the character features are extracted by the character convolutional neural network, and the final character vector length of the character convolutional neural network output is 50. The character convolutional neural network consists of 3 layers, from bottom to top, namely the convolution layer, the pooling layer, and the random drop layer. There are 3 such convolutional neural networks, corresponding to the convolution kernel sizes of 2, 3, and 4 of the convolution layer. The 3 convolutional neural networks are stacked to form a large convolutional neural network. When the convolution kernel size is 3, there are 3 filter parameters, namely w1, w2, and w3, which are convolved with the initial character vector after the output y at position time t. t for

[0076] y t =w1×x t-1 +w2×x t +w3×x t+1

[0077] Among them, x t-1 represents the input at the previous time point, with a weight of w1; x t represents the input at the current time point, with a weight of w2; x t+1 Represents the input at the next time point, with a weight of w3.

[0078] The pooling layer performs feature selection, reduces the number of features, and thus reduces the number of parameters. Assume that the input feature map group of the pooling layer is X∈R M , the output is Y∈RM′ , for each feature map X, divide it into many regions X m , 1≤m≤M′, these regions may or may not overlap. Downsample each region to get a value as a summary of this region, use maximum pooling, and take the maximum value Y of all neurons in a region m :

[0079]

[0080] where x i For region R m The activation value of each neuron in .

[0081] Extract character features. Take the word "Immunohistochemical" as an example. The output after convolution with a convolution kernel of length 2 is as follows Figure 6 As shown in the figure, for the convenience of display, the maximum length of the word is set to 10, and the excess part is truncated. Each character is initialized to a vector of length 6 and convolved with the convolution kernel, such as the vector of the character "I" The final convolution result is [4 4 7 3 3], and the character convolution result of the word after maximum pooling is 7. In addition, two convolution kernels with lengths of 3 and 4 are set, and the convolution process is similar to the convolution process with a convolution kernel length of 2.

[0082] The final character vector of each word after passing through the character convolutional neural network is 50 in length, which is then concatenated with the word vector of the word into a vector with a length of 150 and input into the bidirectional long short-term memory LSTM network.

[0083] 3. Capturing Contextual Features of Biomedical Entities

[0084] The context features are captured by a bidirectional long short-term memory (LSTM) network, which performs linear cyclic information transfer.

[0085] The context feature module is mainly composed of a bidirectional long short-term memory (LSTM) network, which mainly captures the entity context dependencies in biomedical texts and solves the gradient explosion and gradient vanishing problems of simple recurrent neural networks.

[0086] The LSTM network uses a new internal state c t Specializes in linear cyclic information transmission and outputs information to the external state h of the hidden layer t .

[0087]

[0088] h t =o t⊙tanh(c t )

[0089] where f t ,i t , o t are three gates to control the path of information transmission; ⊙ is the product of vector elements; c t-1 is the memory unit of the previous moment; is the candidate state obtained by the hyperbolic tangent function tanh(), The value of the current word input x t And the input weight matrix W c , the external state h of the previous word hidden layer t-1 And the memory weight matrix U c , bias vector b c Sure;

[0090]

[0091] At each time t, the internal state c of the LSTM network t The historical information up to the current moment is recorded, so that the entire network can establish a long dependency relationship.

[0092] Set the hyperparameters of the LSTM network. The hidden dimension of the unidirectional LSTM network is 175, and the hidden dimension of the bidirectional LSTM network is 350. The output linear layer has an input dimension of 350 and an output dimension of 5. The next layer is the CRF layer, and the output dimension is also 5, which corresponds to the conditional probabilities of the five types of labels, namely PAD, UNK, B, I, and O. PAD is the abbreviation of padding, which means the filling category. In actual calculation, long sentences are truncated and short sentences are filled with some special words to the same length for calculation, so there will be some filling-type word categories. UNK is the abbreviation of unknow, which means unknown category. B is the abbreviation of begin, which means the beginning category of the entity category, because entities often contain more than one word or character. I is the abbreviation of inside, which means the internal category of the entity. O is the abbreviation of outside, which means the external category of the entity.

[0093] 4. Extract label context features and output label sequence

[0094] For the labels output by the character convolutional neural network, the label context features are modeled through the conditional random field CRF and the label sequence is output.

[0095] In the decoding process, based on the output layer of the bidirectional long short-term memory (LSTM) network, the conditional random field (CRF) is used again to model the context of the label sequence and capture the label context features.

[0096] Based on the output layer of the bidirectional long short-term memory LSTM network, the output layer of the bidirectional long short-term memory LSTM network is a fully connected layer. Each neuron is connected to all neurons in the previous layer and outputs the word category probability. The conditional random field CRF considers the dependency between labels based on the initial probability distribution of labels generated by the fully connected layer to obtain a consistent label sequence. This constitutes a joint training model of a neural network and a conditional random field.

[0097] Let X and Y be random variables, P(Y|X) is the conditional probability distribution of Y given X, if the random variable Y forms a Markov random field represented by an undirected graph G = (V, E), that is

[0098] P(Y v |X,Y w ; w≠v)=P(Y v |X,Y w ; w~v)

[0099] Where v represents any node, w represents all nodes except v, w~v represent all nodes directly connected to v, then the conditional probability distribution P(Y|X) is a conditional random field.

[0100] The conditional random field is a linear conditional random field, X and Y have the same graph structure; X represents the input observation sequence, X = {x1, ..., x n}; Y represents the corresponding output tag sequence, Y = {y1,…,y n}, i is each position of the output sequence;

[0101]

[0102] in

[0103]

[0104] Where exp() is the exponential function; t k is the transition feature function from one label to another, k is the number of the transition feature function, and λ k is the transfer characteristic function t k The weight of , if there is a connection from one tag to another in the training set, then t k =1, otherwise t k =0;s l is the state feature function of the current word input and the current tag, l is the number of the state feature function, μ l is the state characteristic function s l The weight of , if there is a training sample of the current word input to the corresponding label, then sl =1, otherwise s l =0; t k and l The value of is 1 or 0, so the conditional random field is completely determined by λ k and μ l Sure.

[0105] The following is an analysis of factors affecting the recognition performance of this application:

[0106] The experiment verifies the impact of the number of convolution kernels and the maximum word length on biomedical entity recognition. Through comparative analysis, a set of standard experimental configurations are obtained to compare the effectiveness of the models. Before setting the maximum word length, the lengths of all words in the word dictionary of the experimental dataset are first counted, such as Figure 7 As shown in the figure, the word length distribution of the four data sets shows that the word lengths have similar distribution curves, mainly concentrated in the area within the length of 20. Therefore, the experiment sets the maximum word length to 20, and words exceeding this length are truncated, and words less than this length are filled.

[0107] Another parameter that affects character feature extraction is the number of convolution kernels. The experiment set three different numbers of convolution kernels as shown in Table 3, where the number of convolution kernels is 0, which means no character convolution. Table 3 also sets a comparison between the unidirectional long short-term memory LSTM network and the bidirectional long short-term memory LSTM (abbreviated as BiLSTM). A total of 5 control experiments are set in the table, which are compared with the standard group.

[0108] Table 3 Experimental parameter configuration

[0109] Experimental parameters standard Control group 1 Control group 2 Control group 3 Control group 4 Control group 5 Number of convolution kernels 32 32 0 16 48 32 Maximum word length 20 10 20 20 20 20 Contextual feature layer BiLSTM BiLSTM BiLSTM BiLSTM BiLSTM LSTM

[0110] Effect of word length on performance:

[0111] The comparison between the control group 1 and the standard group shows that the overall recognition performance slightly decreases after the maximum word length decreases, which is particularly obvious in the data sets BC4Chemd and BC5CDR-Chemical. As shown in Table 4, the comprehensive evaluation index F1 index decreases by 7.38% and 5.03% respectively. The experiment shows that under the condition that the number of convolution kernels remains unchanged, the character convolution is more capable of capturing the characteristics of characters when the maximum word length is 20, which is consistent with the previous statistics of word length.

[0112] Table 4 Effect of word length on recognition performance

[0113]

[0114] The impact of the number of convolution kernels on recognition performance:

[0115] The experiment set 4 different numbers of convolution kernels, 0, 16, 32, and 48, corresponding to control group 2, control group 3, labeling group, and control group 4, respectively. The experimental results of control group 2, control group 3, and control group 4 are shown in Tables 5 and 6. The data in the table show that when the number of convolution kernels is 0, that is, there is no character convolution of the convolutional neural network to extract character features, it will have a great impact on the recognition performance, and the accuracy, recall rate, and F1 index will drop significantly. When the number of convolution kernels is 16, 32, and 48, the recognition performance does not change much. As the number of convolution kernels increases, the recognition performance will tend to be stable.

[0116] Table 5 The impact of the number of convolution kernels on recognition performance

[0117]

[0118] Table 6 The impact of the number of convolution kernels and unidirectional long short-term memory LSTM network on recognition performance

[0119]

[0120] The impact of unidirectional long short-term memory LSTM network on recognition performance:

[0121] When the maximum word length is 20 and the number of convolution kernels is 32, the context feature module is set to a unidirectional LSTM network, and the experimental results are shown in the results of control group 5 in Table 6. The data show that under the condition of the unidirectional LSTM network, the performance is slightly reduced compared with the experimental results of the standard group, that is, the ability of the unidirectional LSTM to capture the context features of biomedical texts will be slightly reduced.

[0122] Based on the above experimental results, the maximum word length of 20, the number of convolution kernels of 32, and the bidirectional long short-term memory LSTM network conditions of the context feature module in the experimental setting are used as the experimental standard group to verify the effectiveness of the model.

[0123] The present invention also provides a character convolution feature fusion biomedical entity recognition device, including an encoding module, a context feature module and a decoding module, such as Figure 2 As shown, where:

[0124] The encoding module is used to obtain the training data set, including deduplication processing of the original corpus and establishment of word and character indexes; extracting biomedical entity features in the training data set, including word features and character features, where word features are vectorized using a word embedding model: word embedding directly calls the Word2Vec model. Considering that character convolution will increase the vector dimension of each word embedding, the length of the vector embedded in the word itself is only set to 100. The final output vector length of the character convolutional neural network is 50. Google's word embedding model Word2Vec file word2vec_vi_words_100dims is a vector dictionary with words as keywords and values ​​of length 100. The number of words is 1587507. The trained model is used to map words to vectors of length 100. Call the Word2Vec model to assign a vector representation to each word in the word dictionary. For example, the embedding vector of the word triglyceride "tryglycerid" is as follows Figure 5 As shown. The character embedding in the character dictionary is completed using random initialization. Character features are extracted through a character convolutional neural network: the character convolutional neural network consists of 3 layers, from bottom to top, they are convolution layer, pooling layer, and random drop layer. There are 3 such convolutional neural networks, corresponding to the convolution kernel sizes of 2, 3, and 4 in the convolution layer. Three convolutional neural networks are stacked to form a large convolutional neural network. When the convolution kernel size is 3, there are 3 filter parameters, namely w1, w2, and w3, which are convolved with the output y at position t after the initial character vector. t for:

[0125] y t =w1×x t-1 +w2×x t +w3×x t+1

[0126] Among them, x t-1 represents the input at the previous time point, with a weight of w1; x t represents the input at the current time point, with a weight of w2; x t+1 Represents the input at the next time point, with a weight of w3.

[0127] The pooling layer performs feature selection, reduces the number of features, and thus reduces the number of parameters. Assume that the input feature map group of the pooling layer is X∈R M , the output is Y∈R M′ , for each feature map X, divide it into many regions X m , 1≤m≤M′, these regions may or may not overlap. Downsample each region to get a value as a summary of this region. Here, the maximum pooling is used to take the maximum value Y of all neurons in a region. m,

[0128]

[0129] where x i For region R m The activation value of each neuron in .

[0130] The context feature module consists of a bidirectional long short-term memory (LSTM) network to capture the context features of biomedical entities.

[0131] The bidirectional long short-term memory LSTM network uses a new internal state c t Specializes in linear cyclic information transmission and outputs information to the external state h of the hidden layer t .

[0132]

[0133] h t =o t ⊙tanh(c t )

[0134] Among them, f t ,i t , o t are three gates to control the path of information transmission; ⊙ is the product of vector elements; c t-1 is the memory unit of the previous moment; is the candidate state obtained by the hyperbolic tangent function tanh(), The value of the current word input x t And the input weight matrix W c , the external state h of the previous word hidden layer t-1 And the memory weight matrix U c , bias vector b c Sure;

[0135]

[0136] At each time t, the internal state c of the LSTM network t The historical information up to the current moment is recorded, so that the entire network can establish a long dependency relationship.

[0137] The decoding module models the label context features of the labels output by the bidirectional long short-term memory LSTM network through the conditional random field CRF, and finally outputs the label sequence. The output label sequence is as follows Figure 2 As shown in y1 to y4 in the decoding module. In the decoding process, based on the output layer of the bidirectional long short-term memory LSTM network, the conditional random field CRF is used again to model the context of the label sequence and capture the label context features.

[0138] The output layer of the bidirectional long short-term memory LSTM network is a fully connected layer, where each neuron is connected to all neurons in the previous layer and outputs word category probabilities. The CRF layer considers the dependencies between labels based on the initial probability distribution of labels generated by the fully connected layer to obtain a more consistent label sequence. This constitutes a joint training model of a neural network and a conditional random field.

[0139] Let X and Y be random variables, P(Y|X) is the conditional probability distribution of Y given X, if the random variable Y forms a Markov random field represented by an undirected graph G = (V, E), that is

[0140] P(Y v |X,Y w ; w≠v)=P(Y v |X,Y w ; w~v)

[0141] Where v represents any node, w represents all nodes except v, w~v represent all nodes directly connected to v, then the conditional probability distribution P(Y|X) is a conditional random field.

[0142] The conditional random field is a linear conditional random field, and X and Y have the same graph structure. X represents the input observation sequence, X = {x1, ..., x n}; Y represents the corresponding output tag sequence, Y = {y1,…,y n}, i is each position of the output sequence.

[0143]

[0144] in

[0145]

[0146] Where exp() is the exponential function; t k is the transition feature function from one label to another, k is the number of the transition feature function, and λ k is the transfer characteristic function t k The weight of , if there is a connection from one tag to another in the training set, then t k =1, otherwise t k =0;s l is the state feature function of the current word input and the current tag, l is the number of the state feature function, μ l is the state characteristic function s l The weight of , if there is a training sample of the current word input to the corresponding label, then s l =1, otherwise s l =0; tk and l The value of is 1 or 0, so the conditional random field is completely determined by λ k and μ l Sure.

[0147] Although the present invention has been disclosed as above with preferred embodiments, the embodiments are not intended to limit the present invention. Any equivalent changes or modifications made without departing from the spirit and scope of the present invention also belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the content defined by the claims of this application.

Claims

1. A method for character convolution feature fusion biomedical entity recognition, characterized in that: The steps include: Get the training dataset: De-duplicate the original corpus and create an index of words and characters; Extract biomedical entity features from the training dataset: The biomedical entity features are composed of word features and character features; the word features are vectorized using a word embedding model; and the character features are extracted via a character convolutional neural network. Based on the extracted biomedical entity features, the contextual features of the biomedical entities are captured: Context Features are captured by a bidirectional long short-term memory (LSTM) network, which performs linear cyclic information transfer; Extract label context features and output label sequence: For the labels output by the bidirectional long short-term memory (LSTM) network, the label context features are modeled through the conditional random field (CRF) to output the label sequence.

2. The method for character convolution feature fusion biomedical entity recognition according to claim 1, characterized in that: The character convolutional neural network is composed of stacked convolutional networks; the convolutional network is divided into three layers, which are convolutional layer, pooling layer and random drop layer from bottom to top.

3. The method for character convolution feature fusion biomedical entity recognition according to claim 2, characterized in that: The character convolutional neural network is composed of three stacked convolutional networks, and the convolution kernel sizes of the corresponding convolutional layers are 2, 3, and 4 respectively.

4. The method for character convolution feature fusion biomedical entity recognition according to claim 3, characterized in that: Perform feature selection on the pooling layer to reduce the number of features; the specific method is: Assume that the input feature map group of the pooling layer is X∈R M , the output is Y∈R M′ , for each feature map X, divide it into regions X m , 1≤m≤M′; downsample each region to get a value as a summary of the region; use maximum pooling to take the maximum value Y of all neurons in a region m , where x i For region R m The activation value of each neuron in .

5. The method for character convolution feature fusion biomedical entity recognition according to claim 1, characterized in that: Based on the extracted biomedical entity features, the following method is used to capture the contextual features of biomedical entities: The bidirectional long short-term memory LSTM network uses a new internal state c t Specializes in linear cyclic information transmission and outputs information to the external state h of the hidden layer t ; h t =o t ⊙tanh(c t ) where f t ,i t , o t There are three gates to control the path of information transmission; ⊙ is the product of vector elements; c t-1 is the memory unit of the previous moment; is the candidate state obtained by the hyperbolic tangent function tanh(), The value of the current word input x t And the input weight matrix W c , the external state h of the previous word hidden layer t-1 And the memory weight matrix U c , bias vector b c Sure; At each time t, the internal state c of the bidirectional long short-term memory LSTM network t Record historical information up to the current moment and establish long-term dependencies.

6. The method for character convolution feature fusion biomedical entity recognition according to claim 1, characterized in that: The label context features are modeled through the conditional random field CRF. When the label sequence is output, the method used is as follows: Based on the output layer of the bidirectional long short-term memory LSTM network, the output layer of the bidirectional long short-term memory LSTM network is a fully connected layer. Each neuron is connected to all neurons in the previous layer and outputs the word category probability. The conditional random field CRF considers the dependency between labels based on the initial probability distribution of the labels generated by the fully connected layer to obtain a consistent label sequence. Let X and Y be random variables, P(Y|X) is the conditional probability distribution of Y given X, if the random variable Y forms a Markov random field represented by an undirected graph G = (V, E), that is P(Y v |X,Y w ;w≠v)=P(Y v |X,Y w ;w~v) Where v represents any node, w represents all nodes except v, w~v represent all nodes directly connected to v, then the conditional probability distribution P(Y|X) is a conditional random field; The conditional random field is a linear conditional random field, X and Y have the same graph structure; X represents the input observation sequence, X = {x1, ..., x n }; Y represents the corresponding output tag sequence, Y = {y1,…,y n }, i is each position of the output sequence; in Where exp() is the exponential function; t k is the transition feature function from one label to another, k is the number of the transition feature function, and λ k is the transfer characteristic function t k The weight of , if there is a connection from one tag to another in the training set, then t k =1, otherwise t k =0;s l is the state feature function of the current word input and the current tag, l is the number of the state feature function, μ l is the state characteristic function s l The weight of , if there is a training sample of the current word input to the corresponding label, then s l =1, otherwise s l =0; t k and l The value of is 1 or 0, so the conditional random field is completely determined by λ k and μ l Sure.

7. A device for character convolution feature fusion biomedical entity recognition, characterized in that: The device is used to implement the character convolution feature fusion biomedical entity recognition method described in claim 1, and the device includes: Encoding module: used to obtain the training data set, including deduplication of the original corpus and establishment of word and character indexes; used to extract biomedical entity features in the training data set, including word features and character features, where character features are extracted via a character convolutional neural network; Context feature module: It is composed of a bidirectional long short-term memory (LSTM) network, which captures the context features of biomedical entities. Decoding module: It models the label context features of the labels output by the bidirectional long short-term memory (LSTM) network through the conditional random field (CRF) and outputs the label sequence.

8. The device for character convolution feature fusion biomedical entity recognition according to claim 7, characterized in that: The character convolutional neural network is composed of three stacked convolutional networks. The character convolutional neural network consists of three layers in total, from bottom to top: convolutional layer, pooling layer, and random drop layer.

9. The device for character convolution feature fusion biomedical entity recognition according to claim 8, characterized in that: The pooling layer performs feature selection and reduces the number of features.

10. The device for character convolution feature fusion biomedical entity recognition according to claim 9, characterized in that: When the decoding module uses the conditional random field CRF to model the label context features, it is based on the output layer of the bidirectional long short-term memory LSTM network; the output layer of the bidirectional long short-term memory LSTM network is a fully connected layer, each neuron is connected to all neurons in the previous layer, and outputs the word category probability; the conditional random field CRF extracts the label context features based on the initial probability distribution of the label generated by the fully connected layer.

Images