A method of information extraction based on deep learning

Abstract

The present invention relates to the field of information processing technology, and in particular to a method for information extraction based on deep learning, including creating a model, the model including an entity recognition module and an entity relationship extraction module, wherein the entity recognition module is composed of a CRF added on the basis of a Bi-LSTM, and the entity relationship extraction module is composed of an attention mechanism added on the basis of a Bi-LSTM, and the extraction of entities and relationships is realized simultaneously in a framework through a decoder, and three different Bi-LSTMs are used to perform deep feature extraction respectively for entity features, relationship features, and shared features of the two. The beneficial effects of the present invention: utilizing the feature that the context of a sentence and entity information play an important role in the performance of relationship extraction, a new entity representation strategy is proposed, which can fuse the context and the position, boundary, and type information of the head and tail entities when jointly extracting entity relationships, which not only improves the performance of entity relationship extraction, but also solves the problem of overlapping relationships of shared entities.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to an information extraction method based on deep learning. Background Art

[0002] In the era of big data, information is growing at an accelerated rate. This makes it increasingly difficult to process and analyze large amounts of imperfect, incomplete, and unstructured natural language text data from multiple sources. Natural language expressions are rich and varied and have no fixed pattern, making it difficult for machines to understand the knowledge in the text and the semantic information it contains. Therefore, there is an urgent need for a method to organize and extract a large amount of fragmented information (usually unstructured in the form of natural human language) so that it can be absorbed, reasoned, and ultimately utilized. Information Extraction (IE) refers to extracting specific information from unstructured text, performing structured processing, and converting it into a structured knowledge representation (such as triples). Information extraction is not only the primary link in the process of building a knowledge graph, but is also widely used in intelligence and analysis, intelligent question and answer, knowledge graph construction, risk prediction, data analysis, intelligent search, and decision support.

[0003] The entity-relationship joint extraction network model mainly focuses on solving the problem of overlapping relationships of shared entities and task interaction modeling. Many solutions for entity-relationship joint extraction have been proposed. For example, Gupta et al. proposed a table-filling multi-task recurrent neural network, Miwa and Sasaki et al. introduced an entity relationship table to solve the difficulty of representing entity recognition and relationship extraction tasks, Miwa and Bansal et al. adopted a parameter-sharing joint learning model, Zheng et al. introduced a BILOU labeling strategy, etc. In fact, these methods often decompose the extraction of overlapping relationships into several internal dependency steps, forcing the decoder to decode recursively and requiring the main entity to be cascaded on the basis of advance recognition. Although this step decomposition makes the task easier to handle, it inevitably brings about the problem of exposure bias, resulting in the accumulation of errors in subsequent steps, which ultimately affects the accuracy of extraction.

[0004] In order to address the above problems, there is an urgent need for an information extraction method based on deep learning. Summary of the invention

[0005] The purpose of the present invention is to provide an information extraction method based on deep learning to solve the problems raised in the above background technology.

[0006] To achieve the above purpose, a deep learning-based information extraction method is provided, comprising the following steps:

[0007] S1: Create an end-to-end entity relationship joint extraction network model based on deep learning. The model includes an entity recognition module and an entity relationship extraction module. The entity recognition module is composed of Bi-LSTM plus CRF, and the entity relationship extraction module is composed of Bi-LSTM plus attention mechanism. The network model also includes a shared encoder.

[0008] S2: Take the text data as the input of the entire model, use the word embedding model to perform preliminary feature extraction on the text data, obtain the embedding representation of the feature, and use the extracted embedding representation as the feature sharing pool by sharing parameters through the shared encoder;

[0009] S3: Use three different Bi-LSTM networks to further extract named entity features and entity relationship features from the shared feature pool;

[0010] S4: training the loss function of the network model, training the word embedding, all Bi-LSTM and LSTM weights, and the weights of the CRF feature function in a supervised manner, so as to obtain the loss function of the minimized model of the entity recognition module and the loss function of the minimized model of the entity relationship extraction module respectively;

[0011] S5: The loss function of the minimized model of the entity recognition module and the loss function of the minimized model of the entity relationship extraction module are summed to obtain the overall objective function of the network model.

[0012] Furthermore, in S2, a tokenizer is used to segment words, which are mapped into specific numbers according to the vocabulary, and then a shared encoder is used to map them into a high-dimensional space, converting the text data into a low-density embedding vector so that the input sentences can be processed and calculated.

[0013] Furthermore, in S3, a standard LSTM unit consists of an input vector, a storage unit, an output vector, and several gates, where the calculation formula of the LSTM hidden layer is as follows:

[0014]

[0015] Among them, h t-1 represents the hidden state of the previous time step, c t , c t-1 represents the cell state of the current and previous time steps, i t is the output of the input gate, represents the candidate state at time t, tanh is the activation function, o t is the output of the output gate, W and b represent the weight matrix and matrix bias, ⊙ is the element-by-element multiplication operation, [W, b] is a trainable parameter,

[0016] is the embedding vector representation of each word, σ represents the activation function, i t , o t 、f t Represent the input gate, output gate and forget gate respectively, and the hidden state is a vector representation with context information;

[0017] For Bi-LSTM, it consists of two LSTMs: one extracts the forward hidden features and another to extract backward hidden features The Bi-LSTM function expression is:

[0018]

[0019] Finally, the output latent vector of Bi-LSTM is expressed as

[0020] Furthermore, in S4, the entity recognition module adopts Bi-LSTM and CRF algorithms, wherein Bi-LSTM has two parallel layers in different directions, and the sequence is input in the forward layer in a forward direction to record the above information, which is expressed as The reverse layer is input in order from back to front and is used to record the following information, which is expressed as Finally, the two layers are concatenated and each word is output to calculate the score of each entity label. The calculation formula is as follows:

[0021]

[0022] in, The superscript (e) indicates the symbol used for the NER task, f(·) is the activation function, V (e) ∈R p*l , U (e) ∈R l*2d , b (e) ∈R l , d is the dimension of the hidden vector of Bi-LSTM, p is the number of NER tags, and l is the width of the layer.

[0023] Furthermore, in S4, GRF is used to decode the entity label, and the entity sequence decoding is realized through the linear chain conditional random field CRF. Its CRF sequence annotation layer calculates the scores of all possible label sequences of the entire sentence:

[0024]

[0025] Here, S∈R, R represents a real number, and the superscript (e) indicates that the parameter belongs to the entity annotation subtask. The score of the predicted label for the kth word is the output of Bi-LSTM, with a dimension of [N e ]; is a square transformation matrix, which indicates the transfer score of the adjacent k-th word label to the k+1-th word label, and is the training parameter; represents the label, and n represents the number of labels. Since the start label is considered when constructing the scoring layer in CRF With the closing tag In the transfer process, the dimension k is 2 more than n, then, The dimension is [(N e +2)*(N e +2)], for the input sentence w, the probability of a given label sequence among all possible label sequences is defined as:

[0026]

[0027] For the CRF sequence annotation layer, the probability distribution of the label sequence is obtained Get the entity tag sequence with the highest score through the Viterbi algorithm As the final label sequence of the identified entity; for the identified entity output by CRF, in order to carry out the next step of relationship extraction, it is represented by the embedding layer, and e is set i ∈E, the extracted entity context is represented as h e (e i ), and its calculation formula is:

[0028]

[0029] here is the type feature of the entity, e i represents an entity, E represents the entity type, d f is the entity type feature width, and for the identified entities, the type embedding lookup table is used to embed each of them into i Represented as a vector, and further represented as h e (e i ), so h e (e i ) contains the context as well as entity location, boundary, and type information.

[0030] Further, in S4, the extraction process of the entity relationship extraction module can be divided into the following iterative steps:

[0031] S4.1: The decoder generates a relation for a triple;

[0032] S4.2: The decoder copies an entity from the entity recognition module as the main entity for triple extraction;

[0033] S4.3: The decoder copies the second entity from the entity recognition module as the guest entity for triple extraction;

[0034] S4.4: the above steps are iterated, and the decoder generates multiple triplets until a preset step size is reached, where the step size refers to the maximum number of iterations in the entire decoding process;

[0035] In S4.1, the entity context and the position, boundary and type information of the head and tail entities are fused, as well as the relation features extracted by Bi-LSTM as input and input into the relation decoder composed of unidirectional LSTM, which cooperates with the Attention mechanism and the Copy mechanism to extract relation triplets.

[0036] Furthermore, in S4.1, the relational features extracted by Bi-LSTM are expressed as as well as

[0037] Finally, the two layers are spliced ​​and output Right now

[0038] Calculation of attention distribution α for relational feature extraction: Attention distribution α is when a query Q is given, the i-th key value is K i When the i-th input information V i The degree of attention received is expressed as:

[0039]

[0040]

[0041] Here k = d is the dimension of the attention key K, W q ,W k ,W v ∈R d*d , b q ,b k ,b v ∈R d Q, K and V are used to obtain the weights and biases of their representations, respectively, and the attention value and Bi-LSTM output Entity relationship representation obtained by splicing

[0042] For a given relation hidden state representation and a set of entity type enhancement representations h e (e i ) related key-value pairs and By introducing entity type information into relationship type learning, a new enhanced representation of relationship type is obtained. Used for decoder decoding, the decoding process is as follows:

[0043]

[0044] in, is the weight, and is the bias, represents the context vector at time t, represents the updated hidden layer state, is the splicing symbol;

[0045] The decoder uses a unidirectional LSTM to predict the output from left to right. The last hidden state of the encoder is used to initialize the decoder hidden state. An attention score is assigned to each hidden state of the encoder, and then the attention sum is obtained by adding them up. This sum is then combined with the hidden state decoded at the last time step and sent to the decoder LSTM for decoding. For the output of the decoder LSTM, entity relations are extracted. The output of the relation decoder is calculated according to the following formula and hidden state

[0046]

[0047] Where LSTM() is the decoder function, is the hidden state at time step t-1, which is initialized using the relational feature extraction Bi-LSTM, u t is the input to the decoder at time step t and is calculated as follows:

[0048]

[0049] Here W u 、b u is the weight matrix and bias. After decoding, the fully connected layer is used to extract the relationship between entities. Assuming there are m types of entity relationships, the fully connected layer is used to calculate the score vectors of all valid relationships. Let the score vector be: Then we have:

[0050]

[0051] Among them, W o and b o is the weight matrix and bias. When predicting a relationship, there may be a situation where there is no relationship between two entities, that is, the relationship between them is NA. At this time, the model generates an NA triple. For this case, the calculated score is:

[0052]

[0053] Among them, W NA and b NA is the weight matrix and bias, using q r and q NA The connection operation generates the probability of all relationships, and then uses softmax to obtain the probability distribution. Then we have:

[0054] p r =softmax([q r ;q NA ]) (twenty two)

[0055] The entity relationship with the maximum probability is selected as the extracted entity relationship.

[0056] Furthermore, in S5, for the relationship extraction module, the cross entropy loss function calculation formula is:

[0057]

[0058] Where n is the number of samples, m is the number of relationships, and y i，j For a real relationship, is the relationship predicted by the model, when y i，j When it is 1, it means that the i-th sample has relationship j, and the loss function is simplified to The closer it is to 1, the smaller the loss function is. The closer to 0, the greater the loss function; when y i，j When it is 0, it means that there is no relationship j for the i-th sample, and the loss function is simplified to The closer it is to 0, the smaller the loss function is. The closer to 1, the larger the loss function. The curve of the cross entropy loss function is in line with expectations. When there is a relationship, the prediction probability is close to 1 and the model error is relatively small. When there is no relationship, the prediction probability is close to 0 and the model error is also relatively small.

[0059] Furthermore, in S5, for the named entity recognition module, the loss function is the maximum likelihood estimation loss function:

[0060]

[0061] in, is the word vector x i Predicted as label y i The scoring function score is It is the transfer matrix of the conditional random field, that is, the probability matrix of transferring from one entity label to another entity label, where the start state and the end state are considered, so the dimension of the transfer matrix T is ne +2, and then use the Viterbi algorithm to calculate the highest probability label sequence y * .Then the overall loss function of the entire model is:

[0062] LOSS total =LOSS ner +λrLOSS re

[0063] Among them, λ r is treated as a hyperparameter and allows tuning the relative importance of the NER and RE tasks during training.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. The model uses Bi-LSTM to extract features at multiple levels, including features specifically for entity recognition tasks, features for relationship extraction tasks, and features shared by both, and uses the attention mechanism to achieve the interaction of multi-level feature information;

[0066] 2. Through this strategy, the model can integrate the context, the location, boundary and type information of the head and tail entities when jointly extracting entity relations, which improves the performance of entity relation extraction and effectively solves the problem of overlapping relations of shared entities, that is, the problem that a sentence is shared by multiple overlapping relation triplets.

[0067] 3. Establish a joint extraction model for the two subtasks of entity recognition and relationship extraction. The entity-relationship joint extraction model learns entities and entity-pair relationships at the same time, encodes entity extraction and relationship extraction together, and uses the same model for end-to-end training. Therefore, the back propagation of the loss function will also correct the errors of entity recognition. Both entity and relationship extraction can benefit from joint execution, allowing each task to correct the errors of other tasks. When jointly extracting entity relationships, the context as well as the position, boundary, and type information of the head and tail entities can be integrated, which not only improves the performance of entity relationship extraction, but also solves the problem of overlapping relationships of shared entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a network model diagram for end-to-end entity relationship joint extraction in an embodiment of the present invention;

[0069] Figure 2 4 is a diagram showing the test results of entity relationship overlap in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0071] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0072] See also Figure 1-Figure 2 As shown, a method for information extraction based on deep learning is provided, including the following methods:

[0073] S1: Create an end-to-end entity-relationship joint extraction network model based on deep learning, and realize the extraction of entities and relations in one framework. The model includes an entity recognition module and an entity relationship extraction module. Based on the shared encoder and the back propagation of the entire end-to-end model loss, the model uses three different Bi-LSTMs to perform deep feature extraction for entity features, relationship features, and shared features of the two. The entity recognition module is composed of Bi-LSTM plus CRF, and the entity relationship extraction module is composed of Bi-LSTM plus attention mechanism. Taking advantage of the fact that the context of the sentence and entity information play an important role in the performance of relationship extraction, a new entity representation strategy is proposed. When jointly extracting entity relationships, the context and the position, boundary and type information of the head and tail entities can be integrated, which not only improves the performance of entity relationship extraction, but also solves the problem of overlapping relationships of shared entities.

[0074] S2: Take the text data as the input of the entire model, use the word embedding model to perform preliminary feature extraction on the text data, obtain the feature embedding representation, and use the extracted embedding representation as the feature sharing pool by sharing parameters through the shared encoder. It is worth noting that the parameter sharing method described here is specifically: a shared parameter layer is designed in the model. By sharing the underlying network parameters, three different Bi-LSTM networks can use the same feature extractor together, reducing the number of model parameters, improving training efficiency, and maintaining the expressiveness of the model;

[0075] S3: Use three different Bi-LSTM networks to further extract named entity features and entity relationship features from the shared feature pool;

[0076] S4: training the loss function of the network model, training the word embedding, all Bi-LSTM and LSTM weights, and the weights of the CRF feature function in a supervised manner, so as to obtain the loss function of the minimized model of the entity recognition module and the loss function of the minimized model of the entity relationship extraction module respectively;

[0077] S5: The loss function of the minimized model of the entity recognition module and the loss function of the minimized model of the entity relationship extraction module are summed to obtain the overall objective function of the network model.

[0078] In S2, tokenizer is used to segment words, which are mapped into specific numbers according to the vocabulary, and then a shared encoder is used to map them into a high-dimensional space, converting the text data into a low-density embedding vector to process and calculate the input sentence.

[0079] In S3, a standard LSTM unit consists of an input vector, a storage unit, an output vector, and several gates. The calculation formula of the LSTM hidden layer is as follows:

[0080]

[0081]

[0082] Among them, h t-1 represents the hidden state of the previous time step, c t , c t-1 represents the cell state at the current and previous time steps, represents the candidate state at time t, tanh is the activation function, W and b represent the weight matrix and matrix bias, ⊙ is the element-by-element multiplication operation, [W, b] is a trainable parameter, is the embedding vector representation of each word, σ represents the activation function, i t , o t 、f t Represent the input gate, output gate and forget gate respectively, and the hidden state is a vector representation with context information;

[0083] For Bi-LSTM, it consists of two LSTMs: one extracts the forward hidden features and another to extract backward hidden features The Bi-LSTM function expression is:

[0084]

[0085] Finally, the output latent vector of Bi-LSTM is expressed as

[0086] In S4, the entity recognition module adopts Bi-LSTM and CRF algorithms, where Bi-LSTM has two parallel layers in different directions. The sequence is input from front to back in the forward layer to record the above information, which is expressed as The reverse layer is input in order from back to front and is used to record the following information, which is expressed as Finally, the two layers are concatenated and each word is output to calculate the score of each entity label. The calculation formula is as follows:

[0087]

[0088] in, The superscript (e) indicates the symbol used for the NER task, f(·) is the activation function, V (e) ∈R p*l , U (e) ∈R l*2d , b (e) ∈R l , d is the dimension of the hidden vector of Bi-LSTM, p is the number of NER tags, and l is the width of the layer;

[0089] In S4, GRF is used to decode entity labels. When the CRF model is used for sequence labeling problems, the influence between the current output and the output at the previous moment is fully considered through the feature function. A CRF layer is added on top of the hidden layer of the model to constrain the output at each moment as a decoder for entity recognition. The CRF model can better constrain entities and obtain constraint rules from the training data, so that the labeling relationship between words is interdependent. The probability of illegal sequences appearing in label sequence prediction is greatly reduced, thereby improving the recognition performance of the model. Entity sequence decoding is realized through the linear chain conditional random field CRF. Its CRF sequence labeling layer calculates the scores of all possible label sequences for the entire sentence:

[0090]

[0091] Here, S∈R, R represents a real number, and the superscript (e) indicates that the parameter belongs to the entity annotation subtask. The score of the predicted label for the kth word is the output of Bi-LSTM, with a dimension of [N e ]; is a square transformation matrix, which indicates the transfer score of the adjacent k-th word label to the k+1-th word label, and is the training parameter; Indicates the first to nth labels, n represents the number of labels, because the starting label is considered when constructing the scoring layer of CRF With the closing tag

[0092] In the transfer process, the dimension k is 2 more than n, then, The dimension is [(N e +2)*(N e +2)], for the input sentence w, the probability of a given label sequence among all possible label sequences is defined as:

[0093]

[0094] For the CRF sequence annotation layer, the probability distribution of the label sequence is obtained Get the entity tag sequence with the highest score through the Viterbi algorithm As the final label sequence of the identified entity; for the identified entity output by CRF, in order to carry out the next step of relationship extraction, it is represented by the embedding layer, e i ∈E, the extracted entity context is represented as h e (e i ), and its calculation formula is:

[0095]

[0096] here is the type feature of the entity, e i represents an entity, E represents the entity type, d f is the entity type feature width, and for the identified entities, the type embedding lookup table is used to embed each of them into i Represented as a vector, and further represented as h e (e i ), so h e (e i ) contains the context as well as entity location, boundary, and type information;

[0097] In S4, the extraction process of the entity relationship extraction module can be divided into the following iterative steps:

[0098] S4.1: The decoder generates a relation for a triple;

[0099] S4.2: The decoder copies an entity from the entity recognition module as the main entity for triple extraction;

[0100] S4.3: The decoder copies the second entity from the entity recognition module as the guest entity for triple extraction;

[0101] S4.4: the above steps are iterated, and the decoder generates multiple triplets until a preset step size is reached, where the step size refers to the maximum number of iterations in the entire decoding process;

[0102] In the entire decoding process described above, the model will obtain multiple entity relationship triples. When the number of triples in the input sentence is less than the maximum number of triples set by the model, the process will continue iterating, and the decoder will generate NA triples, indicating empty entity relationship triples.

[0103] In S4.1, the entity context and the position, boundary and type information of the head and tail entities are integrated, as well as the relation features extracted by Bi-LSTM as input to the relation decoder composed of unidirectional LSTM, and the relation triples are extracted in conjunction with the Attention mechanism and the Copy mechanism.

[0104] Furthermore, in S4.1, the relational features extracted by Bi-LSTM are expressed as as well as

[0105] Finally, the two layers are spliced ​​and output Right now

[0106] Calculation of attention distribution α for relational feature extraction: Attention distribution α is when a query Q is given, the i-th key value is K i When the i-th input information V i The degree of attention received is expressed as:

[0107]

[0108] Here k = d is the dimension of the attention key K, W q ,W k ,W v ∈R d*d , b q ,b k ,b v ∈R d Q, K and V are used to obtain the weights and biases of their representations, respectively, and the attention value and Bi-LSTM output Entity relationship representation obtained by splicing

[0109] For a given relation hidden state representation and a set of entity type enhancement representations h e (e i ) related key-value pairs and By introducing entity type information into relationship type learning, a new enhanced representation of relationship type is obtained. Used for decoder decoding, the decoding process is as follows:

[0110]

[0111] in, is the weight, and is the bias, represents the context vector at time t, represents the updated hidden layer state, It is a splicing symbol, representing and Perform matrix splicing;

[0112] The decoder uses a unidirectional LSTM to predict the output from left to right. The last hidden state of the encoder is used to initialize the decoder hidden state. An attention score is assigned to each hidden state of the encoder, and then the attention sum is obtained by adding them up. This sum is then combined with the hidden state decoded at the last time step and sent to the decoder LSTM for decoding. For the output of the decoder LSTM, entity relations are extracted. The output of the relation decoder is calculated according to the following formula and hidden state

[0113]

[0114] Where LSTM() is the decoder function, is the hidden state at time step t-1, which is initialized using the relational feature extraction Bi-LSTM, u t is the input to the decoder at time step t and is calculated as follows:

[0115]

[0116] Here W u , b u is the weight matrix and bias. After decoding, the fully connected layer is used to extract the relationship between entities. Assuming there are m types of entity relationships, the fully connected layer is used to calculate the score vectors of all valid relationships. Let the score vector be: Then we have:

[0117]

[0118] Among them, W o and b ois the weight matrix and bias. When predicting a relationship, there may be a situation where there is no relationship between two entities, that is, the relationship between them is NA. At this time, the model generates an NA triple. For this case, the calculated score is:

[0119]

[0120] Among them, W NA and b NA is the weight matrix and bias, selu is the activation function, and q is used r and q NA The connection operation generates the probability of all relationships, and then uses softmax to obtain the probability distribution. Then we have:

[0121] p r =softmax(p[q r ;q NA ]) (twenty two)

[0122] Select the entity relationship with the highest probability as the extracted entity relationship;

[0123] In S5, for the relation extraction module, the cross entropy loss function is calculated as:

[0124]

[0125] Where n is the number of samples, m is the number of relationships, and y i，j For a real relationship, is the relationship predicted by the model, when y i，j When it is 1, it means that the i-th sample has relationship j, and the loss function is simplified to The closer it is to 1, the smaller the loss function is. The closer to 0, the greater the loss function; when y i，j When it is 0, it means that there is no relationship j for the i-th sample, and the loss function is simplified to The closer it is to 0, the smaller the loss function is. The closer it is to 1, the larger the loss function is. The curve of the cross entropy loss function is in line with expectations. When there is a relationship, the prediction probability is close to 1 and the model error is relatively small. When there is no relationship, the prediction probability is close to 0 and the model error is also relatively small.

[0126] In S5, for the named entity recognition module, the loss function is the maximum likelihood estimation loss function:

[0127]

[0128] in, is the word vector x iPredicted as label y i The scoring function score is It is the transfer matrix of the conditional random field, that is, the probability matrix of transferring from one entity label to another entity label, where the start state and the end state are considered, so the dimension of the transfer matrix T is n e +2, and then use the Viterbi algorithm to calculate the highest probability label sequence y * .Then the overall loss function of the entire model is:

[0129] LOSS total =LOSS ner +λ r LOSS re

[0130] Among them, λ r It is considered as a hyperparameter and allows the relative importance of the NER and RE tasks to be adjusted during training. The final training of both NYT and WebNLG datasets used a value of 5. The overall loss function is minimized during training. The Adam optimizer is used to train the model in this application experiment. Because the algorithm retains different learning rates for each parameter during training and iteratively updates the weights of the network, the network training process is smoother and the convergence speed is faster.

[0131] The input of the problem consists of n words x1, x2, ...x n The sentence X is composed of E = {e1, e2, ...e m} are all possible entity sequences in X with a length not exceeding L, START(i) and END(i) represent e i The start and end index of the .

[0132] Named entity recognition, let T e Represents a set of predefined entity types. The named entity recognition task is to extract each valid entity sequence e i ∈E, the sequence belongs to the predefined entity type y e (e i )∈T e The output of the task is Y e ={(e i , t e ):e i ∈E,t e ∈T e}. This application adopts the entity sequence annotation scheme of BILOU, {B-*, I-*, L-*, O, U-*,}. Among them, B, I, L, O, U represent the word at the beginning of the entity; the word in the middle; the last word; the word outside the entity and the entity consisting of a single word. And *∈ε.

[0133] Relation extraction, let R represent a set of predefined relationship types. The task of relation extraction is to i ∈E,e j ∈E, predict relationship type Y r (e i , e j )∈R, or there is no relationship Y between the two entities r (e i , e j )∈NA, the output of the relation extraction task is a triple Y r ={(e i ,r,e j ):e i , e j ∈E, r∈R}. Here e i , e j They are the head entity and the tail entity respectively.

[0134] In named entity recognition and entity relationship extraction tasks, the quality of the dataset is crucial, especially in entity relationship extraction tasks, where the model needs to accurately extract all relation triplets in the text and avoid missing labels to ensure a good recall rate. To this end, this application selected the NYT dataset and the WebNLG dataset as experimental objects. The NYT dataset is based on the New York Times corpus and is annotated using remote supervision, and is widely used in relation extraction tasks. The WebNLG dataset was originally designed for natural language generation tasks and contains 246 valid relations. Each instance contains not only one or more relation triplets, but also text descriptions related to these triplets, providing a rich and diverse corpus suitable for the evaluation of a variety of natural language processing tasks.

[0135] The NYT dataset is based on the New York Times corpus and uses a remote supervision method to annotate data. The WebNLG dataset was originally created for natural language generation tasks and contains 246 valid relations. Each instance contains one or more relation triplets and one or more sentences of text corpus containing all relation triplets.

[0136] Table 1: Statistics of NYT and WebNLG datasets

[0137]

[0138] The accuracy (Precision, P) is used to calculate the ratio of the number of correctly predicted samples to the number of predicted samples for each category; the recall (Recall, R) is used to calculate the ratio of the number of correctly predicted samples to the number of actual samples for each category; and the F1 value (F1-score) is the harmonic mean of the precision and recall rates. These three indicators are used to measure the model as evaluation indicators for named entity recognition and entity relationship extraction. Among them, the accuracy rate refers to the ratio of the number of correctly recognized entities (number of relationships) to the total number of recognized entities (number of relationships), the recall rate refers to the ratio of the number of correctly recognized entities (number of relationships) to the total number of entities (number of relationships), and the F value is the harmonic mean of the accuracy rate and the recall rate. The specific formulas for each indicator are as follows.

[0139]

[0140] Where: T p Indicates the number of positive classes judged as positive; F p Indicates the number of negative classes judged as positive classes; F N Indicates the number of positive classes that are judged as negative classes. In the application experiment, the correct and incorrect predictions are based on triples, that is, when the head entity, relationship, and tail entity in the triple are all predicted correctly, the result of the triple is considered correct. And then the results of named entity recognition and relationship extraction are considered correct.

[0141] Table 2: Experimental results of typical models on NYT and WebNLG datasets

[0142]

[0143] The experimental results in the table above show that the experimental results show the importance of entity information in the entity relationship extraction task, especially when the entity type representation is combined with the attention mechanism, the performance of relationship extraction can be significantly improved. Specifically, by combining the location information, boundary information and type information of the entity, and associating it with the relationship extraction task using the attention mechanism, the performance of the model in relationship extraction has been significantly improved.

[0144] In the experiment, the model architecture consists of a Bi-LSTM encoding layer and an LSTM decoding layer with a bias objective function. The dimension of the word embedding is 300, and dropout regularization is applied on the embedding layer with a dropout ratio of 0.5. The number of LSTM units in the encoding layer is 300, and the number of LSTM units in the decoding layer is 600.

[0145] It should be noted that the above experimental results are based on the analysis without considering the overlap of relations. The entity overlap problem is one of the key research contents of this paper, so we will examine the performance of the model on sentences with and without entity overlap. To this end, we divide the test dataset into three types of instances: NEO (no entity overlap), EPO (a group of entity pairs overlap), and SEO (single entity overlap). Next, we compare the performance of the model proposed in this paper with the NovelTagging model and the CopyRE model on different sentence patterns. The experimental results are shown in the figure below. Figure 2 shown.

[0146] The model proposed in this application not only achieved excellent experimental performance in non-overlapping relationships, but also had equally excellent results on overlapping relationship datasets. Experiments have shown that the joint extraction used can bring a significant additional improvement in the F1 value, and the performance on overlapping relationship data has also been significantly improved, indicating that the model proposed in this application has good performance in both entity recognition and relationship classification.

[0147] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A deep learning-based information extraction method, characterized in that: The following steps are involved: S1: Create an end-to-end entity relationship joint extraction network model based on deep learning. The model includes an entity recognition module and an entity relationship extraction module. The entity recognition module is composed of Bi-LSTM plus CRF, and the entity relationship extraction module is composed of Bi-LSTM plus attention mechanism. The network model also includes a shared encoder. S2: Take the text data as the input of the entire model, use the word embedding model to perform preliminary feature extraction on the text data, obtain the embedding representation of the feature, and use the extracted embedding representation as the feature sharing pool by sharing parameters through the shared encoder; S3: Use three different Bi-LSTM networks to further extract named entity features and entity relationship features from the shared feature pool; S4: training the loss function of the network model, training the word embedding, all Bi-LSTM and LSTM weights, and the weights of the CRF feature function in a supervised manner, so as to obtain the loss function of the minimized model of the entity recognition module and the loss function of the minimized model of the entity relationship extraction module respectively; S5: The loss function of the minimized model of the entity recognition module and the loss function of the minimized model of the entity relationship extraction module are summed to obtain the overall objective function of the network model.

2. The information extraction method based on deep learning according to claim 1, characterized in that: In S2, a tokenizer is used to segment words, which are mapped into specific numbers according to the vocabulary. A shared encoder is then used to map them into a high-dimensional space, converting the text data into a low-density embedding vector so that the input sentences can be processed and calculated.

3. The method for extracting information based on deep learning according to claim 2, characterized in that: In S3, a standard LSTM unit consists of an input vector, a storage unit, an output vector, and several gates. The calculation formula of the LSTM hidden layer is as follows: Among them, h t-1 represents the hidden state of the previous time step, c t , represents the cell state of the current and previous time steps, i t is the output of the input gate, represents the candidate state at time t, tanh is the activation function, o t is the output of the output gate, W and b represent the weight matrix and matrix bias, ⊙ is the element-by-element multiplication operation, [W, b] is a trainable parameter, is the embedding vector representation of each word, σ represents the activation function, i t , o t 、f t Represents the input gate, output gate and forget gate respectively, and the hidden state is a vector representation with context information; For Bi-LSTM, it consists of two LSTMs: one extracts the forward hidden features and another to extract backward hidden features The Bi-LSTM function expression is: Finally, the output latent vector of Bi-LSTM is expressed as 4. The method for extracting information based on deep learning according to claim 3, characterized in that: In S4, the entity recognition module adopts Bi-LSTM and CRF algorithms, where Bi-LSTM has two parallel layers in different directions. The sequence is input from front to back in the forward layer to record the above information, which is expressed as The reverse layer is input in order from back to front and is used to record the following information, which is expressed as Finally, the two layers are concatenated and each word is output to calculate the score of each entity label. The calculation formula is as follows: in, The superscript (e) indicates the symbol used for the NER task, f(·) is the activation function, V (e) ∈R p*l , U (e) ∈R l*2d , b (e) ∈R l , d is the dimension of the hidden vector of Bi-LSTM, p is the number of NER tags, and l is the width of the layer.

5. The method for extracting information based on deep learning according to claim 4, characterized in that: In S4, GRF is used to decode entity tags, and entity sequence decoding is achieved through linear chain conditional random field CRF. Its CRF sequence annotation layer calculates the scores of all possible tag sequences of the entire sentence: Here, S∈R, R represents a real number, and the superscript (e) indicates that the parameter belongs to the entity annotation subtask. The score of the predicted label for the kth word is the output of Bi-LSTM, with a dimension of [N e ]; is a square transformation matrix, which indicates the transfer score of the adjacent k-th word label to the k+1-th word label, and is the training parameter; represents the label, and n represents the number of labels. Since the start label is considered when constructing the scoring layer in CRF With the closing tag In the transfer process, the dimension k is 2 more than n, then, The dimension is [(N e +2)*(N e +2)], for the input sentence w, the probability of a given label sequence among all possible label sequences is defined as: For the CRF sequence annotation layer, the probability distribution of the label sequence is obtained Get the entity tag sequence with the highest score through the Viterbi algorithm As the final label sequence of the identified entity; For the recognized entities output by CRF, in order to carry out the next step of relation extraction, they are represented by the embedding layer, and e is set i ∈E, the extracted entity context is represented as h e (e i ), and its calculation formula is: here is the type feature of the entity, e i represents an entity, E represents the entity type, d f is the entity type feature width, and for the identified entities, the type embedding lookup table is used to embed each of them into i Represented as a vector, and further represented as h e (e i ), so h e (e i ) contains the context as well as entity location, boundary, and type information.

6. The method for extracting information based on deep learning according to claim 5, characterized in that: In S4, the extraction process of the entity relationship extraction module can be divided into the following iterative steps: S4.1: The decoder generates a relation for a triple; S4.2: The decoder copies an entity from the entity recognition module as the main entity for triple extraction; S4.3: The decoder copies the second entity from the entity recognition module as the guest entity for triple extraction; S4.4: the above steps are iterated, and the decoder generates multiple triplets until a preset step size is reached, where the step size refers to the maximum number of iterations in the entire decoding process; In S4.1, the entity context and the position, boundary and type information of the head and tail entities are fused, as well as the relation features extracted by Bi-LSTM as input and input into the relation decoder composed of unidirectional LSTM, which cooperates with the Attention mechanism and the Copy mechanism to extract relation triplets.

7. The method for extracting information based on deep learning according to claim 6, characterized in that: In S4.1, the relational features extracted by Bi-LSTM are expressed as as well as Finally, the two layers are spliced ​​and output Right now Calculation of attention distribution α for relational feature extraction: Attention distribution α is when a query Q is given, the i-th key value is K i When the i-th input information V i The degree of attention received is expressed as: Here k = d is the dimension of the attention key K, W q ,W k ,W v ∈R d*d , b q ,b k ,b v ∈R d Q, K and V are used to obtain the weights and biases of their representations, respectively, and the attention value and Bi-LSTM output Entity relationship representation obtained by splicing For a given relation hidden state representation and a set of entity type enhancement representations h e (e i ) related key-value pairs and By introducing entity type information into relationship type learning, a new enhanced representation of relationship type is obtained. Used for decoder decoding, the decoding process is as follows: in, is the weight, and is the bias, represents the context vector at time t, Represents the updated hidden layer state, ⊕ is the concatenation symbol; The decoder uses a unidirectional LSTM to predict the output from left to right. The last hidden state of the encoder is used to initialize the decoder hidden state. An attention score is assigned to each hidden state of the encoder, and then added to get the attention sum. This sum is then combined with the hidden state decoded at the last time step and sent to the decoder LSTM for decoding. For the output of the decoder LSTM, entity relations are extracted. The output of the relation decoder is calculated according to the following formula and hidden state Where LSTM() is the decoder function, is the hidden state at time step t-1, which is initialized using the relational feature extraction Bi-LSTM, u t is the input to the decoder at time step t and is calculated as follows: Here W u 、b u is the weight matrix and bias. After decoding, the fully connected layer is used to extract the relationship between entities. Assuming there are m types of entity relationships, the fully connected layer is used to calculate the score vectors of all valid relationships. Let the score vector be: Then we have: Among them, W o and b o is the weight matrix and bias. When predicting a relationship, there may be a situation where there is no relationship between two entities, that is, the relationship between them is NA. At this time, the model generates an NA triple. For this case, the calculated score is: Among them, W NA and b NA is the weight matrix and bias, using q r and q NA The connection operation generates the probability of all relationships, and then uses softmax to obtain the probability distribution. Then we have: p r =softmax([q r ;q NA ]) (22) The entity relationship with the maximum probability is selected as the extracted entity relationship.

8. The method for extracting information based on deep learning according to claim 7, characterized in that: In S5, for the relation extraction module, the cross entropy loss function is calculated as: Where n is the number of samples, m is the number of relationships, and y i,j For a real relationship, is the relationship predicted by the model, when y i,j When it is 1, it means that the i-th sample has relationship j, and the loss function is simplified to The closer it is to 1, the smaller the loss function is. The closer to 0, the greater the loss function; when y i,j When it is 0, it means that there is no relationship j for the i-th sample, and the loss function is simplified to The closer it is to 0, the smaller the loss function is. The closer to 1, the larger the loss function. The curve of the cross entropy loss function is in line with expectations. When there is a relationship, the prediction probability is close to 1 and the model error is relatively small. When there is no relationship, the prediction probability is close to 0 and the model error is also relatively small.

9. The method for extracting information based on deep learning according to claim 8, characterized in that: In S5, for the named entity recognition module, the loss function is the maximum likelihood estimation loss function: in, is the word vector x i Predicted as label y i The scoring function score is It is the transfer matrix of the conditional random field, that is, the probability matrix of transferring from one entity label to another entity label, where the start state and the end state are considered, so the dimension of the transfer matrix T is n e +2, and then use the Viterbi algorithm to calculate the highest probability label sequence y * .Then the overall loss function of the entire model is: LOSS total =LOSS ner +λrLOSS re Among them, λ r is treated as a hyperparameter and allows tuning the relative importance of the NER and RE tasks during training.

Images