An open relation extraction method based on neural network

Abstract

The application discloses an open relation extraction method based on a neural network, and comprises the following steps: replacing biological entity concepts in microorganism field data with corresponding concept types to obtain input data; constructing a hierarchical recurrent neural network decoder; and inputting the obtained input data into the hierarchical recurrent neural network decoder to obtain semi-structured output data. The method is aimed at the characteristics of the microorganism field data, replaces microorganism concepts in the original data, alleviates the sparsity problem of field professional vocabulary, and uses the constructed hierarchical recurrent neural network decoder to process the input data, so that the task of outputting semi-structured data can be well completed.

Description

TECHNICAL FIELD

[0001] The present application relates to a neural network-based open relation extraction method, belonging to the technical field of brain-like computing. BACKGROUND

[0002] With the rapid development of Internet technology and life science technology, a large number of biomedical literatures are showing an explosive growth trend. Today, in the era of big data, a large amount of unstructured text data in the biomedical field has become a huge mine containing human wisdom. With the increasing demand for mining and sorting of massive knowledge, the related technology of relation extraction for the ultimate goal of building a knowledge base has emerged.

[0003] Traditional biological field relation extraction methods need to define the classification system of entity relations in advance, and then use various rule templates for relation instance recognition and relation tuple extraction. However, due to the lack of sufficient basis, it is difficult to define scientific entity relation types in advance for such a subdivided field as microorganisms. Open relation extraction technology, as an unguided method, can realize the discovery and extraction of relations without predefinition. This makes it a preferred method for relation extraction in immature subdivided fields.

[0004] In recent years, open relation extraction methods have mainly focused on the recognition of syntactic relation trees and other features. Although compared with traditional manual rules and grammar rules, grammar rules have made breakthroughs, they have reached a bottleneck. At the same time, neural network models have developed rapidly in recent years and have achieved breakthroughs and successes in many fields of natural language processing, which also provides a new method for the further development of open relation extraction technology.

[0005] The open relation extraction task requires the model to have the ability to output semi-structured data. However, current neural network models are mainly based on RNN (unstructured output), Tree RNN (fully structured output), etc. These model structures cannot be directly applied to the open relation extraction task. SUMMARY

[0006] In view of the deficiencies of the prior art, the technical problem to be solved by the present application is to provide a neural network-based open relation extraction method.

[0007] To achieve the above-mentioned application purposes, the technical scheme adopted by the present application is as follows:

[0008] A neural network-based open relation extraction method, comprising the following steps:

[0009] Replace the biological entity concept in the microorganism field data with the corresponding concept type to obtain input data;

[0010] constructing a hierarchical recurrent neural network decoder;

[0011] inputting the obtained input data into the hierarchical recurrent neural network decoder to obtain semi-structured output data.

[0012] wherein preferably, the biological entity concept in the microbiology field data is replaced by a corresponding concept type to obtain the input data, comprising the following steps:

[0013] identifying the microbiology field concept in the text in a dictionary matching manner;

[0014] annotating the concept type to which each field word in the sentence belongs;

[0015] identifying the rare word in the biological field data, and replacing it with a corresponding concept type to obtain the input data.

[0016] wherein preferably, the hierarchical recurrent neural network decoder is constructed, comprising the following steps:

[0017] setting an internal decoding unit at the top layer according to the hierarchical division of the hierarchical recurrent neural network;

[0018] setting a conjugate decoding unit at each layer except the top layer;

[0019] the conjugate decoding unit comprises an edge judgment classifier, an internal decoding unit and an internal encoding unit;

[0020] wherein the edge judgment classifier is used to perceive the division boundary of the layer;

[0021] the internal encoding unit encodes the input from the next layer, and transmits the encoded information to the previous layer of the current layer when reaching the division boundary of the layer of the current layer;

[0022] the internal decoding unit predicts the output of the next layer of the internal decoding unit according to the input of the previous layer, the output of the next layer of the internal encoding unit and the hidden state of the internal decoder.

[0023] wherein preferably, the obtained input data is input into the hierarchical recurrent neural network decoder to obtain semi-structured output data, comprising the following steps:

[0024] the internal decoding unit of the top layer of the hierarchical recurrent neural network decodes the input data to generate semantic segment information;

[0025] according to the semantic segment information generated by the decoding unit of the upper layer, the internal encoding unit generates semantic information of the next layer granularity one by one, and re-encodes the generated semantic information through the internal encoding unit to generate re-encoded information;

[0026] When the edge judgment classifier determines that the internal decoding unit is at the division boundary, decoding is stopped, and re-encoding information is returned.

[0027] wherein preferably, at time t, the input of the internal decoding unit of the l-th layer comprises: the edge signal from the (l-1)-th layer, the output signal of the internal encoding unit of the (l-1)-th layer, the hidden state of the internal decoding unit at time (t-1), the hidden state of the internal encoding unit at time (t-1), and the output of the internal decoding unit of the (l+1)-th layer at time t.

[0028] wherein preferably, when the output signal of the internal encoding unit of the (l-1)-th layer is 0, the (l-1)-th layer does not reach the division boundary, and the l-th layer remains in the original state without any operation.

[0029] wherein preferably, when the output signal of the internal encoding unit of the (l-1)-th layer is 1 and the output signal of the internal encoding unit of the l-th layer is 0, the (l-1)-th layer reaches the division boundary, and the l-th layer does not reach the division boundary, the encoding unit of the l-th layer generates a new state d0 l ,tfrom the output of the internal encoding unit of the (l-1)-th layer as the input of the internal decoding unit of the l-th layer, and outputs it to the (l-1)-th layer, and the decoding unit of the (l-1)-th layer takes d0 l ,tas the initial state of the new segment of the (l-1)-th layer; wherein d0 l ,tis the output of the conjugate decoding unit of the l-th layer at time t.

[0030] wherein preferably, when the output signals of the internal encoding units of the (l-1)-th layer and the l-th layer are both 1, the l-th layer and the (l-1)-th layer reach the division boundary at the same time; the internal encoding unit of the l-th layer encodes the encoding result of the (l-1)-th layer as the last piece of information of the current segment into the hidden state of the internal decoding unit and transmits it upward, while discarding its current encoding state; after completing the decoding at time t, the internal decoding unit discards its decoding state, and obtains the new decoding information of the next segment from the (l+1)-th layer as the hidden state of the internal decoding unit.

[0031] wherein preferably, the decoder is optimized based on a loss function by gradient descent method when optimization is performed.

[0032] wherein the loss function is the cross entropy of the boundary judgment result and the cross entropy of the generated sequence result.

[0033] wherein preferably, the output data is in the form of tuples, and the average position of each word in the original sentence in the tuple is in the front, so the tuple is in the front in the generated tuple sequence.

[0034] The open relation extraction method based on the neural network provided by the application replaces biological entity concepts in the microorganism field data with corresponding concept types to obtain input data; then, a hierarchical recurrent neural network decoder is constructed; the obtained input data is input into the hierarchical recurrent neural network decoder to obtain semi-structured output data. The output data is in the form of a tuple, and the average position of each word in the tuple in the original sentence is in the front, so that the tuple is in the front in the generated tuple sequence, thereby ensuring the uniqueness of the output. The method converts the task of open relation extraction, so that it is close to the output characteristics of the neural network. On the other hand, the microorganism concepts in the original data are replaced according to the characteristics of the microorganism field data, thereby alleviating the sparsity problem of the field professional vocabulary. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The flowchart of the open relation extraction method based on the neural network provided by the application;

[0036] Figure 2 The flowchart of the HRNN-CDU decoding in the embodiment provided by the application;

[0037] Figure 3 The decoding flowchart of the existing Tree LSTM decoder;

[0038] Fig. 4(a) is a schematic diagram of the operation condition when the output signal of the l-1 layer internal coding unit is 0 in the embodiment provided by the application;

[0039] Fig. 4(b) is a schematic diagram of the operation condition when the output signal of the l-1 layer internal coding unit is 1 and the output signal of the l layer internal coding unit is 0 in the embodiment provided by the application;

[0040] Fig. 4(c) is a schematic diagram of the operation condition when the output signals of the l-1 layer and the l layer internal coding units are both 1 in the embodiment provided by the application;

[0041] Figure 5 The schematic diagram of the HRNN-CDU decoding in the embodiment provided by the application. DETAILED DESCRIPTION

[0042] The technical content of the application will be described in detail below in combination with the drawings and specific embodiments.

[0043] Most of the open relation extraction technologies are based on traditional methods, and an important reason is that the model characteristics of the neural network are not naturally suitable for the open relation extraction task: the final output result of the open relation extraction task is generally semi-structured data.

[0044] At present, the neural network model is mainly RNN (recursive neural network) with unstructured output and Tree RNN (tree recursive neural network) with completely structured output. The model structure cannot be directly applied to the open relationship extraction task. Among the methods of neural network, there is no suitable model for semi-structured output. Therefore, the general model of neural network needs to be reasonably adjusted to adapt to such output structure. The neural network model used in the application can combine the characteristics of Tree RNN and RNN to realize the output of semi-structured data. The Decoder neural network structure for realizing semi-structured data output is also the key to further realize the open relationship extraction algorithm.

[0045] As shown in Figure 1 The open relationship extraction method based on neural network provided by the application comprises the following steps: first, replacing biological entity concepts in microbiological field data with corresponding concept types to obtain input data; then, constructing a hierarchical recursive neural network decoder; finally, inputting the obtained input data into the hierarchical recursive neural network decoder to obtain semi-structured output data. The process is described in detail below.

[0046] S1, replacing biological entity concepts in microbiological field data with corresponding concept types to obtain input data.

[0047] Compared with common sentences, microbiological field data involves a large number of low-frequency professional words. If the original natural sentences (such as paper titles or paper abstracts) in the microbiological field are used as the input of the neural network model, the data sparsity caused by the low-frequency words in the microbiological field will make the neural network unable to learn the reasonable distribution representation of the words, and further affect the final performance of the model.

[0048] Therefore, it is necessary to use a strategy in the algorithm to replace biological entity concepts in biological field data with corresponding concept types to obtain input data; specifically comprising the following steps:

[0049] S11, identifying microbiological field concepts in the text in a dictionary matching manner;

[0050] S12, labeling the concept type to which each field word in the sentence belongs;

[0051] S13, identifying rare words in biological field data and replacing them with corresponding concept types to obtain input data.

[0052] Specifically, in the embodiments provided in the present application, the metamap of the UMLS can be introduced, which can identify the microorganism field concept in the text based on the UMLS in a dictionary matching manner, and mark the concept type to which each field word in the sentence belongs.

[0053] Before processing the data set for the microorganism field, the rare words belonging to the microorganism field in the microorganism field data are replaced by the corresponding microorganism concept types, and all the replaced rare words are restored according to the corresponding relationship in the final output triple result.

[0054] In an embodiment provided in the present application, the MetaMap tool of the UMLS is used to replace the biological entity concept in the corpus with the corresponding concept type in a dictionary-based manner. Thus, the potential concept of the order of magnitude of millions is mapped to more than one hundred words, greatly reducing the size of the decoder prediction word library, so that the decoder word library only contains the commonly used words that it needs to predict. The specific replacement mode is as follows:

[0055] Before replacement:

[0056] Original sentence: Doctors used to inject penicillin to patients as a treatment method of pneumonia;

[0057] Triple: inject penicillin, is a treatment method of, pneumonia.

[0058] Replace the concept entity:

[0059] inject→action1, penicillin→antibiotic1, pneumonia→disease1

[0060] After replacement:

[0061] Original sentence: Doctors used to <action1> <antibiotic1>to patients as a treatment method of <disease1>;

[0062] triplet: <action1> <antibiotic1>is a treatment method of, <disease1>

[0063] It can be seen that after the replacement operation is completed, the remaining biological concept can retain the key semantic information required for relation extraction, while reducing the possibility of low-frequency professional vocabulary, and greatly alleviating the data sparseness caused by professional vocabulary.

[0064] S2, construct a hierarchical recurrent neural network decoder.

[0065] Before describing the hierarchical recurrent neural network decoder provided by the present application, the basic assumptions of the hierarchical RNN decoder model are introduced. Among numerous neural network models, the hierarchical RNN model provides the most natural possibility for implementing semantic modeling and semi-structured output based on neural networks. Based on the hierarchical RNN model and its characteristics, the present application proposes a decoder based on this structure, a conjugate decoding unit, which can decode and output semi-structured data based on neural network methods. The model obtained is trained, tested and compared in the word segmentation task and the open relation extraction task in the general field. The results show that the hierarchical recurrent neural network decoder has certain superiority in the prediction of semi-structured output data.

[0066] In the embodiments provided by the present application, the hierarchical RNN decoder model is assumed to be close to the output characteristics of the neural network. In natural language processing tasks, the hierarchical RNN encoder model assumes that semantics can be divided according to different granularities, such as letters, words, phrases and sentences, and different sequences formed should be modeled respectively. In the hierarchical RNN model, there are multiple RNN encoder models corresponding to language sequences of different division granularities, and an RNN encoder model is set for each level.

[0067] In the process of encoding in time sequence, the RNN receives from the lower layer RNN with finer granularity (lower level) until it reaches the division edge, and when it is at the division edge, the RNN encodes the semantic information obtained as the input of the upper layer RNN. For example: for the RNN encoding unit at the letter level, the letter sequence is encoded one by one from the start position of the word, until the end of the word, and then the encoding result of the letter sequence is input into the RNN encoder at the word level as the distribution representation of the word, and the modeling object of the RNN encoder itself is a language segment of coarse granularity; for the lower layer RNN encoder, the encoding object thereof is a language segment of fine granularity.

[0068] Compared with the general RNN encoding model, in the hierarchical RNN decoding model: the decoding output result can be divided into different sequences (alphabet sequence, character sequence, word sequence, phrase sequence, sentence sequence, etc.) according to different granularities. In the decoding process, the decoding information is gradually transmitted and refined from the upper layer to the lower layer. The "phrase" hierarchical RNN decoding unit sequentially predicts the distribution representation of each "phrase" in the sentence and takes it as the input of the "word" hierarchical RNN decoding unit; the "word" hierarchical RNN decoding unit then predicts the distribution representation of each word in the sentence and further predicts each word.

[0069] Based on the above basic assumptions of the hierarchical RNN decoder model, combined with the design of the reference Tree RNN decoder, the application proposes a decoding unit based on the hierarchical RNN structure - the conjugate decoding unit (Conjugate Decoder Unit, referred to as CDU). On this basis, the hierarchical recurrent neural network decoder provided by the application is constructed, which specifically includes the following steps: first, according to the hierarchical division of the hierarchical recurrent neural network, an internal decoding unit is set at the top layer. In the embodiments provided by the application, the internal decoding unit at the top layer is an RNN decoder. Then, a conjugate decoding unit is set at each level except the top layer; the conjugate decoding unit includes a classifier (edge judgment classifier) for perceiving the division boundary, and two RNN (LSTM) internal decoding units (internal decoder) and internal encoding units (internal encoder) performing opposite tasks. Among them, the internal encoding unit encodes the input from the next layer and transmits the encoded information to the previous layer of the current layer when reaching the division boundary of the current layer of the hierarchical recurrent neural network; the internal decoding unit predicts the output of the internal decoding unit of the next layer according to the input of the previous layer and the output of the internal encoding unit of the next layer. In the embodiments provided by the application, according to the hierarchical division of the hierarchical recurrent neural network, an internal decoding unit is set at each layer; an edge judgment classifier and an internal decoding unit are set at each level except the top layer. The overall decoding process of a single-layer CDU is shown in Figure 2 .

[0070] The CDU generates semantic information of the next layer granularity one by one from the internal encoding unit according to the semantic segment information generated by the upper layer decoder, and re-encodes the generated semantic information through the internal encoding unit to generate re-encoding information. When the edge judgment classifier determines that the decoder is in the edge state (bs=1), the decoding is stopped, and the re-encoding information is returned. The design of the internal encoding unit is derived from the design of the LD-lstm in the existing Tree LSTM structure shown in Figure 3 .

[0071] In the embodiments provided in the present application, when decoding is performed by using a hierarchical recurrent neural network decoder, the specific individual process details of the CDU are as follows:

[0072] The edge judgment classifier is based on the design of the state unit, which is based on the ability of the hidden state h of the HRNN to perceive edges during encoding. The output of the edge judgment classifier of the l-th conjugate decoding unit is denoted as be l (boundray estimate), and the corresponding judgment signal value is bs (boundray signal), and the unit function is denoted as:

[0073] bs l = sigmoid (h) (1)

[0074] wherein bs l is a discrete signal quantity determined by h in theory, and here, for the convenience of derivation, the sigmoid function is used instead. bs l = 1 indicates that the l-th layer reaches the division boundary; and bs l = 0 indicates that the l-th layer does not reach the division boundary.

[0075] The LSTM responsible for continuously encoding the input from the lower layer is referred to as the inner encoding unit (Inner Encode Unit) in the present patent application, and the hidden state transmitted by the l-th inner encoding unit over time t is denoted as eh l,t (encoder hidden), the memory state transmitted over time t is denoted as ec l,t , and the output at time t is denoted as eo l,t (encoder output), and the input at time t is denoted as the output eo l-1,t of the lower encoding unit. The inner encoding unit performs the same operation as in the HRNN encoder: when the lower bs l-1 is 1, the inner encoding unit is responsible for encoding the input from the lower layer, and at the end of the encoding segment (bs l = 1), that is, when the division boundary of the current layer is reached, the inner encoding unit transmits its encoding information to the higher layer. The function form of the inner encoding unit is:

[0076] eh l,t , ec l,t = lstm (eh l,t-1 , ec l,t-1 , bs l-1,t * eo l-1,t ) (2)

[0077] eo l,t = bs l,t * eh l,t (3)

[0078] Another LSTM is the Inner Decode Unit, and let the hidden state of the Inner Decode Unit at time t be dh t (decoder hidden), and the state of the output of the Inner Decode Unit at time t be do t (decoderoutput). The Inner Decode Unit is responsible for generating the prediction of the next lower-level language segment based on the output i t from the next layer, the output of the previous layer CDU, and dh t When at the segment edge (bs t = 1), the Inner Decode Unit will obtain a new dh t from the higher layer, and continue to generate the prediction of the new language segment. In the embodiments provided by the present application, the case of decoding a sentence into corresponding words of triplets is described: the Inner Decode Unit of the second layer CDU generates the prediction of the next lower-level language segment based on the output of the first layer CDU, the output of the next layer Inner Encode Unit, and the input hidden state of the triplet, and the decoding of the triplet is completed after the edge judgment classifier detects the division boundary. The Inner Decode Unit of the second layer obtains the hidden state of the new triplet, generates the prediction of the next lower-level language segment based on the output of the first layer CDU, the output of the next layer Inner Encode Unit, and the hidden state of the new triplet, and the decoding is completed after the edge judgment classifier detects the division boundary.

[0079] In the decoding stage, dh t is itself derived from the decoding information of the segment of the upper layer, and here it is considered that dh t has a better ability to perceive the edge of the segment. Therefore, in the embodiments provided by the present application, dh t can be used to generate the signal value bs t .

[0080] Based on the above description, at time t, the input of the decoding unit of the lth layer includes the following contents:

[0081] 1) bs l-1,t , the edge signal from the next layer;

[0082] 2) i l (that is, eo l-1 ), from the input signal, which is also the output signal of the next layer Inner Encode Unit;

[0083] 3) dh t-1 : the hidden state of the Inner Decoder at time t-1;

[0084] 4) eh t-1 : hidden state of internal encoder from time t-1

[0085] 5) do l+1 , t, output from upper internal decoding unit at time t.

[0086] The conjugate decoding unit will make different operations and outputs according to the values of bs l,t and bs l-1,t , as shown in Fig. 4(a)~(c).

[0087] As shown in Fig. 4(a), when bs l-1,t = 0, it indicates that the (l-1)th layer has not reached its edge, so the lth layer only maintains the original state without any operation.

[0088] As shown in Fig. 4(b), when bs l-1,t = 1, bs l,t = 0, it indicates that the (l-1)th layer has reached its edge, while the lth layer has not reached its edge. At this time, the encoding unit of the lth layer generates a new state do l , t from the output of the encoding unit of the (l-1)th layer as the input of the decoding unit of the lth layer, and outputs it to the (l-1)th layer. The decoding unit of the (l-1)th layer takes do l , t as the initial state of the new segment of the (l-1)th layer.

[0089] As shown in Fig. 4(c), when bs l-1,t = 1, bs l,t = 1, it indicates that the lth layer and the (l-1)th layer reach the edge at the same time. The internal encoding unit of the lth CDU encodes the encoding result of the (l-1)th layer as the last piece of information of the current segment into the hidden state dh t of the internal decoding unit, and transmits it upwards, while discarding its current encoding state. After completing the decoding at time t, the decoding unit also discards its decoding state, and obtains the new decoding information of the next segment from the upper layer (the (l+1)th layer) as the hidden state dh t of its internal decoding unit.

[0090] In summary of all operations, in one embodiment provided by the present application, the decoding process of the neural network along the time expansion is illustrated by taking a triple as an example, as shown in Fig. 5. Figure 5

[0091] Figure 5 The decoding process shown in Fig. 5 occurs after the end of the encoding stage (not shown), and the output target is a plurality of triple information. Therefore, three layers of decoders need to be set:

[0092] ​1. Top layer decoder (l=3) which is a decoding unit of usual RNN form (typically lstm) for decoding the semantic vectors generated in the encoding phase at the triple level: sequentially decoding generates the next triple semantic vector which will be passed to the lower layer (l=2) for decoding at the finer semantic granularity (phrase) until the decoding output special token <eos>.

[0093] 2. Tuple decoder (l=2), decoding unit is CDU, the internal decoding unit of CDU decodes the triplet semantic vector decoded by l=3 layer to get phrase level decoding: sequentially decoding generates the phrase semantic vector of subject, relation, object of triplet, and hands over to the lower layer (l=1) for further decoding. Meanwhile, the internal encoding unit of CDU collects the re-encoding information of each phrase of the lower layer to re-encode the whole triplet. When the decoding of the current triplet is finished (bs 2,t =1), the re-encoding information of the triplet will be passed up to the top layer decoder.

[0094] 3. Phrase decoder (l=1), the decoding unit of this layer is CDU, the internal decoding unit of CDU decodes the phrase semantic vector decoded by l=2 layer to get word level decoding, sequentially outputs each word constituting the phrase, meanwhile, the internal encoding unit of CDU re-encodes all the generated words. When the decoding of the current phrase is finished (bs 1,t =1), the re-encoding information of the current phrase will be passed up to the (l=2) tuple decoder.

[0095] At the initial moment of the decoding stage (t=0), bs 1,0 , bs 2,0 in the phrase decoding layer and the tuple decoding layer are forced to be 1, the start symbol ' <sos>will be encoded in turn, passed to the top layer. The top decoding unit receives the semantic vectors of the encoding stage (not shown), as well as <sos>the encoding information, generates a semantic vector containing the first tuple (<Barack Obama, is the president of, US>), passes down layer by layer, and decodes the first word of the first phrase of the triple: Barack.

[0096] At the T=1 moment of the decoding stage, neither the phrase decoding nor the tuple decoding ends (bs 1,0 =0, bs 2,0 =0), the top layer decoding unit and the tuple decoding unit are in Fig. 4(a), and each state remains unchanged to pass to the next moment. While the phrase decoding unit collects the semantic information of Barack generated at t=0 to re-encode, and simultaneously predicts Obama.

[0097] At the T=2 moment, the phrase decoding unit collects the semantic information of Obama generated at t=1, and judges that the decoding of the first phrase of the first tuple ends (bs 1,0 =1), at this time the CDU of the phrase decoding layer is in Fig. 4(c), and the phrase <barackobama>The re-encoding information is passed up. The CDU of the tuple decoding layer is in Fig. 4(b) and will collect the re-encoding information of the phrase <Barack Obama> while generating the semantic vector of the second phrase <is the president of> of the first triple, which is passed down to the CDU of the phrase decoding layer and decodes the first word of the second phrase of the first triple: is.

[0098] T = 3 ~ 5, similar to T = 1. T = 6, similar to T = 2

[0099] T = 7, the phrase decoding unit collects the semantic information of the word US generated at t = 6, and judges that the decoding of the third phrase of the first triple is completed (bs 1,0 = 1), at this time the CDU of the phrase decoding layer is in Fig. 4(c) and will decode the phrase <us>re-encode information up the chain. The CDU of the tuple decode layer will collect the phrases <us>re-encode information of the first tuple and determine that the decoding of the first tuple is finished (bs 2,0 =1), the CDU of the tuple decoding layer is also in Fig. 4(c), and the re-encode information of the triplet <Barack Obama, is the president of, US> is passed upwards. At this time, the top encoding unit is in the state of Fig. 4(b), and based on the hidden state decoded at t=0 and the first triplet re-encode information received at the current time (t=7), the semantic vector of the second triplet <Barack Obama, gave speech to, thousands of people> is generated, and is passed downwards layer by layer, to predict the first word of the first phrase of the second triplet: Barack.

[0100] The time T=8 is similar to the time T=1, the time T=9 is similar to the time T=2, the time T=10-11 is similar to the time T=1, the time T=12 is similar to the time T=2, the time T=13-14 is similar to the time T=1, and the time T=15 is similar to the time T=7. At the time T=15, after the top decoder receives the re-encode information of the second tuple, the decoding result of the "next tuple" information generated is the special character <eos>At this time, it is considered that all the tuples have been decoded. At this point, the entire decoding process ends.

[0101] The hierarchical recurrent neural network decoder finally takes the cross-entropy of the boundary judgment result and the cross-entropy of the generated sequence result as a loss function, and optimizes the model based on the loss function through gradient descent method. If the predicted value of the generated sequence is denoted as y i , the reference sequence is denoted as y i , the edge signal value generated by the l-th layer at the i-th time is denoted as , and the reference value is denoted as , then the loss function can be expressed as:

[0102]

[0103] Among them, CrossEntropyLoss is a commonly used loss function: cross-entropy, see formula (5).

[0104] CrossEntropyLoss(x,z)=∑p(x i )logz i +∑(1-p(x i ))log(1-z i ) (5)

[0105] Among them, x is the predicted probability value, z=0 or 1, and is generally the reference value.

[0106] In summary, the hierarchical recurrent neural network decoder constructed by the application includes an edge judgment classifier, an LSTM internal decoding unit and an LSTM internal encoding unit. The edge judgment classifier takes the decoding LSTM states of the upper, middle and lower three layers as input and performs classification. The other two LSTM units are respectively used for bottom-up hierarchical encoding and top-down hierarchical decoding. For the task of outputting "requirement to generate output as semi-structured data", the target sequence contains segmented structure information. Then the model is verified. The verification result shows that the model has certain ability in segment edge perception.

[0107] S3, input the obtained input data into the hierarchical recurrent neural network decoder to obtain semi-structured output data.

[0108] Among them, the process of inputting the input data into the hierarchical recurrent neural network decoder for decoding to obtain semi-structured output data has been described in detail in the construction process of the neural network decoder, and will not be repeated here.

[0109] The following takes a given natural language sequence input hierarchical recurrent neural network decoder to obtain the output data containing a number of triples unordered set as an example to further illustrate the neural network-based open relationship extraction method provided by the present application.

[0110] Specifically, since the mainstream neural network model is essentially a classification model, the idea of open relationship extraction task is not directly applicable to the neural network model structure. In order to be able to apply deep neural networks to the specific task of open relationship extraction, the task is further analyzed and transformed:

[0111] The open relationship extraction task can be generally described as follows (denoted as Description A):

[0112] Task Description A:

[0113] For a given natural language sequence (usually in the form of a sentence):

[0114] S=w1,w2,w3,…,w n (n≥2) (6)

[0115] The output contains an unordered set of triples:

[0116] O={t1,t2,t3…t m}(m≥0, when the original sequence semantics does not contain any relationship, m=0) (m≥0, when the original sequence semantics does not contain any relationship, m=0) (7)

[0117] Where t i is a relationship triple:

[0118] t i =(sub i ,rel i ,obj i ) (i≥0) (8)

[0119] And (sub i ,rel i ,obj i ) is a sequence of words representing an entity or a relationship, each of which comes from the original sentence:

[0120]

[0121]

[0122]

[0123] Obviously, t i ,(sub i ,rel i ,obj i ),w i,1 respectively belong to three different granularity divisions of basic units. If the output is expressed according to the three granularity divisions, the following three expression forms of the output can be obtained:

[0124] O={t1,t2,t3…t m}(m≥0) (12)

[0125] O={(sub1,rel1,obj1),(sub2,rel2,obj2),…(sub i ,rel i ,obj i )}(i≥0) (13)

[0126]

[0127] It can be seen intuitively that the above sequence characteristics meet the data input characteristics of a hierarchical RNN neural network (HRNN) model. Due to the directionality of the triple relationship, the internal sequence of each output tuple is stable. However, since the task description A does not make special instructions on the sequence relationship between the tuples, the relative positions of the output tuples are not unique. In order to ensure the uniqueness of the output, in the embodiments provided in the present application, the output sequence in the form of a tuple is further constrained in a certain way:

[0128] It is specified that if the average value of the position indicators of the words in the tuple in the original sentence is small (the average position is in the front), the tuple is in the front in the generated tuple sequence.

[0129] Therefore, the unique output result can be obtained, that is, under the premise of a given input sequence, the state of the output sequence (12), (13), (14) is unique.

[0130] In addition, by comparing and observing the output sequences (12), (13), (14), it can be seen intuitively that the output sequences meet the data characteristics of the hierarchical RNN neural network (HRNN). Considering the output requirements of the hierarchical RNN model, the open relationship extraction task can be converted into the following description form (denoted as description B):

[0131] Task description B:

[0132] For a given natural language word sequence input:

[0133] sequence=[w1,w2,w3,……,w4] (15)

[0134] The A word sequence is required to be output:

[0135] O = [w'1, w'2,... w'N] (1) m ](m>=3) (16)

[0136] and B two sets of binary signal sequences:

[0137]

[0138]

[0139] where the word sequence satisfying can be segmented according to the signal sequence of Layer1, Layer2, and each segment after segmentation meets the requirements of the triple rule;

[0140] and C two sets of boundary signal (bs) sequences.

[0141] where, The boundary signal corresponding to the relationship or entity word sequence: indicates that w i ' is the boundary of the entity (relationship) phrase.

[0142] The boundary signal corresponding to the tuple. indicates that w i ' is the boundary of the tuple.

[0143] Based on the example input and output of expression B as follows:

[0144] Original text:

[0145] The U.S. president Barack Obama gave his speech on Tuesday to thousands of people.

[0146] Output:

[0147] Layer 1, bs 1 ]]> Layer 2, bd 2 ]] Barack 0 1 Obama 1 0 is 0 0 the 0 0 president 0 0 Of 1 0 the 0 0 U.S. 1 1 Barack 0 0 Obama 1 0 gave 0 0 speech 0 0 to 1 0 thousands 0 0 of 0 0 people 1 1

[0148] Table 1. Example of converted open relationship extraction task output

[0149] Obviously, after the conversion of the task description B, the original task of the entity, relationship, triple sequence and other sequence constraints are replaced by the boundary signal which is easier to process, and the possible sequence of the output is unique. This description will make the open relationship extraction task more in line with the characteristics of the neural network model, especially the generation data characteristics of the hierarchical RNN of the CDU decoding unit.

[0150] After the transformation of the task description, we can apply the CDU decoding unit to the open relation extraction task in the general field to test the performance of the CDU model on this type of task.

[0151] Unlike the Chinese word segmentation task, the open relation extraction task requires the model to complete all the predictions from the structure to the sequence. The effect of the open relation extraction method based on the neural network provided by the application in the open relation extraction in the field of microorganisms is verified.

[0152] In the experiment, the commonly used LSTM unit is selected as the encoder to model the input sentence, and the 2-layer CDU is selected as the open relation extraction decoder.

[0153] The data set selected in experiment 1 is mainly the open relation extraction data set constructed by Lei Cui et al. in the general field.

[0154] The pre-processing of the data set in the experiment includes:

[0155] 1) Replace the number with a special character <num>;

[0156] 2) Considering the interference of bracket information on language structure and semantics, remove the sentence brackets and the content in them;

[0157] 3) For easy comparison, all English characters are converted to lowercase;

[0158] 4) Delete the remaining special characters.

[0159] In this experiment, the commonly used accuracy (P), recall (R) and F1 value are selected as the evaluation indexes of model performance. The specific calculation method and the evaluation standard of the similarity between triplets can be referred to

[0160] In this experiment, these evaluation indexes can be expressed as:

[0161] Accuracy (P):

[0162]

[0163] Recall (R):

[0164]

[0165] F1 value:

[0166]

[0167] Among them, the standard of judging the correct prediction of triplets can be expressed as:

[0168]

[0169] In terms of the selection of benchmark model method, this experiment selects the triplets with confidence higher than 80% in the OLLIE system and the extraction results of the SOIE system as the benchmark of this experiment.

[0170] Experimental conclusion:

[0171] After multiple rounds of cross-validation, the average value of the experimental results of each system is obtained, and the accuracy, recall and F1 value corresponding to the three systems are shown in the following table:

[0172] Model R P F OLLIE 73.8％ 70.5％ 72.11％ SOIE 71.5％ 73.2％ 72.34％ CRDU 73.8％ 74.5％ 74.14％

[0173] Table 2 Experimental results of open relation extraction in general field

[0174] It can be seen that compared with OLLIE and SOLIE systems, the hierarchical RNN model based on CDU shows higher performance in this dataset, but not outstanding.

[0175] In the second experiment, the data set is mainly from the annotated data of Pubmed in the US biomedical database MEDLINE (mainly containing the title and abstract of the biomedical literature database), and part of it is from the corpus data of "disease-gene" relationship by Bravo et al., and the mixed data set of "disease-symptom" relationship by Barbara Rosario et al.

[0176] In this experiment, some preprocessing is also done for the data set:

[0177] 1) Replace numbers with special characters <num>;

[0178] 2) Remove the sentence brackets and their contents, considering the interference of bracket information to language structure and semantics;

[0179] 3) Replace some strings that match chemical formula features with special characters <num>;

[0180] 4) For easy comparison, all English characters are converted to lowercase;

[0181] 5) Delete the remaining special characters.

[0182] In this experiment, the commonly used accuracy (P), recall (R) and F1 value are selected as the evaluation indexes of the model performance. The specific calculation method and the evaluation standard of the similarity between triplets can be referred to

[0183] In this experiment, these evaluation indexes can be described in detail (4.14-4.16), and the standard expression of the correct prediction of triplets is described in detail (4.17).

[0184] In the selection of benchmark methods, in this experiment, the triplets with a confidence higher than 80% in the OLLIE system and the extraction results of the SOIE system are selected as the benchmark of this experiment.

[0185] 4.5.4 Experimental conclusion

[0186] After multiple cross-validation and averaging the experimental results of each system, the accuracy, recall and F1 value of the three systems in the open relation extraction technology in the field of microorganisms are as follows:

[0187] Model R P F OLLIE 70.3％ 68.7％ 69.9％ SOIE 70.9％ 72.5％ 71.6％ CDU+MetaMap 74.2％ 75.1％ 74.6％

[0188] Table 3 Experimental results of open relation extraction in the field of microorganisms

[0189] As can be seen from the results, thanks to the relatively accurate concept recognition and the ability of the CDU-based hierarchical RNN model itself in open relation extraction, the algorithm after dissimilation has obviously shown higher performance than OLLIE in this data set.

[0190] The above has carried on the detailed explanation to the open relation extraction method based on neural network provided by the application. For the general technical personnel in the art, any obvious change made to it without departing from the essential spirit of the application will constitute an infringement of the patent right of the application, and will bear the corresponding legal responsibility.< / num> < / num> < / num> < / eos> < / us> < / us> < / barackobama> < / sos> < / sos> < / eos> < / action1> < / action1>

Claims

1. A neural network-based open relation extraction method, characterized in that The method comprises the following steps: The biological entity concept in the microorganism field data in the text is replaced by the corresponding concept type to obtain input data; The hierarchical recurrent neural network decoder is constructed by the following steps: setting an internal decoding unit at the top layer according to the hierarchical division of the hierarchical recurrent neural network, and setting a conjugate decoding unit at each layer except the top layer; the conjugate decoding unit comprises an edge judgment classifier, an internal decoding unit and an internal encoding unit; wherein the edge judgment classifier is used to perceive the division boundary of the layer; the internal encoding unit encodes the input from the next layer, and transmits the encoded information to the previous layer of the current layer when reaching the division boundary of the layer of the current layer; the internal decoding unit predicts the output of the internal decoding unit next layer according to the input of the previous layer, the output of the internal encoding unit next layer and the hidden state of the internal decoder; The input data obtained is input into the hierarchical recurrent neural network decoder to obtain semi-structured output data; wherein the internal decoding unit of the top layer of the hierarchical recurrent neural network decodes the input data to generate semantic fragment information; according to the semantic fragment information generated by the internal decoding unit of the previous layer, the internal encoding unit generates semantic information of the next layer granularity one by one, and the generated semantic information is re-encoded by the internal encoding unit to generate re-encoded information; when the edge judgment classifier determines that the internal decoding unit is at the division boundary, the decoding is stopped, and the re-encoded information is returned. 2.The neural network-based open relation extraction method of claim 1, wherein The biological entity concept in the microorganism field data is replaced by the corresponding concept type to obtain input data, comprising the following steps: The microorganism field concept in the text is identified in the form of dictionary matching; Each concept type to which the words in the sentence belong is marked; The rare words in the biological field data are replaced by the corresponding concept type to obtain the input data.

3. The neural network-based open relationship extraction method of claim 1, wherein: At time t, the input of the internal decoding unit of the lth layer comprises: the edge signal from the (l-1)th layer, the output signal of the internal encoding unit of the (l-1)th layer, the hidden state of the internal decoding unit at time t-1, the hidden state of the internal encoding unit at time t-1, and the output of the internal decoding unit of the (l+1)th layer at time t.

4. The neural network-based open relationship extraction method of claim 3, wherein: When the output signal of the inner encoding unit of the (l-1)th layer is When the output signal of the inner encoding unit of the (l-1)th layer is 5. The neural network-based open relationship extraction method of claim 3, wherein: When the output signal of the internal encoding unit of the (l-1)th layer is 1 and the output signal of the internal encoding unit of the lth layer is 0, the (l-1)th layer reaches the division boundary, the lth layer does not reach the division boundary, and the internal encoding unit of the lth layer generates a new state with the output of the internal encoding unit of the (l-1)th layer as the input of the internal decoding unit of the lth layer , and outputs to the (l-1)th layer, and the internal decoding unit of the (l-1)th layer takes as the initial state of the new segment of the (l-1)th layer; wherein is the output of the conjugate decoding unit of the lth layer at time t.

6. The neural network-based open relationship extraction method of claim 3, wherein: When the output signals of the internal encoding units of the (l-1)th layer and the lth layer are both 1, the lth layer reaches the division boundary at the same time as the (l-1)th layer; the internal encoding unit of the lth layer encodes the encoding result of the (l-1)th layer as the last piece of information of the current segment into the hidden state of the internal decoding unit and transmits it upward, while discarding its current encoding state; after completing the decoding at the t time, the internal decoding unit discards its decoding state, and obtains the new decoding information of the next segment from the (l+1)th layer as the hidden state of the internal decoding unit.

7. The neural network-based open relation extraction method of claim 1, wherein: the loss function is based on a boundary judgment result and a cross-entropy of a generated sequence result.

8. The neural network-based open relation extraction method of claim 1, wherein: the output data is in units of tuples, and each word in a tuple is in an earlier average position in the original sentence, so the tuple is in an earlier position in the generated tuple sequence. ​ ​ ​

Images