A method and system for extracting text causal relationship based on time relationship

Abstract

The present invention provides a method and system for extracting text causal relationships based on temporal relationships, including: marking the relationships between words in a sentence to generate a relationship matrix; constructing a TC-GAT model; training the TC-GAT model through the words in the sentence and the relationship matrix; and extracting the causal relationship of the sentence through the trained TC-GAT model. The present invention improves the accuracy of causal relationship extraction through a graph neural network model that balances graph attention mechanism and self-attention mechanism.

Description

Technical Field

[0001] The present invention relates to the field of intelligent devices, and in particular to a method and system for extracting text causal relationships based on time relationships. Background Art

[0002] Causal extraction is a basic task in NLP (Neuro-Logical Pattern). Causal extraction is to obtain the cause and effect relationship between entities in the text. Through causal analysis, events can be predicted and the causes of events can be inferred. Causal extraction can be applied to many fields, such as finance, medical reasoning analysis, etc. Causal extraction is also often used in question-answering systems to help give the reasons for various questions. The time relationship plays an important role in the extraction of causal relationships.

[0003] Currently, there are few models that use the temporal relationship between events for causal reasoning. Mostafazadeh et al. proposed a labeling structure (CaTeRS) to obtain temporal relationship discrimination related to causal relationships through a large number of story-like events. Ning et al. proposed a joint learning framework TCR, which integrates conditional constraint models and integer linear programming to infer event causal relationships that meet specific rules. The above two typical works are limited to the use of feature engineering and statistical machine learning methods; moreover, causal relationships must have temporal relationships, but temporal relationships are not necessarily accompanied by causal relationships, and there is a lack of public data sets that simultaneously mark both relationships; therefore, a balanced mechanism is needed to balance the impact of temporal information on causal relationship extraction, so as to improve the accuracy of causal relationships obtained by causal reasoning using temporal relationships between events. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for extracting text causal relationships based on time relationships, which can improve the accuracy of causal relationship extraction.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method and system for extracting text causal relationships based on time relationships, the method comprising:

[0007] Mark the relationship between words in a sentence and generate a relationship matrix;

[0008] Construct TC-GAT model;

[0009] Training the TC-GAT model using the words in the sentence and the relationship matrix;

[0010] The causal relationship of the sentence is extracted through the trained TC-GAT model.

[0011] Optionally, the TC-GAT model includes a BiLSTM model, a GAT model, a BERT model, and a gating model of an equalization mechanism.

[0012] Optionally, the training the TC-GAT model by using the words in the sentence and the relationship matrix specifically includes:

[0013] Taking the words in the sentence as input, training the BiLSTM model, and outputting a bidirectional vector of the words in the sentence;

[0014] Taking the bidirectional vectors of the words in the sentence and the relationship matrix as input, training the GAT model, and outputting the graph attention features of the words in the sentence;

[0015] Taking the words in the sentence as input, training the BERT model, and outputting the states of the words in the sentence;

[0016] Performing vector concatenation of the state of the word in the sentence and the word embedding of the word in the sentence to obtain the self-attention feature of the word in the sentence;

[0017] The graph attention features of the words in the sentence and the self-attention features of the words in the sentence are used as input to train the gating model of the balancing mechanism to obtain the causal relationship of the sentence.

[0018] Optionally, the BiLSTM model is:

[0019] H BiLSTM =BiLSTM([E1,...,E t ,...,E l ]);

[0020]

[0021] Among them, E t ∈IR l×n is the t-th word embedding, l is the length of the sentence, is the bidirectional vector representation of the tth word in the sentence generated by BiLSTM, is the output obtained by the forward lstm, is the output obtained by backward lstm, H BiLSTM is the output of the BiLSTM model.

[0022] Optionally, the gating model of the balancing mechanism is:

[0023]

[0024]

[0025] Among them, W g is the parameter matrix, b g is the bias term, is the vector concatenation operation, g is the equalization coefficient, h t ′ is the vector representation of the final output of the equalization mechanism, α∈IR 2l×m is the coefficient weight matrix, is the sum of the state and word embedding of the tth word generated by the top four hidden layers of BERT, Graph attention features for words in a sentence.

[0026] A system for extracting text causal relationships based on temporal relationships, the system comprising: a labeling module and a construction module;

[0027] The annotation module is used to annotate the relationship between words in the sentence and generate a relationship matrix;

[0028] The construction module is used to construct a TC-GAT model, take the words in the sentence and the corresponding relationship matrix as input, train the TC-GAT model, and output the causal relationship of the sentence.

[0029] Optionally, the building modules include a BiLSTM model module, a GAT model module, a BERT model module and a gating model module of a balancing mechanism.

[0030] Optionally, the BiLSTM model module is used to train an input bidirectional long-term memory network model using word embeddings of words in the sentence as input, and output a bidirectional vector of the words in the sentence;

[0031] The GAT model module is used to train a graph attention network model using the bidirectional vector of the words in the sentence and the relationship matrix as input, and output the graph attention features of the words in the sentence;

[0032] The BERT model module includes a BERT pre-trained language representation model unit and a concatenation unit; the BERT pre-trained language representation model unit is used to take the words in the sentence as input, train the BERT pre-trained language representation model, and output the state of the words in the sentence; the concatenation unit is used to concatenate the state of the words in the sentence with the word embedding of the words in the sentence to obtain the self-attention features of the words in the sentence;

[0033] The gating model module of the attention balancing mechanism is used to train the gating model of the balancing mechanism with the graph attention features of the words in the sentence and the self-attention features of the words in the sentence as input to obtain the causal relationship of the sentence.

[0034] Optionally, the BiLSTM model is:

[0035] H BiLSTM =BiLSTM([E1,...,E t ,...,E l ]);

[0036]

[0037] Among them, E t ∈IR l×n is the t-th word embedding, l is the length of the sentence, is the bidirectional vector representation of the tth word in the sentence generated by BiLSTM, is the output obtained by the forward lstm, is the output obtained by backward lstm, H BiLSTM is the output of the BiLSTM model.

[0038] Optionally, the gating model of the balancing mechanism is:

[0039]

[0040]

[0041] Among them, W g is the parameter matrix, b g is the bias term, is the vector concatenation operation, g is the equalization coefficient, h t ′ is the vector representation of the final equalization mechanism output, α∈IR 2l×m is the coefficient weight matrix, is the sum of the state and word embedding of the tth word generated by the top four hidden layers of BERT, Graph attention features for words in a sentence.

[0042] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0043] The present invention provides a method and system for extracting text causal relationships based on temporal relationships, including: marking the relationships between words in a sentence to generate a relationship matrix; constructing a TC-GAT model; training the TC-GAT model through the words in the sentence and the relationship matrix; and extracting the causal relationship of the sentence through the trained TC-GAT model. The present invention improves the accuracy of causal relationship extraction through a graph neural network model that balances graph attention mechanism and self-attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 A flow chart of a method for extracting text causal relationships based on time relationships according to the present invention;

[0046] Figure 2 It is a structural block diagram of the system for extracting text causal relationship based on time relationship of the present invention;

[0047] Figure 3 It is a schematic diagram of the time sequence on the time axis of the present invention;

[0048] Figure 4 A schematic diagram of marking the relationship between words in a sentence according to the present invention;

[0049] Figure 5 The overall framework diagram of the TC-GAT model constructed by the present invention;

[0050] Figure 6 A network structure diagram of the GAT in the TC-GAT model constructed in the present invention;

[0051] Figure 7 A schematic diagram of a relationship matrix between words in a sentence according to the present invention;

[0052] Figure 8 The figure shows the comparison results of precision (P), recall (R) and F1 score (F) in the SemEval2010 Task8 dataset;

[0053] Fig. 9 Schematic diagram of the comparison results of precision (P), recall (R) and F1 score (F) in the TC-AltLex dataset;

[0054] Fig.10The invention marks the news example sentences to obtain the corresponding time relationship diagram;

[0055] Fig.11 This is a schematic diagram of the results of applying the TC-GAT model to predict the causal relationship of news sentences in advance.

[0056] Explanation of symbols:

[0057] Labeling module—1, construction module—2, BiLSTM model module—3, GAT model module—4, BERT model module—5, gating model module of balancing mechanism—6. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The purpose of the present invention is to provide a method and system for extracting text causal relationships based on time relationships, which can improve the accuracy of causal relationship extraction.

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] like Figure 1 As shown, the method for extracting text causal relationship based on time relationship provided by the present invention includes:

[0062] Step 101: Mark the relationships between words in a sentence and generate a relationship matrix.

[0063] Step 102: Construct a TC-GAT model. Specifically, the TC-GAT model includes a BiLSTM model, a GAT model, a BERT model, and a gating model of an equalization mechanism.

[0064] Step 103: Train the TC-GAT model using the words and relationship matrix in the sentence. Specifically, it includes:

[0065] The BiLSTM model is trained with the words in the sentence as input, and the output is the bidirectional vector of the words in the sentence.

[0066] Specifically, the BiLSTM model is:

[0067] H BiLSTM =BiLSTM([E1,...,Et ,...,E l ]);

[0068]

[0069] Among them, E t ∈IR l×n is the t-th word embedding, l is the length of the sentence, n is the dimension of the sentence, is the bidirectional vector representation of the tth word in the sentence generated by BiLSTM, is the output obtained by the forward lstm, is the output obtained by backward lstm, H BiLSTM is the output of the BiLSTM model.

[0070] The GAT model is trained with the bidirectional vectors and relationship matrices of the words in the sentence as input, and the output is the graph attention features of the words in the sentence.

[0071] The BERT model is trained with the words in the sentence as input, and the output is the state of the words in the sentence.

[0072] The state of the word in the sentence is vectorized with the word embedding of the word in the sentence to obtain the self-attention feature of the word in the sentence.

[0073] The graph attention features of the words in the sentence and the self-attention features of the words in the sentence are used as input to train the gating model of the balancing mechanism to obtain the causal relationship of the sentence.

[0074] Specifically, the gating model of the balancing mechanism is:

[0075]

[0076]

[0077] Among them, W g is the parameter matrix, b g is the bias term, is the vector concatenation operation, g is the equalization coefficient, h′ t is the vector representation of the final equalization mechanism output, α∈IR 2l×m is the coefficient weight matrix, l is the sentence length, m is the dimension of the coefficient weight matrix, is the sum of the state and word embedding of the tth word generated by the top four hidden layers of BERT, Graph attention features for words in a sentence.

[0078] Step 104: Extract the causal relationship of the sentence through the trained TC-GAT model.

[0079] like Figure 2 As shown, the system for extracting text causal relationships based on time relationships provided by the present invention includes: a labeling module 1 and a construction module 2.

[0080] The annotation module 1 is used to annotate the relationship between words in a sentence and generate a relationship matrix.

[0081] The construction module 2 is used to construct the TC-GAT model, which takes the words in the sentence and the corresponding relationship matrix as input, trains the TC-GAT model, and outputs the causal relationship of the sentence. Specifically, the construction module 2 includes the BiLSTM model module 3, the GAT model module 4, the BERT model module 5, and the gated model module 6 of the balancing mechanism.

[0082] Furthermore, the BiLSTM model module 3 is used to train the input bidirectional long-term memory network model with the word embedding of the word in the sentence as input, and output the bidirectional vector of the word in the sentence. Specifically, the BiLSTM model is:

[0083] H BiLSTM =BiLSTM([E1,...,E t ,...,E l ]);

[0084]

[0085] Among them, E t ∈IR l×n is the t-th word embedding, l is the length of the sentence, n is the dimension of the sentence, is the bidirectional vector representation of the tth word in the sentence generated by BiLSTM, is the output obtained by the forward lstm, is the output obtained by backward lstm, H BiLSTM is the output of the BiLSTM model.

[0086] GAT model module 4 is used to train the graph attention network model with the bidirectional vectors and relationship matrix of the words in the sentence as input, and the output is the graph attention features of the words in the sentence.

[0087] The BERT model module 5 includes a BERT pre-trained language representation model unit and a splicing unit; the BERT pre-trained language representation model unit is used to train the BERT pre-trained language representation model with words in a sentence as input, and the output is the state of the words in the sentence; the splicing unit is used to vector splice the state of the words in the sentence with the word embedding of the words in the sentence to obtain the self-attention features of the words in the sentence.

[0088] The gating model module 6 of the balancing mechanism is used to train the gating model of the balancing mechanism with the graph attention features of the words in the sentence and the self-attention features of the words in the sentence as input to obtain the causal relationship of the sentence. Specifically, the gating model of the balancing mechanism is:

[0089]

[0090]

[0091] Among them, W g is the parameter matrix, b g is the bias term, is the vector concatenation operation, g is the equalization coefficient, h t ′ is the vector representation of the final output of the equalization mechanism, α∈IR 2l×m is the coefficient weight matrix, l is the sentence length, m is the dimension of the coefficient weight matrix, is the sum of the state and word embedding of the tth word generated by the top four hidden layers of BERT, Graph attention features for words in a sentence.

[0092] The method and system for extracting text causal relationships based on time relationships provided by the present invention are applied in information extraction scenarios. In information extraction scenarios, most entities and events are interdependent and often present co-occurrence relationships, and the purpose of causal relationship extraction is to mark causal entities in the text describing events. Through summary, it is found that the occurrence of causal relationships is often accompanied by time clues, namely:

[0093] First, the cause occurs before the effect, and the effect occurs after the cause.

[0094] Second, an event that occurs simultaneously with a cause (result) event has a higher probability of being a result (cause) event.

[0095] Third, events containing cause entities have a higher probability of being cause events, and vice versa.

[0096] like Figure 3 As shown, in order to illustrate the relationship between time relationship and causal relationship, two examples of co-occurrence of time relationship and causal relationship are given, as shown below:

[0097] Specific example 1, The <e1:rain>casedthe <e2:floods>.

[0098] Specific example 2, Heavy <e1:rain>has been falling,causing <e2:floods>, people’shouses were <e3:damaged>,and somepeople <e4:died>in the flood.

[0099] Specific example 1 is a simple temporal relationship, that is, "rain" (cause) occurs before "floods" (before), and contains a bunch of causal relationship entities. Specific example 2 expresses a complex temporal relationship, "rain" contains "flood", and due to the symmetry of time, "flood" is also involved in the relationship, while "damaged" and "died" occur at the same time, both appearing after "rain" and "flood" as the result of the cause "rain". Therefore, predicting causal relationships can benefit from considering temporal relationships. In addition, the use of temporal and causal relationship information can better improve the performance of downstream tasks, such as dialogue generation, machine reading comprehension, and knowledge graph construction.

[0100] The design ideas of the method and system for extracting text causal relationships based on time relationships provided by the present invention are as follows:

[0101] In order to make full use of the time information in the text and construct a time relationship diagram for each sentence, the present invention first designs a labeling system based on the existing methods that can fully characterize the time relationship between events in the sentence and generate a relationship matrix.

[0102] Temporal relationships are reflected in the entire life cycle of an entity. Influenced by the entity life cycle, temporal relationships can be briefly divided into five types: Before (B), After (A), Include (I), Be Included (N), and Simultaneous (S). Finer-grained divisions will produce more types, increase the difficulty of data annotation, and reduce the performance of causal relationship extraction.

[0103] Specifically, Figure 3 As shown in the figure, for the five events e1, e2, e3, e4 and e5, e1 occurs before e2, e3, e4 and e5. Due to the symmetry of the time relationship, e2, e3, e4 and e5 occur after e1; while e2 and e3 occur at the same time, e4 and e5 belong to the inclusion and inclusion relationship. Since the currently disclosed causal relationship dataset does not contain time relationship information, the present invention proposes a data annotation mode with time relationship according to the above rules, which can generate a relationship matrix. Figure 4 As shown in Figure 1, the annotation system mainly includes time tags and causal tags. For an entity, the entity itself is marked with R, followed by its corresponding time relationship and other entities. Furthermore, R indicates that the entity has a time relationship, M indicates itself in the time relationship, and B and A indicate that the time relationship between the events rain and flood occurs before and after, respectively.

[0104] like Figure 5 As shown, the present invention proposes a model TC-GAT (Temporal-Causality by GraphAttentionNetwork) for predicting causal relationships using temporal relationships. The model consists of two main steps: the first step is to introduce temporal relationships into causal relationship extraction, obtain the graph attention representation of temporal information through the graph neural network, encode the sentence through BERT, and obtain the self-attention representation of contextual information; the second step is to use the balance mechanism to balance the influence of temporal information and self-attention information on causal relationship extraction. First, the vector representation of the sentence containing the causal entity is obtained through word embedding. Then enter the two parallel information flow paths designed by this method: one is to input the sentence vector into BiLSTM for shallow encoding, and then based on the temporal relationship information, take the word as the node, and the BiLSTM output as the node initial vector, to construct a graph network, and obtain the feature vector based on graph attention through GAT; the other is to splice the word embedding into BERT for deep encoding to obtain the feature vector based on self-attention. Finally, the two features are adjusted through a balancing mechanism to extract causal entities. Specifically, in the initialization stage, the temporal relationship of events in the input sentence is given according to the annotation system proposed in this patent. After the two information paths of graph attention and self-attention, feature fusion is performed to finally obtain Figure 5 The label sequence in the lower right corner, i.e., rain is the cause (Cause, C) and flood is the effect (Effect, E).

[0105] Among them, the graph attention-based features are obtained through the BiLSTM model and the GAT model.

[0106] Furthermore, the bidirectional long-term memory network (BiLSTM) has three gate structures (input gate, forget gate and output gate) to selectively forget some historical information, add some current input information, and include time-variable features from front to back and from back to front, which are finally integrated into the current time step and generate an output state. BiLSTM is widely used in natural language processing tasks because of its characteristics that each time step can correspond to a word and combine two directional information. According to the above advantages, considering that the graph structure representing the temporal relationship between events pays more attention to the location and direction of the event, and does not require complex semantic understanding, the present invention uses BiLSTM for shallow encoding. The word embedding corresponding to sentence x is used as the input of BiLSTM, and the context state is output:

[0107] H BiLSTM =BiLSTM([E1,...,E t ,...,E l ]);

[0108]

[0109] Among them, E t ∈IR l×n is the t-th word embedding, l is the length of the sentence, n is the dimension of the sentence, is the bidirectional vector representation of the tth word in the sentence generated by BiLSTM, is the output obtained for the forward long short-term memory, is the output obtained from the backward long short-term memory, H BiLSTM is the output of the BiLSTM model.

[0110] Further, such as Figure 6 As shown, the graph attention network (GAT) proposes a method for weighted summation of adjacent node features using an attention mechanism. The weights of adjacent node features depend entirely on the node features and are independent of the graph structure. GAT replaces the fixed normalization operation in the graph convolutional network (GCN) with an attention mechanism. In essence, GAT combines the original GCN normalization function with a neighbor node feature aggregation function using attention weights. In the model of the present invention, we use GAT to obtain the attention weights of temporal relationships, which can effectively integrate different temporal relationships into the model. Compared with GCN, GAT indicates the importance of each temporal relationship and each event word within the relationship, which helps to improve the accuracy of prediction.

[0111] To construct the time relationship graph, first, use the time tags in the sentence to generate a time matrix. As shown in Figure 7, each word is marked with a self-loop time relationship M; words without a time relationship are marked with O; mutual time relationships include A (After), B (Before) and S (Simultaneous) to indicate the order of occurrence, and N (be-include) and I (Include) to indicate the subordinate relationship. The corresponding matrix adj is obtained. A 、adj B 、adj S 、adj N and adj I , corresponding to five types of mutual time relations, and the self-loop time relation matrix adj M .

[0112] Specifically, we use the self-loop time relationship and the five mutual time relationships as nodes to calculate the similarity coefficient, and use the word's own M as the adjacent node for each mutual time type, that is:

[0113] e r =α([h t W M ,h t W r ]);

[0114] Among them, W r ∈ n×m is the weight matrix for each temporal relationship, is the node feature initialized with BiLSTM output, α∈IR 2l×m is the coefficient weight matrix, t represents the tth word, h t represents the output of the previous layer, W M Represents the weight coefficient corresponding to the M relationship, e r Represents the similarity coefficient.

[0115] After obtaining the similarity coefficient, we can merge various relationship matrices, namely:

[0116]

[0117] Among them, α r is the attention weight of the relation, adj r ∈{adj B , adj A , adj S , adj I , adj M , adj N } is the corresponding time relationship matrix, K is a relationship in R, e k Represents the similarity coefficient of the corresponding time relationship, LeakyReLU(e r ) and LeakyReLU(e k ) are activation functions.

[0118] According to the calculated attention weights, the node features can be updated, namely:

[0119]

[0120] in, represents the obtained feature output, σ represents the sigmoid function, W r ′ represents the weight coefficient corresponding to the time relationship.

[0121] Finally, the present invention also uses a multi-head attention mechanism to divide the features into multiple subspaces, and then merges the node features of each subspace to obtain the final result, namely:

[0122]

[0123] Among them, K represents the number of heads of the multi-head attention mechanism, k and t both represent counts and have no practical meaning, and h t represents the output of the previous layer, represents the attention weight of the k-th head corresponding to the time relationship, W r k Represents the temporal relationship weight corresponding to the k-th head.

[0124] Specifically, the self-attention based features are obtained through the BERT model.

[0125] Furthermore, BERT is one of the most famous pre-trained language models today. This method believes that the traditional unidirectional language model or the shallow concatenation method of two unidirectional language models is not suitable for pre-training, and proposes a more efficient masked language model that can generate vector representations with richer semantic information.

[0126] In the model of the present invention, we use the vector obtained by the previous word embedding to integrate it into the BERT module to constrain the two information flow paths to learn text representations without too much difference. In BERT, for each word in sentence S, its embedding representation (embedding) is generated by adding its corresponding word embedding (word embedding), segment embedding (segment embedding) and position embedding (position embedding). Then fill the identifiers at the beginning and end of the sentence, CLS represents the beginning of the sentence, and SEP represents the end of the sentence. The input of the subsequent Transformer of BERT is [CLS, E S ,SEP], that is:

[0127] H BERT =BERT([CLS,x1,...,x t ,...,x l ,SEP]);

[0128]

[0129] Among them, E t ∈IR n represents the word embedding of the tth word used in the path based on the graph attention feature, represents the sum of the state and word embedding of the tth word generated by the top four hidden layers of BERT, E S represents the embedding representation of sentence S, H BERT represents the output representation obtained by BERT, x t represents the tth word in the sentence, l represents the length of the sentence, and Represent the outputs of the last four layers of the BERT model respectively.

[0130] Specifically, the influence of temporal relationships on causal relationships is balanced through the attention balance mechanism.

[0131] Furthermore, the temporal relationship has a certain negative impact on the extraction of causal relationships. Since the occurrence of event entities often contains a time sequence, but there is not necessarily an obvious causal relationship between events. Therefore, event entities with temporal relationships will affect the model's judgment of causal relationships. Therefore, the present invention proposes a balancing mechanism unit, the purpose of which is to balance the impact of temporal relationships on causal relationships by learning an information gating, that is, to control the degree of influence of graph attention-based features and self-attention-based features on the final vector representation of word t. The gating design of the balancing mechanism is as follows:

[0132]

[0133]

[0134] Among them, W g is the parameter matrix, b g is the bias term, is the vector concatenation operation, g is the equalization coefficient, h t ′ is the vector representation of the final output of the equalization mechanism, α∈IR 2l×m is the coefficient weight matrix, l is the sentence length, m is the dimension of the coefficient weight matrix, is the sum of the state and word embedding of the tth word generated by the top four hidden layers of BERT, Graph attention features for words in a sentence.

[0135] Finally, the TC-GAT model prediction and optimization include:

[0136] First, in order to predict the label sequence corresponding to the input sentence, the word features obtained by the equalization mechanism are mapped to the label space and then normalized, that is:

[0137] P(x t )=softmax(W o h′ t +b o );

[0138] Among them, P(x t ) represents the predicted value of the tth word in the sentence, W O represents the weight, b o represents the offset value, and o represents the difference from the previous value.

[0139] Secondly, during training, we use the cross entropy loss function as the model optimization objective, namely:

[0140]

[0141] Among them, L represents the loss value, t represents the tth word, l represents the sentence length, and y t represents the true label of the t-th word.

[0142] The experimental results of the TC-GAT model provided by the present invention on two data sets, SemEval2010 Task8 and TC-AltLex, show that compared with the existing baseline model based on deep learning, the causal relationship extraction accuracy is higher and the performance is better.

[0143] In addition, if Figure 8 As shown in the figure, compared with the SemEval2010 Task8 dataset, the newly annotated dataset TC-AltLex uses manually labeled temporal relations in addition to causal relations, contains fewer repeated causal entities, and most sentences contain multiple pairs of causal entities. The temporal relations are more random and complex, and the negative impact is greater. Therefore, the accuracy of all baseline models will be reduced. However, our model can still get better results, which shows that our model has stronger generalization and is more stable.

[0144] like Fig. 9 As shown in the figure, in the experiment, the LSTM, BiLSTM, BERT and BiLSTM+CRF models did not use the time relationship, and the extraction results could only be obtained by relying on context features, and the overall performance was poor. After the introduction of the time relationship, the extraction effects of the C-GCN and TC-GAT models of the present invention were significantly improved, proving that the time relationship has a positive impact on the recognition of causal relationships. At the same time, by comparing the C-GCN model and the TC-GAT model, we found that the introduction of the balancing mechanism has a good effect on offsetting the negative impact of the time relationship.

[0145] The present invention proposes a deep learning-based model TC-GAT, which uses a deep neural network to extract causal relationships based on temporal relationships. Specifically, the graph attention network GAT is first used to obtain the temporal information of the modeled sentence, and then the contextual information of the sentence is encoded through BERT. Although temporal information can intuitively characterize causal relationships, it also has a negative impact on prediction, that is, causal relationships must have temporal relationships, but temporal relationships are not necessarily accompanied by causal relationships. Therefore, the present invention proposes a balancing mechanism that balances the impact of temporal information on causal relationship extraction. Comparative experiments show that this method significantly improves the accuracy of causal relationship extraction.

[0146] Another major obstacle to using temporal relations to discover causal relations is the lack of public datasets that label both relations. This paper designs a labeling mode to characterize temporal relation information. We practiced it on two datasets: first, adding time tags to the SemEval2010 Task8 dataset and selecting other relation samples as negative samples to add time tags; second, sampling some data from the Wikipedia causal relation corpus and manually labeling sentences with time tags and causal tags.

[0147] like Fig.10 and Fig.11 As shown, as an embodiment of an implementation method and system for extracting text causal relationships based on time relationships provided by the present invention, the present invention can be applied to the current epidemic situation. Through this deep learning method, current hot epidemic issues can be assisted and referenced to improve prevention and control methods.

[0148] Specifically, the news example sentence: "The new coronavirus spreads easily and quickly. It multiplies in the lungs of the human body and invades the lung cells, causing the necrosis and hardening of the lung cells, causing the human body to have difficulty breathing and severe suffocation to death." This news example sentence is labeled and the corresponding time relationship is obtained as follows Fig.10 shown.

[0149] After data preprocessing, the data is input into the TC-GAT model provided by the present invention, and the result is shown in FIG11. Fig.11 The experimental results shown in the figure show that "multiplies" and "invade" are causes, and "necrosis", "hardening", "body" and "severe" are results, indicating that the new coronavirus multiplies and invades lung cells, causing necrosis and hardening of lung cells. The prediction results are generally accurate.

[0150] In the present invention, the Chinese expressions corresponding to the English words involved are as follows:

[0151] BERT (Bidirectional Encoder Representation from Transformers) is based on the bidirectional encoding representation of Transformer;

[0152] GAT (Graph Attention Network) graph attention network;

[0153] GCN (Graph Convolution Network) graph convolution network;

[0154] SA (SelfAttention) self-attention;

[0155] BiLSTM (Bi-directional Long Short-Term Memory) bidirectional long short-term memory model;

[0156] MLM (Masked Language Model) masked language model.

[0157] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and scope of application. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for extracting text causal relationships based on temporal relationships, characterized in that: The method comprises: Mark the relationship between words in a sentence and generate a relationship matrix; Constructing a TC-GAT model; the TC-GAT model includes a BiLSTM model, a GAT model, a BERT model, and a gating model of an equalization mechanism; Training the TC-GAT model using the words in the sentence and the relationship matrix; Extract the causal relationship of the sentence through the trained TC-GAT model; The TC-GAT model is trained by using the words in the sentence and the relationship matrix, specifically including: Taking the words in the sentence as input, training the BiLSTM model, and outputting a bidirectional vector of the words in the sentence; Taking the bidirectional vectors of the words in the sentence and the relationship matrix as input, training the GAT model, and outputting the graph attention features of the words in the sentence; Taking the words in the sentence as input, training the BERT model, and outputting the states of the words in the sentence; Performing vector concatenation of the state of the word in the sentence and the word embedding of the word in the sentence to obtain the self-attention feature of the word in the sentence; The graph attention features of the words in the sentence and the self-attention features of the words in the sentence are used as input to train the gating model of the balancing mechanism to obtain the causal relationship of the sentence.

2. The method for extracting text causal relationships based on time relationships according to claim 1 is characterized in that: The BiLSTM model is: H BiLSTM =BiLSTM([E1,...,E t ,...,E l ]); Among them, E t ∈IR l×n is the t-th word embedding, l is the length of the sentence, is the bidirectional vector representation of the tth word in the sentence generated by BiLSTM, is the output obtained by the forward lstm, is the output obtained by backward lstm, H BiLSTM is the output of the BiLSTM model.

3. The method for extracting text causal relationships based on time relationships according to claim 1 is characterized in that: The gating model of the balancing mechanism is: Among them, W g is the parameter matrix, b g is the bias term, is the vector concatenation operation, g is the equalization coefficient, h is t ′ is the vector representation of the final output of the equalization mechanism, α∈IR 2l×m is the coefficient weight matrix, is the sum of the state and word embedding of the tth word generated by the top four hidden layers of BERT, Graph attention features for words in a sentence.

4. A system for extracting causal relationships in text based on temporal relationships, characterized in that: The system comprises: a marking module and a construction module; The annotation module is used to annotate the relationship between words in the sentence and generate a relationship matrix; The construction module is used to construct a TC-GAT model, taking the words in the sentence and the corresponding relationship matrix as input, training the TC-GAT model, and outputting the causal relationship of the sentence; The building blocks include a BiLSTM model module, a GAT model module, a BERT model module, and a gating model module of an equalization mechanism; The BiLSTM model module is used to train an input bidirectional long-term memory network model using the word embedding of the word in the sentence as input, and output a bidirectional vector of the word in the sentence; The GAT model module is used to train a graph attention network model using the bidirectional vector of the words in the sentence and the relationship matrix as input, and output the graph attention features of the words in the sentence; The BERT model module includes a BERT pre-trained language representation model unit and a splicing unit; the BERT pre-trained language representation model unit is used to train the BERT pre-trained language representation model with the words in the sentence as input, and the output is the state of the words in the sentence; the splicing unit is used to perform vector splicing on the state of the words in the sentence and the word embedding of the words in the sentence to obtain the self-attention features of the words in the sentence; The gating model module of the balancing mechanism is used to train the gating model of the balancing mechanism with the graph attention features of the words in the sentence and the self-attention features of the words in the sentence as input to obtain the causal relationship of the sentence.

5. The system for extracting text causal relationships based on time relationships according to claim 4 is characterized in that: The BiLSTM model is: H BiLSTM =BiLSTM([E1,...,E t ,...,E l ]); Among them, E t ∈IR l×n is the t-th word embedding, l is the length of the sentence, is the bidirectional vector representation of the tth word in the sentence generated by BiLSTM, is the output obtained by the forward lstm, is the output obtained by backward lstm, H BiLSTM is the output of the BiLSTM model.

6. The system for extracting text causal relationships based on time relations according to claim 4, characterized in that: The gating model of the balancing mechanism is: Among them, W g is the parameter matrix, b g is the bias term, is the vector concatenation operation, g is the equalization coefficient, h is t ′ is the vector representation of the final output of the equalization mechanism, α∈IR 2l×m is the coefficient weight matrix, is the sum of the state and word embedding of the tth word generated by the top four hidden layers of BERT, Graph attention features for words in a sentence.

Images