A method for event causal relationship identification integrating part-of-speech and causal indicator features

Abstract

The present invention provides a method for identifying event causal relationships by integrating part-of-speech and causal indicator features, belonging to the field of deep learning and natural language processing. The method comprises: converting the causal relationship identification task into a binary classification task at the short text level and using a pre-trained XLNet model to generate a preliminary word embedding vector, and adding part-of-speech features to the word embedding vector; artificially constructing a causal indicator vocabulary, using "0-1" encoding to convert whether a word is a causal indicator into a binary feature, and adding the binary feature to the word embedding vector that integrates part-of-speech features; incorporating the embedding vector that integrates part-of-speech and causal indicator features into a bidirectional gated recurrent neural network; then, through a multi-head attention mechanism, enhancing the weight coefficient of the causal indicator in the entire text, so that the model pays more attention to semantic information related to causal relationships; and finally, inputting the output embedding vector into a Softmax classifier to perform a binary classification task, thereby realizing causal relationship identification.

Description

Technical Field

[0001] The present invention relates to technologies related to deep learning and natural language processing, and in particular to an event causal relationship recognition method that integrates part-of-speech and causal indicator features and can be used to identify whether a short text contains a causal relationship. Background Art

[0002] As a subtask of information extraction, event causal relationship extraction can accurately reflect the mutual influence between events, helping people deepen their logical understanding and grasp of large texts. Accurately grasping the causal relationship between events can also provide strong support for upper-level application tasks such as paragraph comprehension, event prediction, and intelligent question and answer. The event causal relationship extraction task can be further divided into two subtasks: event causal relationship identification and event causal relationship detection. Compared with causal relationship extraction, these two subtasks focus more on studying causal relationships at the short text level. The main goal of causal relationship identification is to identify whether a short text contains a causal relationship; causal relationship detection is to detect which part represents the cause and which part represents the result when it is known that a causal relationship exists in a short text. The two complement each other and are interrelated, and both are research hotspots in the field of natural language processing.

[0003] The present invention focuses on the task of causal relationship identification. Currently, most researchers in the industry convert causal relationship identification tasks into text classification tasks when studying them. According to different research methods, they can be roughly divided into two categories: methods based on the combination of statistics and machine learning algorithms and methods based on deep learning. Methods based on the combination of statistics and machine learning algorithms usually require a large amount of manual annotation, and are only applicable to specific fields, and their portability for downstream tasks is also weak. In contrast, methods based on deep learning have achieved great success in text classification tasks in recent years. The most typical deep learning models include convolutional neural networks, recurrent neural networks, and their variants, such as long short-term memory and gated recurrent units. Later, unsupervised pre-trained language models such as BERT were introduced, which returns an embedding with context for each token, greatly improving the performance of many natural language processing tasks. However, when using deep learning methods to implement causal relationship identification tasks, researchers mostly ignore the part-of-speech features in the text and the semantic information of causal indicators.

[0004] For example, in the sentence "The Leave win led to an 11 percent drop in GBP / USD overnight.", "led to" is both the predicate verb and a causal indicator. Such significant words often play a decisive role in determining whether a sentence contains a causal relationship. Therefore, studying the part-of-speech features and causal indicator features in text is essential. However, simply incorporating these features into feature representation is insufficient. Traditional deep learning methods also fail to effectively extract the features of important words in text. Summary of the Invention

[0005] To address the shortcomings of current technologies, the present invention provides a method for identifying event causal relationships that integrates part-of-speech and causal indicator features. The present invention first transforms causal relationship identification into a short-text-level text classification task, namely, a binary classification task to determine whether a causal relationship exists within a given short text. By studying a large number of short-text datasets, it was found that sentences with causal relationships often contain causal indicators such as "lead to," "so," and "therefore." Furthermore, these causal indicators often appear as predicate verbs, conjunctions, or adverbs within the sentence. This information is crucial for identifying whether a sentence contains a causal relationship, so the present invention incorporates this information as a feature into the model. Furthermore, traditional BERT and its related variants often fail to effectively extract this effective information. To address this, the present invention uses a bidirectional gated recurrent neural network to enhance contextual semantic information and incorporates a multi-head attention mechanism. This allows the model to focus on words that play a key role in semantic information, further improving metrics such as precision and recall. The part-of-speech features in this invention are encoded using one-hot encoding, while the causal indicator features are encoded using "0-1" encoding. Initial word embedding vectors are trained using the XLNet pre-trained model.

[0006] A method for identifying event causal relationships that integrates part-of-speech and causal indicator features, the method comprising: using an XLNet model as a base model to convert a text sequence into a preliminary word embedding vector, using one-hot encoding to encode the part-of-speech of all words in the text sequence to generate part-of-speech features, concatenating and fusing the initial word embedding vector with the part-of-speech features to improve the feature representation of each word; manually constructing a causal indicator vocabulary that contains all words or phrases that may represent a causal relationship, using "0-1" encoding to encode whether all words in the text sequence are causal indicators to generate causal indicator features, and adding this binary feature to the word embedding vector that integrates the part-of-speech features to enhance causal semantic features; inputting the word embedding vector that integrates the part-of-speech and causal indicator features, output from an upper layer, into a bidirectional gated recurrent neural network to capture positive and negative contextual semantic feature information; then adding a multi-head attention mechanism to increase the weight coefficient of causal indicators in the text, so that the model can pay more attention to information related to causal relationships; finally, incorporating the generated vector into a softmax layer for result conversion, presenting the classification result in the form of probability, and obtaining a recognition result based on the final probability.

[0007] Compared with the prior art, the present invention has the following beneficial effects:

[0008] 1. The present invention incorporates part-of-speech and causal indicator features into the event causal relationship recognition task, enhancing the grammatical structure representation and causal semantic representation of causal sentences in the event causal relationship recognition task.

[0009] 2. A bidirectional gated recurrent neural network is used to learn the deep semantic information of the text and capture the positive and negative contextual relationships for the word embedding vector that combines part-of-speech features and causal indicator features.

[0010] 3. A multi-head attention mechanism is added to the model, which enables the model to pay more attention to the features of key words when analyzing semantic information in the text, and weaken the features of words that are not closely related to the sentence semantics, thereby more effectively extracting the features of causal indicator words and making the model pay more attention to information related to causality. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a diagram of the overall model architecture of the event causal relationship identification method of the present invention;

[0012] Figure 2 This is a case diagram of the fusion vector feature representation of the event causal relationship identification method of the present invention. Specific implementation methods

[0013] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to examples and accompanying drawings.

[0014] Figure 1 The overall model architecture diagram of the event causal relationship identification method of the present invention that integrates part-of-speech and causal indicator features is shown. The present invention includes the following steps:

[0015] Step 1: Use the word segmenter of the XLNet model to standardize the causal event dataset according to the input format. For a text sequence T of length s in the dataset, s ], add [CLS] and [SEP] at the beginning and end of the text sequence T respectively, and set the maximum length of the sequence to n. For the sequence whose length does not meet n, the remaining part is padded with [PAD]. Then input it into the XLNet model, and use the encoder of the XLNet model to obtain the vector representation of all words in the sequence, and finally generate a preliminary word embedding vector.

[0016] Step 2: Represent the parts of speech of CLS, SEP, and PAD in the sequence with a code p0. Then use the StanfordNLP toolkit to obtain the parts of speech of all words in the text sequence T. Perform one-hot encoding on all parts of speech. Generate the corresponding part-of-speech features for the standardized text sequence T as shown in the following formula.

[0017] POS_Features=(p0,p1,...,p s ,p0,...,p0)

[0018] Among them, the length of POS_Features is n; p0 represents the part-of-speech encoding of CLS, SEP and PAD; p1~p s Indicates t1~t in the initial text sequence T s The corresponding part-of-speech code; p i ∈R p , p represents the dimension of part-of-speech encoding.

[0019] At the same time, we manually build a causal indicator graph library, add all words and phrases that may represent causal relationships, use "0-1" encoding to perform binary encoding on all words in the text sequence to determine whether they are causal indicators, and generate causal indicator features. The initial word embedding vector generated by XLNet is concatenated with the part-of-speech features and causal indicator features to obtain a new embedding feature W = [w1, w2, ..., w n ]. Figure 2 A diagram showing a case study of the new embedding feature representation.

[0020] Step 3: Use a bidirectional gated recurrent neural network to extract bidirectional semantic features. The gated recurrent neural network improves on the long short-term memory network by combining the forget gate and input gate into a single update gate and merging the cell state and hidden state, making the model simpler to calculate than the standard long short-term memory network. The calculation formula of the gated recurrent neural network is as follows:

[0021] z t =σ(W z [h t-1 ,w t ]+b z )

[0022] r t =σ(W r [h t-1 ,w t ]+b r )

[0023]

[0024]

[0025] Where σ represents the Sigmoid activation function, w t Represents the input text vector of this unit, h t-1 Indicates the status information of the previous moment, z t represents the update gate output, r t Represents the reset gate output, Indicates the amount of information temporarily stored in this unit, h t Indicates the amount of information output by this unit, W z 、W r and W h is the weight matrix, b z 、b r and b h is the offset vector.

[0026] Using a bidirectional gated recurrent neural network for bidirectional semantic feature extraction ensures that the model can fully capture the contextual information in the text, thereby improving the processing power and performance of complex tasks. First, the text sequence is input into the forward gated recurrent neural network. During this process, the forward gated recurrent neural network processes from the first word of the sequence to the last word, generating a hidden state at each time step. These hidden states capture the contextual information of the current word and all previous words. At the same time, the same text sequence is input into the backward gated recurrent neural network. The backward gated recurrent neural network processes from the last word of the sequence forward and also generates a hidden state sequence with the same length as the input sequence. These hidden states contain the contextual information of the current word and all subsequent words. These two hidden states are then concatenated and fused to ultimately obtain a bidirectional semantic feature vector sequence, where each vector integrates the previous and next contextual information of the word. The calculation formula of the bidirectional gated recurrent neural network is as follows:

[0027]

[0028]

[0029]

[0030] H=[h1,h2,...,h n ]

[0031] in, They represent the hidden states of the forward GRU and the reverse GRU at time t respectively. The calculation formula of GRU is consistent with the calculation formula of the aforementioned gated recurrent neural network. GRU represents the parameters of the gated recurrent neural network model, h t represents the text feature vector output by the bidirectional gated recurrent neural network at time t, n represents the sequence length, and H represents the output of the bidirectional gated recurrent neural network.

[0032] Step 4: Add a multi-head attention mechanism to enable the model to automatically focus on words that have a decisive influence on causal relationship identification and capture the most important semantic information in the sentence. The calculation formula for single-head attention is as follows:

[0033]

[0034] Among them, Q, K, and V represent query vector, key vector, and value vector respectively; d k Indicates the dimension of the input vector.

[0035] Since a single-head attention unit has limited ability to learn input features, this paper uses a multi-head attention mechanism to learn different attention representations. In this method, the model uses h different parallel attention heads to obtain different attention representations of the input sequence. The calculation formula for multi-head attention is as follows:

[0036] MultiHead(H,H,H)=Concat(head1,head2,…,head h )W O

[0037]

[0038] in, Represents the parameter matrix of the i-th linear projection of Q, K, and V, head i represents the attention representation vector of the i-th head.

[0039] Step 5: Pass the upper layer output value to the Softmax layer for result calculation. The function of the Softmax classification layer is to obtain the predicted label y of sentence T with respect to the category set Y = {0, 1} based on the calculated probability. The calculation formula is as follows:

[0040] p(y|T)=softmax(W (T) Z+b (T) )

[0041] Among them, Z is the high-dimensional vector output by the upper layer, y∈Y is the predicted category, and b is the bias vector.

[0042] At the same time, this model uses the cross entropy function to predict the authenticity of the model classification. It is judged by the size of the cross entropy loss value. The gap between the predicted distribution and the true distribution becomes larger as the cross entropy loss value increases. For a batch of input texts of size N, the formula of the cross entropy loss function is:

[0043]

[0044] Finally, the causal relationship classification result is obtained based on the calculated probability.

[0045] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for identifying event causal relationships by integrating part-of-speech and causal indicator features, characterized in that: include: Generate initial word embedding vectors and integrate part-of-speech features. Use the XLNet model as the base model to convert text sequences into preliminary word embedding vectors. Use one-hot encoding to encode the parts of speech of all words in the text sequence to generate part-of-speech features. Concatenate and fuse the initial word embedding vectors with the part-of-speech features to improve the feature representation of each word. Construct a causal indicator vocabulary and integrate causal indicator word features. A causal indicator vocabulary is manually constructed, containing all words or phrases that may indicate causal relationships. A 0-1 encoding is used to encode whether all words in a text sequence are causal indicators, generating causal indicator word features. This binary feature is then added to a word embedding vector that incorporates part-of-speech features to enhance causal semantic features. To enhance the semantic information of the context, the word embedding vector that combines the features of part of speech and causal indicators is input into a bidirectional gated recurrent neural network to capture the positive and negative contextual semantic feature information; The multi-head attention mechanism is integrated to increase the weight coefficient of causal indicators and capture long-distance dependencies, allowing the model to pay more attention to semantic information related to causality. Event causal relationship recognition inputs the high-dimensional vector output by the upper layer into the Softmax layer for result conversion, ultimately achieving the task of identifying causal relationships.

2. The method for identifying event causal relationships by integrating part-of-speech and causal indicator features as claimed in claim 1, characterized in that: The generating of the initial word embedding vector and fusing the part-of-speech features specifically includes: The word segmenter of the XLNet model is used to standardize the text in the causal event dataset. For a text sequence T of length s in the dataset, s ], after being standardized according to the input format, it is input into the XLNet model, and the encoder of the XLNet model is used to obtain the vector representation of all words in the sequence and generate a preliminary word embedding vector; with the help of the StanfordNLP toolkit, the parts of speech of all words in the sequence are obtained, and the parts of speech are One-hot encoded. The corresponding part-of-speech feature vector is generated for the standardized sequence, and the initial word embedding vector generated by XLNet and the part-of-speech feature vector are spliced ​​and fused to obtain a new embedding feature vector.

3. The method for identifying event causal relationships by integrating part-of-speech and causal indicator features as claimed in claim 1, characterized in that: The construction of the causal indicator word library and integration of causal indicator word features specifically includes: A causal indicator word library is manually constructed, and words that may contain causal semantics are added to the causal indicator word library. All words are coded "0-1" to determine whether they are causal indicators to generate causal indicator word features. Words that are causal indicators are coded as 1, and words that are not causal indicators are coded as 0. This binary feature is then added to the word embedding vector that incorporates part-of-speech features to enhance the causal semantic features in the text.

4. The method for identifying event causal relationships by integrating part-of-speech and causal indicator features as claimed in claim 1, characterized in that: The semantic information of the enhanced context specifically includes: After the word embedding vector that integrates the part-of-speech and causal indicator features, a layer of bidirectional gated recurrent neural network is added to capture the dependencies between positive and negative words and enhance the contextual semantic information.

5. The method for identifying event causal relationships by integrating part-of-speech and causal indicator features as claimed in claim 1, characterized in that: The fusion multi-head attention mechanism specifically includes: A multi-head attention mechanism is added after the bidirectional gated recurrent neural network layer. By generating an attention vector, the feature vector of the entire sentence sequence is assigned a weight. This allows the semantic information in the text sentence to be analyzed with more attention paid to the features of causal indicators, while weakening the features of words that are not closely related to the sentence semantics. This increases the sentence's attention to local key features, thereby improving the effectiveness of causal relationship recognition.

6. The method for identifying event causal relationships by integrating part-of-speech and causal indicator features as claimed in claim 1, characterized in that: The event causal relationship identification specifically includes: The embedded features output by the upper layer are connected to a Softmax layer for result conversion, so that the model can learn the weight relationship between different features in the text and output the probability distribution, and finally identify whether the text contains causal relationships based on the probability.

Images