Event implicit relation extraction method

Through the joint feature extraction and attention mechanism of BiLSTM+GCN+Attention model, the problem that traditional methods are difficult to deal with implicit event relationships is solved, and the accurate extraction and understanding of event relationships is achieved, and the performance of relationship extraction is improved.

CN118939767APending Publication Date: 2024-11-12ZHENGZHOU UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410967273.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Traditional relationship extraction methods are difficult to deal with the situation where implicit event relationships lack obvious hint words and Chinese syntactic variant diversity, resulting in insufficient accuracy and robustness of relationship extraction.

Method used

The BiLSTM+GCN+Attention model is used for implicit relationship extraction, through the joint feature extraction of bidirectional long and short-term memory network and graph convolution network, the key information is automatically identified in combination with the attention mechanism, and the graph representation between event elements is constructed, and the vocabulary-level semantics are captured. and classify information.

Benefits of technology

It improves the accuracy and robustness of event relationship extraction, can effectively understand and extract the complex implicit logical connections between events, and improves the performance of relationship extraction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939767A_ABST
    Figure CN118939767A_ABST
Patent Text Reader

Abstract

The invention discloses an event implicit relation extraction method for causal relation classification and regression identification. The model is combined with a bidirectional long-short memory network and a graph convolutional network to perform fusion feature extraction, and distance information of a center trigger word is integrated into semantic features to enrich extraction of the semantic features. Experiments show that the model provided by the invention has better performance in an implicit relation extraction task. Comprising the following steps: converting a text into a context semantic representation vector by an Embedding layer; the feature extraction layer performs joint feature extraction by using a bidirectional long short-term memory network (BiLSTM) and a graph convolutional network (GCN), the BiLSTM is responsible for capturing context information in a text sequence, on this basis, graph representation between event elements is constructed by using the GCN, and graph structure information is fully utilized to capture vocabulary-level semantic information and understand a complex relationship between vocabularies; the Attenton layer adopts an attention mechanism to further enhance the attention ability of the model to key information, the complex relationship between events is extracted from a text, and the Softmax output layer uses a Softmax model to carry out classification and outputs a result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing and relates to an event implicit relationship extraction method integrating a graph convolutional network and a long short-term memory network. Background Art

[0002] Event relationship extraction is an important task in the field of natural language processing. This task aims to identify and extract the logical relationships between events from text, mainly including but not limited to causal, sequential and other relationship types. By accurately extracting these relationships, it provides support for in-depth understanding of the development context and interaction of events. After identifying the event entities in the text, it is necessary to determine the logical relationship between these event entities. This requires the model to not only perform in-depth semantic analysis of the text, but also be able to understand and infer the implicit logical connection between events.

[0003] In order to improve the accuracy of relation extraction, research works in recent years have increasingly adopted deep learning techniques, especially the latest advances in the field of natural language processing (NLP), such as pre-trained language models (such as BERT, ERNIE, etc.) and graph neural networks. These technologies can effectively capture complex semantic information in text and understand the relationship between events in a wider context, thus achieving significant performance improvements in relation extraction tasks. In addition, some studies have also tried to integrate additional knowledge information, such as using external knowledge bases or using knowledge enhancement methods to enrich event representations and enhance the model's ability to recognize complex event relationships. By introducing external knowledge, this method provides the model with richer and more accurate semantic information, which helps to improve the accuracy and robustness of relation extraction.

[0004] Since implicit relations lack obvious causal relationship prompt words, traditional relation extraction methods are difficult to deal with the situation where implicit event relations lack obvious prompt words and the diversity of Chinese syntactic variants. This model uses the BiLSTM+GCN+Attention model for implicit relation extraction: BiLSTM and GCN are used for joint feature extraction, and the attention mechanism is used to further enhance the model's ability to focus on key information. Among them, BiLSTM is responsible for capturing contextual information in the text sequence, and GCN is used on this basis to build a graph representation between event elements, making full use of graph structure information to capture lexical semantic information, understand the complex relationship between words, and extract the complex relationship between events from the text, laying the foundation for building an accurate event graph. Summary of the invention

[0005] The present invention regards event relationship extraction as a classification task, takes the structured events output by the event extraction model as input, and uses the BiLSTM+GCN+Attention model to extract implicit relationships: a bidirectional long short-term memory network (BiLSTM) and a graph convolutional network (GCN) are used for joint feature extraction, and the attention mechanism is used to further enhance the model's ability to focus on key information. Among them, BiLSTM is responsible for capturing contextual information in the text sequence, and GCN is used on this basis to construct a graph representation between event elements, making full use of graph structure information to capture lexical-level semantic information, understand the complex relationships between words, and extract the complex relationships between events from the text.

[0006] A method for extracting implicit relations of events, comprising:

[0007] Embedding layer, which converts text into contextual semantic representation vectors;

[0008] Feature extraction layer, obtaining feature vectors of BiLSTM and GCN respectively;

[0009] Attention layer, automatically identifies and tags key words;

[0010] Softmax, classify and output the results.

[0011] A method for extracting implicit relations of events comprises the following steps:

[0012] Step 100, use the Embedding layer to vectorize the sentence;

[0013] Step 200, using a bidirectional long short-term memory network (BiLSTM) and a graph convolutional network (GCN) for joint feature extraction;

[0014] Step 300, the attention mechanism further enhances the model's ability to focus on key information;

[0015] Step 400: Use SoftMax to output the event relationship classification result.

[0016] In some embodiments, step 100 includes the following sub-steps:

[0017] Step 110, use the ERNIE pre-trained model to process the data and convert the event<ei,ej> Transformed into a sentence consisting of a sequence S = {x1, x2, ..., xt} containing T words.

[0018] Step 120, through the multi-head self-attention module, obtain the contextual semantic representation T = [t1, t2, t3…, tn] of S

[0019] In some embodiments, step 200 includes the following sub-steps:

[0020] Step 210, the bidirectional BiLSTM memory network includes two networks with opposite propagation directions. The output at time t is determined by the input Tt at time t, the forward propagation and the backward propagation. When t=0, T0 is the upper layer output T.

[0021]

[0022] H B =LSTM(T)

[0023] Step 220, GCN is further used in the feature extraction process. The core operation of this network is convolution, which performs weighted averaging on a specific range of nodes in the graph structure data. Specifically, the features of the neighboring nodes of each node are weighted averaged to obtain the aggregated feature value of the node. In this way, GCN can effectively capture the relationships between nodes in the graph structure and extract richer and more expressive feature information. This step plays a key role in subsequent analysis and improvement of model performance. The feature value of the node is then propagated to the next layer as input. When calculating and updating the orange node, only the information of the two adjacent nodes around the orange is needed. l is the number of convolution layers, w is the weight, and C is the weight. ij is the connection information between node i and node j, N i is the adjacent node, σ is the activation function. The first half of the sum is the feature value of the node in the previous layer, and the second half is the aggregated feature value of the adjacent nodes. When l = 0,

[0024]

[0025] Step 230, input the matrix corresponding to the sample sentence, aggregate the features through the hidden layer, and finally obtain the output result. In the formula, l represents the number of layers, W represents the weight matrix, σ represents the activation function, and the joint features are obtained by fusing GCN and BiLSTM.

[0026] H l+1 =σ(AH l W l )

[0027] H G =GCN(T)

[0028]

[0029] In some embodiments, step 300 includes the following sub-steps:

[0030] Step 310, using the Attention mechanism to automatically identify and learn the key parts of these corpora that have an impact on the classification of evolutionary relations. This process involves the identification and annotation of relevant important words to facilitate the execution of the evolutionary relationship extraction task. The representation form r of the sentence is constructed by calculating the weighted sum of the output vectors.

[0031] M=tanh(H)

[0032] α=softmax(w T M)

[0033] r=Hα T

[0034] Step 320, w is the trained parameter vector, w T is the transpose, r is the vector representation of the sentence, and the dimension is d w . is the output vector produced by the LSTM-GCN layer, T is the sentence length, and d w is the dimensionality of the word vector.

[0035] h * =tanh(r)

[0036] In some implementations, step 400 includes the following sub-steps:

[0037] Step 410: Through iterative optimization, the Attention layer parameters of the Softmax classifier complete the recognition learning of the trigger words in the sentence. This process eventually iterates out the optimal parameters for event evolution relationship extraction, thereby improving the effect of relationship extraction.

[0038] p(y|S)=softmax(W S h * +b S )

[0039] Step 420, the Softmax classifier is constructed by using its hidden state as input h * , predicting candidate event causal relationship pairs from a set of discrete categories y in sentence S <e i , e j >Category Tags To achieve accurate classification of relationships.

[0040] BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is the structure diagram of the event implicit relationship extraction method.

[0042] Figure 2It is an overall flow chart of the method of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0044] like Figure 1 As shown in the figure, the event implicit relationship extraction method includes an Embedding layer, a feature extraction layer, an Attention layer, and a Softmax layer.

[0045] S1: The pre-trained model transforms events into <e i , e j >The converted vocabulary sequence S = {x1, x2, ..., x t} is converted into contextual semantic representation T = [t1, t2, t3…, t n ];

[0046] S2: BiLSTM effectively extracts the temporal feature H by capturing the forward and reverse dependencies of the sequence data B =LSTM(T)

[0047] S3: GCN focuses on processing graph structure data and extracts graph structure features by aggregating the features of nodes and their neighbors.

[0048] S4: The matrix A corresponding to the input sample sentence is propagated through the hidden layer aggregation feature.

[0049] H l+1 =σ(AH l W l );

[0050] S5: GCN can effectively capture the relationship between nodes in the graph structure and extract richer and more expressive feature information. Finally, the GCN module output result H is obtained. G =GCN(T);

[0051] S6: GCN and BiLSTM are fused, and the joint features obtained are shown in the formula

[0052] S7: Attention mechanism automatically identifies and learns the key influential parts, M = tanh (H), α = softmax (w T M), r=Hα T ;

[0053] S8: Get the final representation of the sentence, h * =tanh(r);

[0054] S9: Hidden state h* As input, predictions represent candidate event causal pairs <e i , e j >Category Tags To achieve accurate classification of relationships,

[0055] like Figure 2 As shown, an event extraction method mainly includes the following four steps.

[0056] Step 100, use the Embedding layer to vectorize the sentence;

[0057] Step 200, using a bidirectional long short-term memory network (BiLSTM) and a graph convolutional network (GCN) for joint feature extraction;

[0058] Step 300, the attention mechanism further enhances the model's ability to focus on key information;

[0059] Step 400: Use SoftMax to output the event relationship classification result.

[0060] The model consists of four main layers: the Embedding layer converts text into contextual semantic representation vectors; the feature extraction layer uses a bidirectional long short-term memory network (BiLSTM) and a graph convolutional network (GCN) for joint feature extraction. BiLSTM is responsible for capturing contextual information in text sequences. On this basis, GCN is used to construct a graph representation between event elements, making full use of graph structure information to capture lexical-level semantic information and understand the complex relationship between words; the Attenton layer uses the attention mechanism to further enhance the model's ability to focus on key information and extract the complex relationship between events from the text. The Softmax output layer uses the Softmax model for classification and outputs the results.

[0061] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. All that is within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for extracting implicit relations between events, characterized in that: The method comprises: 1) Embedding layer, which converts text into contextual semantic representation vector; 2) Feature extraction layer, obtaining feature vectors of BiLSTM and GCN respectively; 3) Attention layer, automatically identifies and marks key words; 4) Softmax, classify and output the results.

2. The method for extracting implicit event relations according to claim 1, characterized in that: Using the Embedding layer to vectorize sentences includes the following steps: Use the ERNIE pre-trained model to process the data and convert the event<ei,ej> Transformed into a sentence consisting of a sequence S = {x1, x2, ..., xt} containing T words; Through the multi-head self-attention module, the contextual semantic representation T = [t1, t2, t3…, tn] of S is obtained.

3. A method for extracting implicit event relations according to claim 1 or 2, characterized in that: The bidirectional long short-term memory network (BiLSTM) and graph convolutional network (GCN) are used for joint feature extraction. The steps include: The bidirectional BiLSTM memory network contains two networks with opposite propagation directions, and its calculation formula is: Among them, the output at time t is determined by the input Tt at time t, the forward propagation and the backward propagation, and the output is recorded as When t=0, T0 is the upper layer output T, and the final output of the bidirectional BiLSTM memory network is recorded as: H B =LSTM(T) In the process of feature extraction, GCN is further used. The core operation of this network is convolution, which performs weighted averaging on a specific range of nodes in the graph structure data, and weighted averages the features of each node's neighbor nodes to obtain the aggregate feature value of the node. In this way, GCN can effectively capture the relationship between nodes in the graph structure and extract richer and more expressive feature information. The feature value of the node is then propagated to the next layer as input, and the calculation formula is: Among them, l is the number of convolution layers, w is the weight, C ij is the connection information between node i and node j, N i is the adjacent node, σ is the activation function, the first half of the sum is the eigenvalue of the node in the previous layer, and the second half is the aggregated eigenvalue of the adjacent nodes. When l = 0, The matrix corresponding to the input sample sentence is aggregated through the hidden layer to obtain the output result. The calculation formula is: H l+1 =σ(AH l W l ) Among them, l represents the number of layers, W represents the weight matrix, and σ represents the activation function; The final output of GCN is expressed as: H G =GCN(T) Finally, the feature vectors calculated by GCN and BiLSTM are fused to obtain the joint features. The calculation formula is:

4. A method for extracting implicit event relations according to any one of claims 1 to 3, characterized in that: The attention mechanism further enhances the model's ability to focus on key information. The steps are as follows: The Attention mechanism is used to automatically identify and learn the key parts of these corpora that have an impact on the classification of evolutionary relations. This process involves the identification and annotation of relevant important words to facilitate the execution of the evolutionary relationship extraction task. The representation form r of a sentence is constructed by calculating the weighted sum of the output vectors. The calculation process of the representation form r of a sentence is: M=tanh(H),α=softmax(w T M),r=Hα T ; Among them, w is the trained parameter vector, w T is the transpose, r is the vector representation of the sentence, and the dimension is d w . is the output vector produced by the LSTM-GCN layer, T is the sentence length, and d w is the dimension of the word vector, and the final output is: h * =tanh(r)。 5. A method for extracting implicit event relations according to any one of claims 1 to 4, characterized in that: Use Softmax to output the event relationship classification results. The steps include: Through iterative optimization, the Attention layer parameters of the Softmax classifier complete the recognition and learning of the trigger words in the sentence. This process finally iterates the optimal parameters for event evolution relationship extraction, thereby improving the effect of relationship extraction. The calculation formula for the classification result is: p(yS)=softmax(W S h * +b S ) The Softmax classifier uses its hidden state as input h * , predicting candidate event causal relationship pairs from a set of discrete categories y in sentence S <e i , e j >Category Tags In order to achieve accurate classification of the relationship, the calculation result of the classification label is:

Citation Information

Patent Citations

  • Text entity relationship extraction method based on multi-feature information enhancement

    CN112163425A

  • Natural language text triple extraction method and system based on attention mechanism

    CN113869059A

  • Water level prediction method based on GCN-LSTM neural network model

    CN116029403A

  • Air quality prediction method

    CN116341763A