An event extraction method integrating attention mechanism and convolutional neural network

Abstract

This invention discloses an event extraction method integrating an attention mechanism and a convolutional neural network. The steps include: 1) using a text encoder to represent the features of the text to be extracted, obtaining distributed features of the text; 2) using a feature extractor to extract contextual features and word association information of the text from the distributed features; 3) inputting the contextual features and word association information of the text into an event trigger word classifier, outputting event trigger words of the text, and then determining the event type of the text based on the event trigger words; 4) using an event element classifier to sequentially determine whether each word in the text is an event element based on the event type and contextual features; 5) using an element role classifier to identify the role category of each event element. This invention significantly improves the accuracy and efficiency of event extraction.

Description

Technical Field

[0001] This invention pertains to information extraction technology in the field of natural language processing, and in particular, implements an event extraction method that integrates attention mechanisms and convolutional neural networks. Background Technology

[0002] Information extraction is the process of extracting specific information from natural language text, thereby automatically classifying, extracting, and reconstructing massive amounts of heterogeneous text content. Information extraction mainly includes entity extraction, relation extraction, and event extraction. Among these, event extraction is a task that extracts various entities and relations in a highly structured manner. Event extraction identifies important arguments related to a target event from semi-structured and unstructured text. This involves acquiring event trigger words and related argument information, organizing them into event information, and is widely used in semantic search, intelligence analysis, event reasoning, risk warning, and intelligent question answering. The event extraction task is divided into two parts: event identification and event argument identification. This involves discovering event trigger words and determining the event type, and identifying key event arguments and determining their roles. Because understanding the world and solving problems in terms of events aligns better with human cognitive habits, event extraction has become a hot topic in industry and academia both domestically and internationally. In recent years, most of these research efforts have relied on deep learning to acquire event information. Currently, existing event extraction methods are mainly classified into the following categories based on their neural network architectures:

[0003] (1) Event extraction based on convolutional neural networks: Convolutional neural networks are used to induce k-grams information, capture local semantic features, learn the semantic features of sentence composition, and complete the identification and extraction of event information. This method can handle multi-event scenarios. However, it cannot encode lexical semantics in different contexts. If more global information is to be obtained, it is necessary to stack layers to increase the receptive field.

[0004] (2) Event extraction based on recurrent neural networks: This method uses recurrent neural networks to model sequence information, capture the dependencies between parameter roles and trigger word subtypes, and mine temporal information and long-distance relationships in the text. This method can effectively analyze text with long dependencies, but it is essentially a Markov decision process, which cannot learn global structural information well, ignores the positional information of entity pairs, and cannot be executed in parallel, resulting in a slow speed.

[0005] (3) Event extraction based on attention mechanism learns the dependency relationship between words at different distances, analyzes the weight between words to determine semantic relationship, reuses event pattern information, and simplifies data annotation. This method can capture important semantic information and effectively utilize global information. However, directly comparing sequences pairwise cannot model the position well.

[0006] (4) Based on the combination of the above neural networks, the ability to extract semantic features is improved by stacking two or more neural networks. It can be verified that the fusion of different neural networks is effective in improving the accuracy and efficiency of event extraction.

[0007] However, current event extraction methods still lack full utilization of semantic features in text, neglecting the role of semantic relevance between event types and event arguments in improving event extraction accuracy. Integrating multiple neural networks is an effective feature extraction method; however, how to design suitable neural networks to fully utilize semantic relevance remains to be studied. Summary of the Invention

[0008] The purpose of this invention is to fully utilize the semantic relevance in text and design an event extraction method that integrates multiple neural networks. This method can improve the accuracy of event extraction while ensuring the efficiency of event extraction.

[0009] To achieve the above objectives, the technical solution adopted in this invention is: an event extraction method that integrates attention mechanism and convolutional neural network, characterized by: (1) effectively obtaining combined semantic feature vectors in literary sentences through convolutional neural network, and performing max pooling operation on the combined semantic feature vectors to obtain sentence-level local features and semantic structure; (2) solving the long-distance dependency problem by calculating the mutual influence between words in the same sentence through the integration of attention mechanism, and integrating different knowledge generated from the same attention convergence, the differences of these knowledge come from different subspace representations of the same query, key and value, which can effectively reduce the influence of text noise on the extraction effect; (3) mapping the word vectors generated by loading pre-trained model (such as BERT) to the corresponding word vector representation of each input word, that is, by fine-tuning and dynamically generating feature vectors that integrate context information, the pure text is represented as distributed feature information, so that the hidden state of each word contains the influence of words in different positions in the sentence, while avoiding the sparsity of parameter space.

[0010] An event extraction method integrating attention mechanisms and convolutional neural networks, comprising the following steps:

[0011] 1) Use a text encoder to represent the features of the text to be extracted, and obtain the distributed features of the text to be extracted;

[0012] 2) Use a feature extractor to extract the contextual features and word association information of the text to be extracted from the distributed features;

[0013] 3) Input the contextual features and word association information of the text to be extracted into the event trigger word classifier, output the event trigger words of the text to be extracted, and then determine the event type of the text to be extracted based on the event trigger words of the text to be extracted;

[0014] 4) The event element classifier determines whether each word in the text to be extracted is an event element based on the event type and contextual features of the text to be extracted.

[0015] 5) Use the element role classifier to identify the role category of each event element.

[0016] Furthermore, the method for obtaining the distributed features of the text to be extracted is as follows: the text encoder generates a text sequence S = [x1, x2, x3, ..., x...] based on the text to be extracted. n ], where n is the number of word segments in the text to be extracted, x i Let be the i-th word in the text to be extracted; then, for each word in the text sequence S, generate a word embedding vector, a segmentation vector, and a position vector respectively. Convert the text sequence S into an input sequence T = (t1, t2, t3, ..., t...) by summing the word embedding vector, segmentation vector, and position vector. n ), t n This represents the fused vector obtained by combining the word embedding vector, segmentation vector, and position vector of the nth word segment; the sequence T = (t1, t2, t3, ..., t... n The input to the Transformer layer uses a self-attention function to obtain inter-word associations and assign weights, resulting in a feature vector that incorporates contextual information. This feature vector is then input into a pre-trained model to obtain sequence E. m ={e1,e2,e3,...,e n}, that is, the distributed features of the text to be extracted; e n This is the word vector representation corresponding to the nth word segment.

[0017] Furthermore, the feature extractor includes a bidirectional long short-term memory network, a convolutional layer, and an attention unit. The method for extracting the contextual features and word associations of the text to be extracted is as follows: First, the distributed features are input into the bidirectional long short-term memory network to obtain a sequence feature vector, which is then input into the convolutional layer. The convolutional layer performs convolution calculations on the sequence feature vector to obtain the local features and high-dimensional semantic structure feature vectors of the text to be extracted. The high-dimensional semantic structure feature vectors are then input into the attention unit to obtain the sum of the features of each word x in the text to be extracted. i The relevance feature vector r to the target entity i , where i∈[1,n].

[0018] Furthermore, the event-triggered word classifier will and r i After concatenation, the data is input into a conditional random field to obtain each word x in the text to be extracted. i Corresponding event types; For word segmentation x i The corresponding semantic structure high-dimensional feature vector.

[0019] Furthermore, the pre-trained model is a BERT model.

[0020] A server is characterized by comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the methods described above.

[0021] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the above-described method.

[0022] This invention extracts events using a text encoder, feature extractor, event trigger word classifier, event element classifier, and element role classifier. The steps include:

[0023] First, a text encoder is used to represent the text content to be extracted, thus representing the plain text as distributed feature information.

[0024] Secondly, a feature extractor is used to successively acquire features at different levels of abstraction. First, a bidirectional long short-term memory network is used to automatically capture text sentence features from the distributed features processed by the text encoder. Then, a convolutional neural network is used to extract local features around each word. By combining the local feature vectors extracted by the convolutional layer, the contextual features and key inter-word information features are further extracted from the text sentence features captured by the bidirectional long short-term memory network. An attention mechanism is used to assign different weights to different vector representations to learn the association information between words from the text sentence features captured by the bidirectional long short-term memory network and reduce the impact of text noise on the extraction effect.

[0025] Then, using the deep features extracted by the convolutional neural network and attention mechanism, the new samples are classified by the event trigger word classifier to determine whether the words are event trigger words, and the event category is classified based on the trigger word information to complete the event type identification.

[0026] Next, based on the characteristics of the event type and the acquired characteristics, the event element classifier is used to judge each entity in the sentence in turn to determine whether the phrase is an element of the event.

[0027] Finally, the element role classifier is used to determine the role category of the acquired event elements, resulting in a structured event that includes the trigger word, the event element, and its category.

[0028] The text encoder performs feature representation on the text to be extracted, which involves two processes: feature representation model construction and model learning. The specific execution steps are as follows:

[0029] 1) Feature representation model construction: Prepare a text sequence S = [x1, x2, x3, ..., xn] containing n words. n ], where x i This represents the i-th character of the input text sequence; the text sequence S is transformed into the input sequence T = (t1, t2, t3, ..., tn) by summing the word embedding vector, segmentation vector, and position vector. n ), where the word embedding vector is obtained by querying the word vector table, the segmentation vector represents the sentence to which the word belongs, and the position vector represents the word's position information, t n This represents the word embedding vector + segmentation vector + position vector of the nth word in S; the sequence T = (t1, t2, t3, ..., tn) is used to represent the word embedding vector, segmentation vector, and position vector of the nth word in S. n The input Transformer layer uses a self-attention function to obtain inter-word associations and assign weights, resulting in a feature vector that incorporates contextual information.

[0030]

[0031] Q represents the latent vector of the decoded word, and K is the latent vector corresponding to different words in the encoder. Q is multiplied by the vector of each word in the encoding region to calculate a value, and the weights are calculated using the softmax function. Intuitively, Q, K, and V represent the same sentence. The relationship matrix between words is obtained through vector multiplication and normalized, then applied to the input sentence itself. That is, single-head attention is obtained by projecting multiple different linear transformations onto Q, K, and V.

[0032]

[0033] Multi-head attention mechanisms combine the results of different self-attention mechanisms to calculate location information in different spatial dimensions.

[0034] MultiHead(Q,K,V)=Concat(Head1,Head2,Head3,..,Head n W 0 ;

[0035] 2) Model Learning: The model is trained on the feature representation model using the literary work "King Gesar" as a training set, dynamically generating feature vectors that incorporate contextual information to obtain sequence E. m =BERT(T), mapped to Em ={e1, e2, e3, ..., e n}, e i = [w1, w2, ..., w m ], E m This is the final distributed characteristic, e i For each input character, there is a word embedding vector, w∈[-1, 1], and m is the specified word vector dimension.

[0036] The feature extractor performs distributed feature extraction in two processes: sentence-level feature capture and high-level semantic feature capture. The specific execution steps are as follows:

[0037] 1) Sentence-level feature capture: A bidirectional long short-term memory network is used to capture sentence vector context information, and distributed features E m Input a bidirectional long short-term memory network, concatenate the forward vectors of the i-th character. and backward output The sequence feature vector of the i-th character is obtained, denoted as . Obtain the sentence-level semantic representation L of the i-th character. i ;

[0038] 2) High-level semantic feature capture: computation The obtained L i Local feature extraction is achieved by combining multiple convolutional kernels in a convolutional layer. k This indicates that a convolution operation is performed, with a kernel of size k; the calculation... Max pooling is performed after processing with different convolutional kernels to obtain a high-dimensional feature vector that fuses the local features and semantic structure of the sentence. The attention mechanism is integrated, and α = softmax(Q*K) is calculated, where Q represents the latent vector of the decoded word, K is the latent vector corresponding to different words in the encoder, Q is calculated as a value by the vector dot product of each word in the encoding region, and the weights are calculated by the softmax function; the output vector is then calculated. Obtain the relevance feature vectors between characters in the sentence and the target entity, r i For the word segmentation x in the text to be extracted i The relevance feature vector to the target entity, where i∈[1,n].

[0039] Event-triggered word classifier utilizes feature extractor and r i The vector concatenation is input into the conditional random field to obtain the event type, which is divided into two parts: building event type dependencies and inferring event types. The specific execution steps are as follows:

[0040] 1) Constructing event type dependencies: Modeling dependencies between labels using conditional random fields. For a given statement S = {x1, x2, ..., x...} n} and its corresponding sequence label y = {y1, y2, ..., y n Given y, calculate the conditional probability p(y\m) for all possible label sequences; calculate... Where β(S) represents the possible event type label sequence in S; calculate f is a mapping function that maps feature vectors to event type labels; calculate W y It is the prediction weight matrix. These are transition weights; the loss function is calculated as L = -∑ x logp(y|S);

[0041] 2) Inferring event type: Using the Viterbi algorithm to decode the input model τ=(A,B,π) and observations O={o1,o2,..,o T}, initialize δ1(i)=π i b i (o1), i∈[1,N]; recursively δ t (i)=max 1≤j≤ N[δ t-1 (j)a ji ]b i (o t )and Until P * =max 1≤i≤N δ r (i) and i * r =argmax 1≤i≤N [δ r (i)];Backtracking the optimal path Find the optimal path The analysis results are obtained, and the corresponding event type of the trigger word is output.

[0042] The event element classifier consists of three parts: feature concatenation, constructing event argument dependencies, and inferring event argument types. The specific execution steps are as follows:

[0043] 1) Feature concatenation: This involves combining each word x from the text to be extracted. i The corresponding event type is mapped to the event vector V. i spliced ​​to r i Then, the feature vector R of the fused event type information is obtained. i ={V i ,r i};

[0044] 2) Constructing event argument dependencies: Modeling dependencies between labels using conditional random fields. For a given statement S = {x1, x2, ..., x...} n} and its corresponding sequence label g={g1,g2,..,g n Given y, calculate the conditional probability p(g\S) for all possible label sequences; calculate... Where β(S) represents the possible sequence of event argument labels in S; calculate f is a mapping function that maps feature vectors to event argument labels; calculate W g It is the prediction weight matrix. These are transition weights; the loss function is calculated as L = -∑ s log(g|S);

[0045] 3) Inferring event argument types: Using the Viterbi algorithm to decode the input model τ=(C,D,π) and observations O={o1,o2,..,o T}, initialize δ1(i)=π i b i (o1), i∈[1,N]; recursively δ t (i)=max 1≤j≤N [δ t-1 (j)a ji ]b i (o t )and Until P * =max 1≤i≤N δ r (i) and i * r =argmax 1≤i≤N [δ r (i)];Backtracking the optimal path Find the optimal path The analysis results are obtained, and the arguments and their roles are output.

[0046] Compared with the prior art, the positive effects of the present invention are as follows:

[0047] This invention combines convolutional neural networks, attention mechanisms, and pre-trained models through the above-mentioned event extraction method that integrates multiple neural networks. This can effectively improve the training speed of the model while ensuring the training effect. On the other hand, this method pays attention to the importance of the semantic relationship between event type and event arguments for event extraction and focuses on utilizing this relationship, so that the model has significantly improved accuracy and efficiency compared with other similar models. Attached Figure Description

[0048] Figure 1 This is an overall flowchart of the method of the present invention.

[0049] Figure 2 This is an algorithm flowchart of an event extraction method that integrates attention mechanism and convolutional neural network as described in this invention. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0051] like Figure 1 The diagram shown is a system model of an event extraction method that integrates an attention mechanism and a convolutional neural network, as described in this invention. The method mainly consists of the following five steps to complete event extraction:

[0052] Step 101: Prepare the event text dataset "King Gesar" for training;

[0053] Step 102: Use a text encoder to construct a text feature representation and obtain a distributed feature representation of the text S to be extracted;

[0054] Step 103: Use a feature extractor to capture sentence-level semantic representation, local features and high-dimensional feature vectors of semantic structure, and correlation feature vectors between words and target entities in the distributed feature representation.

[0055] Step 104: Use the event trigger word classifier to detect the trigger words representing the event from the feature representation obtained in step 103, and obtain the event type corresponding to the text through the event trigger words;

[0056] Step 105: Use the event element classifier and the element role classifier to merge the event type obtained in step 104 with the feature representation obtained in step 103. Use the merged feature vector to judge each entity in the sentence in turn, and judge whether the phrase is an element of the event.

[0057] Furthermore, in step 101, the event text dataset was collected from the literary work *The Epic of King Gesar*, and the dataset was divided into training and testing sets in a 9:1 ratio. The training set was input into the text encoder to train event extraction. The event extraction model adopted supervised learning, and the event corpus was labeled using a two-pointer annotation method. The two-pointer annotation method effectively solves the entity nesting problem by assigning each character a corresponding start and end position to each label. The labels cover entities, event trigger words, event elements, and their corresponding roles. Each corpus contains both single-sentence and multi-sentence forms, and has consistency in event theme, considering only the main events. There are a total of 22 event types in the experimental corpus.

[0058] Furthermore, in step 102, the pre-trained model used in the text encoder adopts the bert-base-chinese version from the huggingface.co website, automatically learning the vector representation of text in space for this dataset, obtaining a refined semantic feature metric to improve the accuracy of subsequent feature extraction. In step 102, the Adam method is used to learn the model parameters, including:

[0059] Step 201: Select the training dataset M for model learning, initialize the algorithm input: event type set C and relation set L, and set the model learning parameters: maximum number of iterations epochs, learning rate λ, maximum input sequence length b, and batch size k (in this example, epochs is 100, λ is 5e-5, b is 128, and k is 32).

[0060] Step 202: Train the feature representation model constructed by the text encoder in step 201 to learn the model parameters. During the model learning process, the Adam method is used to learn the model parameters.

[0061] Furthermore, in step 103, the feature extractor comprises a stack of three parts: a bidirectional LSTM neural network, a convolutional neural network, and an attention mechanism, including:

[0062] Step 301: Use a bidirectional LSTM neural network to extract features from the learned feature representations, and use the dropout function to prevent overfitting during training.

[0063] Step 302: Use a convolutional neural network to perform secondary feature extraction on the shallow feature representation obtained in step 301. Set m convolutional kernels of different lengths, and set n numbers for each convolutional kernel (in this example, m is 3, the lengths are 2, 3, and 4 respectively, and n is 20).

[0064] Step 303: Use the attention mechanism to calculate the attention of the shallow feature representation obtained in step 301, obtain key information between words and sentences, and use the cosine distance formula to calculate similarity.

[0065] Step 304: Concatenate the feature representations obtained in steps 302 and 303 to obtain a highly abstract feature representation;

[0066] Furthermore, in step 104, the event type recognition module takes the feature representation from step 304 as input, uses the transition matrix of the conditional random field layer to find the relationship between the entities and labels in the text annotation of the training set, extracts event trigger words, and obtains the event type;

[0067] Furthermore, in step 105, the event element classifier maps the event types obtained in step 104 into a one-dimensional array, concatenates it to step 304, introduces event type label information to enhance the descriptive ability, uses the transition matrix of the conditional random field layer to find the relationship between labels, predicts the constraint relationship between labels, and extracts event arguments and their corresponding roles.

[0068] This invention provides an event extraction method that integrates attention mechanisms and convolutional neural networks. By combining convolutional neural networks, attention mechanisms, and pre-trained models, this invention effectively acquires sentence-level event features from literary descriptions, learns inter-word relationships, and dynamically generates feature vectors that incorporate contextual information. While ensuring extraction efficiency, it ultimately fully utilizes the semantic relevance between event types and event arguments in the event extraction process, significantly improving the accuracy of event extraction.

[0069] Of course, the present invention may have other embodiments and is not limited to the embodiments described in the specific embodiments. Any other embodiments proposed by those skilled in the art based on the technical solution of the present invention shall also fall within the protection scope of the appended claims.

Claims

1. An event extraction method fusing attention mechanism and convolutional neural network, comprising the steps of: 1) using a text encoder to perform feature representation on a text content to be extracted, to obtain distributed features of the text content to be extracted; 2) extracting contextual features and association information between words of the text to be extracted from the distributed features using a feature extractor; wherein the feature extractor extracts distributed features into two processes: sentence-level feature capture and high-level semantic feature capture; the method of the sentence-level feature capture is: using a bidirectional long short-term memory network to capture sentence vector context information, capturing distributed features inputting the bidirectional long short-term memory network, concatenating the forward and backward outputs of the i-th character to obtain a sequence feature vector of the i-th character, denoted as , obtaining a sentence-level semantic representation of the i-th character ; the method of the high-level semantic feature capture is: calculating , obtaining , combining multiple convolution kernels through a convolution layer to complete local feature extraction, denoting convolution operation, and the convolution kernel being k; calculating , completing maximum pooling operation after processing by different convolution kernels to obtain a high-dimensional feature vector of the fused sentence ; calculating , denoting a decoding word hidden vector, being a hidden vector corresponding to different words in the encoder, calculating a value through vector dot product between each word in the encoding area and the target entity, and calculating a weight through a function; calculating an output vector , obtaining a relevance feature vector of the word in the sentence and the target entity, being a relevance feature vector of the segmented word in the text to be extracted and the target entity, wherein ; 3) input the contextual features of the text to be extracted and the associated information between the words into an event trigger word classifier, output the event trigger word of the text to be extracted, and then determine the event type of the text to be extracted based on the event trigger word of the text to be extracted; wherein the event trigger word classifier will input the segmented words into a conditional random field after splicing, and obtain the semantic structure high-dimensional feature vector corresponding to each segmented word of the text to be extracted; of the text to be extracted; corresponding to the event type; 4) using an event element classifier to judge whether each word in the text content to be extracted is an event element according to the event type and context features of the text content to be extracted; 5) using an element role classifier to identify the role category of each event element.

2. The method of claim 1, wherein, The method for obtaining the distributed features of the text to be extracted is as follows: the text encoder generates a text sequence based on the text to be extracted. Where n is the number of word segments in the text to be extracted. For the first part of the text to be extracted Each word is segmented; then the text sequence is processed. For each word segment, a word embedding vector, a segmentation vector, and a position vector are generated, thus transforming the text sequence... The input sequence is converted by summing the word embedding vector, segmentation vector, and position vector. , The fused vector obtained by combining the word embedding vector, segmentation vector, and position vector of the nth word segment; the sequence The input to the Transformer layer uses a self-attention function to acquire inter-word associations and assign weights, resulting in a feature vector that incorporates contextual information. This feature vector with fused contextual information is then input into a pre-trained model to obtain the sequence. That is, the distributed features of the text to be extracted; This is the word vector representation corresponding to the nth word segment.

3. The method of claim 2, wherein, The pre-training model is a BERT model.

4. A server, characterized by A computer program is stored in a memory and executed by a processor, and the computer program comprises instructions for executing the steps of any one of claims 1 to 3.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of any one of claims 1 to 3.

Images