A semantic perception method for event detection

Abstract

The present application relates to a semantic-aware method for event detection. Since high-frequency event triggers tend to dominate the prediction of ED models, and ED models cannot well understand the context information, leading to the inability to identify event types in some cases, the present application can pay more attention to the context to change the dominance of event triggers. The novel semantic representation is proposed to deeply mine the potential relationship between event triggers and context, making the ED model better understand the context information. Only a light training on the validation set is needed to carry on any trained model without expensive retraining and fine-tuning. The effectiveness and portability of the method are shown by a large number of experimental results on the public ACE2005 dataset. The present application is superior to the previous 12 most advanced (SOTA) event detection models, and the F1 value and F 0.5 value are improved by 5.1% and 2.2% respectively.

Description

Technical Field

[0001] This invention relates to a semantic awareness method for event detection, belonging to the field of natural language processing. Background Technology

[0002] Event detection (ED) is a key step in event extraction (EE) and a crucial task in natural language processing (NLP). It aims to identify event trigger words and categorize them into predefined event types. ED tasks help computers understand human language and are used in many applications such as automatic content extraction, question answering, and knowledge base populations.

[0003] Previous work typically treated event trigger word (ED) as a trigger word-based classification problem. They first used an event trigger word recognizer to find event trigger words in a given sentence, and then used an event trigger word classifier to determine the event type of the trigger word. However, these ED models fall into trigger word bias, merely learning lexical patterns (i.e., word-trigger mappings), and therefore cannot adequately understand the semantics of a given sentence. Therefore, other works have introduced different contextual semantic information to understand the content of a given text, thereby improving ED performance. Some studies have introduced structured information to understand text content, such as using event parameters to represent contextual information. Other studies have introduced unstructured information as contextual semantic information through graph convolutional neural networks, attention mechanisms, or language models.

[0004] In fact, previous ED models suffered from the following drawbacks. First, the ACE2005 dataset exhibited an imbalanced data distribution, with an unbalanced proportion of event trigger words for each event type. For already trained models, high-frequency event trigger words often dominated event type identification; for example, trigger words like "war," "attack," "battle," "shooting," and "bomb" covered approximately 78% of attack event types. Once a given sentence contained such dominant event trigger words, it was classified into the corresponding event type. This phenomenon essentially stemmed from previous ED models' over-reliance on event trigger words and their lexical pattern problems. Second, when event trigger words were ambiguous, ED models could only extract useful information from the context. However, in previous methods, ED models often failed to correctly identify the event type of sentences with ambiguous event trigger words using their contextual information. Essentially, current ED models did not truly learn contextual information. To address these issues, there are generally two solutions: first, labeling a large amount of data to construct a new balanced dataset; second, improving the ability to learn contextual information. However, labeling data is too costly, and existing research requires designing complex model-level strategies to learn contextual information, a lengthy process.

[0005] To address this, a semantic-aware method for event detection is proposed. It requires neither labeled data nor complex model-level strategies; lightweight training on a validation set is sufficient to allow it to focus more on context, shifting the dominance of event trigger words and deeply exploring the potential connections between event trigger words and their context, leading to a better understanding of contextual information. Furthermore, the proposed plugin is highly portable, allowing it to be integrated onto any pre-trained event detection model without requiring extensive model parameter updates and fine-tuning. Summary of the Invention

[0006] This invention provides a semantic awareness method for event detection. Through lightweight training, this invention can alleviate the shortcomings of traditional event detection methods, such as the influence of dominant trigger words and the inability to understand contextual information well, without the need for complex model-level strategies, thereby improving the performance of event detection.

[0007] The technical solution of the present invention is: a semantic perception method for event detection, the method comprising processing a public dataset; inputting the processed data into a baseline model to obtain trigger words and predicted probability distributions; inputting the processed data, the obtained trigger words, and the predicted probabilities into a plugin to intervene in the prediction of the baseline model, making the result more semantically consistent; and finally using a similarity metric to determine whether the final result uses the result of the baseline model or the prediction result of the plugin.

[0008] As a further aspect of the present invention, the specific steps of the method are as follows:

[0009] Step 1: Prepare the dataset. Preprocess the ACE2005 public dataset and prepare the processed data to be input into the model.

[0010] Step 2: Input the processed data into the baseline model (DMBERT) to obtain the probability distribution of trigger words and predictions;

[0011] Step 3: Input the processed data, the obtained trigger words, and their probabilities into the semantic awareness plugin (SAP). The plugin includes a semantic representation module and a semantic awareness interaction layer.

[0012] Step 4: Use similarity metrics to determine the final prediction result.

[0013] As a further aspect of the present invention, the specific steps of Step 1 are as follows:

[0014] Step 1.1: Download the publicly available dataset ACE2005. This dataset contains 599 documents, with 529 documents for the training set, 30 for the validation set, and 40 for the test set, and defines 34 event types. Then, insert delimiter symbols (i.e., [CLS], [SEP]) into the text, using the symbol S. w This indicates that the processed data is then ready to be input into the model.

[0015] As a further aspect of the present invention, the specific steps of Step 2 are as follows:

[0016] Step 2.1: Input the processed data into the baseline model (DMBERT). First, it undergoes BERT encoding, as shown below:

[0017] {h1, ..., h t , ..., h m}=BERT{S w}

[0018] Where {h1, ..., h} t , ..., h m} represents the hidden state encoded by BERT, h t This indicates the hidden state of the trigger word t.

[0019] Step 2.2: Given a sequence with n trigger words, the context is divided into n+1 parts. A max-pooling layer is used to process each part to obtain context features, as shown below:

[0020] c = [c1; ...; c n+1 ]

[0021]

[0022]

[0023] Where c is the context feature, [;] denotes the concatenation operation, max(·) denotes the max pooling operation, and j n , (n=1,2,...,n) represents the position of the trigger word t.

[0024] Step 2.3: Obtain the probability distribution of the context feature c through linear transformation and layer normalization, and define the loss function as follows:

[0025]

[0026]

[0027] Where f(·) represents the linear transformation and layer normalization operation. p represents the probability distribution of the prediction. i，y Distribution of real labels It is the defined loss function.

[0028] As a further aspect of the present invention, the specific steps of Step 4 are as follows:

[0029] Step 3.1: Given sequence S w In the example, replacing the position corresponding to the trigger word 't' with the placeholder [MASK], we can obtain:

[0030] S mark =([CLS],w1,...,[MASK],...,w m [SEP])

[0031] Where S mark This represents the text after the mask;

[0032] Step 3.2: Masked Text S mark Using a masked language model, we can predict the words that might appear at the masked positions, as shown below:

[0033] H mask =BERT(S mark )

[0034] p mask =softmax(f(H) mask ))

[0035] Where H mask It is the masked text S mark The hidden state, f(·) represents the linear transformation and layer normalization operation, p mask Indicates the probability of predicting a word;

[0036] Step 3.3: Select Top-K1 candidate trigger words based on the probability of the predicted words. The semantic features and trigger word features are obtained by encoding the Top-K1 candidate trigger words and the trigger word t obtained by the baseline model (DMBERT), as shown below:

[0037] K i =LargestK(p mask K1)

[0038]

[0039] Where LargestK(p mask K1) returns the probability p corresponding to the predicted word. maskThe K1 largest candidate words are selected from the pool, and g(·) represents the glove embedding that transforms each candidate trigger word into a 50-dimensional vector. Represents semantic features, Indicates trigger word features;

[0040] Step 3.4: Semantic features and trigger word features After concatenation, the semantic representation is obtained, as follows:

[0041]

[0042] in [;] represents semantic representation, and [;] represents concatenation operation;

[0043] Step 3.5: Sort the predicted probabilities of the baseline model for each event type, and finally select the Top-K2 event types. and its prediction probability Then, through the semantically aware interaction layer (SAInt), semantic information is incorporated into the baseline model (DMBERT) to influence the prediction results of the baseline model. Finally, the prediction results of the plugin can be obtained, as shown below:

[0044]

[0045]

[0046] in These are learnable parameters. The Semantic Aware Interaction Layer (SAInt) consists of 5 layers of multilayer perceptrons.

[0047] As a further aspect of the present invention, the specific steps of Step 4 are as follows:

[0048] Step 4.1: Considering that not all predictions from the baseline model (DMBERT) are incorrect, a similarity metric is used to determine whether the final result should be the result of the baseline model or the result of the plugin, as shown below:

[0049]

[0050]

[0051] Where E old This is the prediction result of the baseline model, E SAInt It is the prediction result of the semantic awareness plugin, and δ∈(0,1) represents the similarity threshold for deciding which result to use.

[0052] The beneficial effects of this invention are:

[0053] 1. A semantically aware method for event detection is proposed, which can pay more attention to the context to change the dominance of event trigger words without updating a large number of model parameters or retraining the model. Furthermore, this invention is highly portable, requiring only simple training on a validation set to be applied to any event detection paradigm.

[0054] 2. A novel contextual semantic representation is proposed, making the predictions of the ED model more semantically consistent. This invention can mine the potential connections between event trigger words and context, and learn how events are described in text. Furthermore, this invention provides a more suitable semantic representation that eliminates the need to retrain the ED paradigm, thereby improving ED performance.

[0055] 3. The proposed semantic-aware method for event detection, when applied to Dynamic Multi-Pool BERT (DMBERT), effectively outperforms 12 strong baselines on the ACE2005 dataset, improving the F1 score by 5.1%. 0.5 The value increased by 2.2%, achieving state-of-the-art performance. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the semantic awareness method for event detection proposed in this invention;

[0057] Figure 2 These are different semantic representations of the performance of the SAP in this invention;

[0058] Figure 3 It is a case study from the ACE2005 test set. Detailed Implementation

[0059] Example 1: A semantic awareness method for event detection, using the ACE2005 dataset as an example to train the model. The method structure is as follows: Figure 1 As shown, the specific steps are as follows:

[0060] Step 1: Download the publicly available dataset ACE2005. This dataset contains 599 documents, with 529 documents for the training set, 30 for the validation set, and 40 for the test set, and defines 34 event types. Then, insert delimiter symbols (i.e., [CLS], [SEP]) into the text, using the symbol S. w This indicates the process. Finally, the processed data is prepared for input into the model.

[0061] Step 2: Input the processed data into the baseline model (DMBERT) to obtain the probability distribution of trigger words and predictions;

[0062] As a further aspect of the present invention, the specific steps of Step 2 are as follows:

[0063] Step 2.1: Input the processed data into the baseline model (DMBERT). First, it undergoes BERT encoding, as shown below:

[0064] {h1, ..., h t , ..., h m}=BERT{S w}

[0065] Where {h1, ..., h} t , ..., h m} represents the hidden state encoded by BERT, h t This indicates the hidden state of the trigger word t.

[0066] Step 2.2: Given a sequence with n trigger words, the context is divided into n+1 parts. A max-pooling layer is used to process each part to obtain context features, as shown below:

[0067] c = [c1; ...; c n+1 ]

[0068]

[0069]

[0070] Where c is the context feature, [;] denotes the concatenation operation, max(·) denotes the max pooling operation, and j n , (n=1,2,...,n) represents the position of the trigger word t.

[0071] Step 2.3: Obtain the probability distribution of the context feature c through linear transformation and layer normalization, and define the loss function as follows:

[0072]

[0073]

[0074] Where f(·) represents the linear transformation and layer normalization operation. p represents the probability distribution of the prediction. i，y Distribution of real labels It is the defined loss function.

[0075] Step 3: Input the processed data, the obtained trigger words and probabilities into the semantic awareness plugin (SAP); the plugin includes: a semantic representation module and a semantic awareness interaction layer;

[0076] As a further aspect of the present invention, the specific steps of Step 3 are as follows:

[0077] Step 3.1: Given sequence S w In the example, replacing the position corresponding to the trigger word 't' with the placeholder [MASK], we can obtain:

[0078] S mark =([CLS],w1,...,[MASK],...,w m [SEP])

[0079] Where S mark This represents the text after the mask.

[0080] Step 3.2: Masked Text S mark Using a masked language model, we can predict the words that might appear at the masked positions, as shown below:

[0081] H mask =BERT(S mark )

[0082] p mask =softmax(f(H) mask ))

[0083] Where H mask It is the masked text S mark The hidden state, f(·) represents the linear transformation and layer normalization operation, p mask This indicates the probability of predicting a word.

[0084] Step 3.3: Select Top-K1 candidate trigger words based on the probability of the predicted words. The semantic features and trigger word features are obtained by encoding the Top-K1 candidate trigger words and the trigger word t obtained by the baseline model (DMBERT), as shown below:

[0085] K i =LargestK(p mask K1)

[0086]

[0087] Where LargestK(p mask K1) returns the probability p corresponding to the predicted word. mask The K1 largest candidate words are selected from the pool, and g(·) represents the glove embedding that transforms each candidate trigger word into a 50-dimensional vector. Represents semantic features, This indicates the characteristics of the trigger word.

[0088] Step 3.4: Semantic features and trigger word features After concatenation, the semantic representation is obtained, as follows:

[0089]

[0090] in [;] represents semantic representation, and [;] represents concatenation operation.

[0091] Step 3.5: Sort the predicted probabilities of the baseline model for each event type, and finally select the Top-K2 event types. and its prediction probability Then, through the semantically aware interaction layer (SAInt), semantic information is incorporated into the baseline model (DMBERT) to influence the prediction results of the baseline model. Finally, the prediction results of the plugin can be obtained, as shown below:

[0092]

[0093]

[0094] in These are learnable parameters. The Semantic Aware Interaction Layer (SAInt) consists of 5 layers of multilayer perceptrons.

[0095] Step 4: Use similarity metrics to determine the final prediction result.

[0096] As a further aspect of the present invention, the specific steps of Step 4 are as follows:

[0097] Step 4.1: Considering that not all predictions from the baseline model (DMBERT) are incorrect, a similarity metric is used to determine whether the final result should be the result of the baseline model or the result of the plugin, as shown below:

[0098]

[0099]

[0100] Where E old This is the prediction result of the baseline model, E SAInt It is the prediction result of the semantic awareness plugin, and δ∈(0,1) represents the similarity threshold for deciding which result to use.

[0101] To illustrate the effectiveness of this invention, six sets of comparative experiments were set up. The first set was the main experiment, which was compared with previous work. The second set of experiments compared the previous ED model before and after adding the plugin. The third set of experiments compared different semantic features. The fourth set of experiments compared the performance of the semantic features used in this invention and the ED model that needed to be retrained. The fifth set of experiments was an ablation experiment to verify the importance of each component in the plugin. The sixth set of experiments was a case study to classify the performance of this invention using examples.

[0102] (1) Results of the main experiment

[0103] We selected 12 of the most advanced event detection models from the past and compared them on the ACE2005 dataset.

[0104] Table 1: Model performance on the AC2005 dataset

[0105]

[0106] As shown in Table 1, SAP outperforms all baseline models and achieves the state-of-the-art (SOTA) F1 score. Since the proposed SAP uses the event type probabilities predicted by the trained model to influence the final prediction result, the recall index of the model using SAP is higher. For fairness, in addition to the three authoritative evaluation metrics Precision (P), Recall (R), and F1, an F-index is added. 0.5 Indicators are used to comprehensively evaluate our approach. 0.5 The score considers R and P to be equally important, thus increasing the emphasis on P and relatively decreasing the proportion of R. Therefore, F... 0.5 This is more suitable for evaluating SAP performance. The present invention addresses P, R, F1, and F... 0.5 In terms of performance, it surpasses the state-of-the-art (SOTA) models EDGP / SSJDN by 0.4% / 3.9%, 10.3% / 10.5%, 5.1% / 7%, and 2.2% / 5.1% respectively on the ACE2005 dataset. This can be attributed to the introduction of semantic representation and post-processing methods in this invention. It is difficult to increase the weight of contextual semantic information during training; therefore, this invention uses post-processing methods to achieve this efficiently. The semantic representation of this invention focuses on the contextual information of the entire sentence, rather than the contextual information of words near the event trigger word. It learns the potential connections between the event trigger word and the context, which improves the ability to understand contextual information.

[0107] (2) The impact of semantic awareness plugins on different models

[0108] Baseline models with four different strategies were selected, lightly trained on the validation set, and tested on the test set. This demonstrates that the invention can be easily implemented on any trained event detection model with different strategies, and that it can influence the predictions of trained models, making the predictions of trained ED models more semantic and mitigating the shortcomings of previous ED models.

[0109] Table 2 shows the performance of the baseline model before and after using the semantic awareness plugin.

[0110]

[0111] As shown in Table 2, for overall performance, the present invention achieves significant improvements across all four evaluation metrics on the four baseline models. This indicates that the present invention has a positive impact on the prediction results of the previous baseline models and improves their recognition accuracy. 0.5 The values ​​increased by 2.6%, 3.5%, 1.9%, and 4.2% respectively, combining P and R in a reasonable ratio to comprehensively measure the performance of the four baseline models before and after incorporating this invention. Regarding the number of parameters, calculations show that FSED and MOGANED have 109.48M and 2.76M parameters, respectively. Since BERT is the base encoder, SSJDN and DMBERT have 341.48M and 335.17M parameters, respectively. This means that retraining the model would consume significant time and space resources. In contrast, this invention only needs to learn 0.028M parameters, greatly reducing time costs and improving performance.

[0112] Overall, the experimental results directly confirm the effectiveness and portability of the present invention.

[0113] (3) The impact of different semantic features on SAP

[0114] To verify the necessity of using Top-K words as semantic features, three semantic features, V1-V3, were selected to compare their performance differences. V0 represents Top-K words; V1 represents the concatenation of all word representations in the context; V2 is the event parameter; and V3 represents no semantic features.

[0115] like Figure 2 As shown, V1 / V2 / V3 in F 0.5 The values ​​of V0 and V0 are 2.0%, 1.2%, and 2.8% lower, respectively, indicating that the ED model can achieve the best results by using Top-K words as semantic features of this invention.

[0116] (4) Performance of semantic representation on retrained and trained models

[0117] To demonstrate that the semantic representation of this invention is more suitable for plug-in strategies, it is incorporated into DMBERT for retraining. C1 indicates that the semantic representation of this invention is used in SAP; C2 indicates that the semantic representation of this invention is incorporated into DMBERT for retraining.

[0118] Table 3. Performance of semantic representation in SAP and DMBERT

[0119]

[0120] As shown in Table 3, C1 in P, R, F1 and F 0.5 The values ​​above C2 are 1.4%, 7.4%, 4.2%, and 2.4% higher, respectively. This demonstrates that the semantic representation of the present invention is more suitable for plug-in strategies.

[0121] (5) Ablation test

[0122] Feature ablation: In order to explore the necessity of trigger word features and semantic features, they are ablated.

[0123] Decision ablation: To explore the necessity of similarity measurement, it is ablated.

[0124] Table 4 Ablation Experiment

[0125]

[0126] As shown in Table 4, removing either trigger word features or context features reduces the performance of this invention, indicating that both features are necessary for this invention. Removing similarity decisions also reduces the performance of this invention because the predictions of a well-trained model are not entirely wrong; this invention only addresses the cases of dominant event trigger words and poor understanding of the context. Therefore, it is necessary to incorporate similarity decisions.

[0127] (6) Case Analysis

[0128] Four case studies were selected to better illustrate the effectiveness of the invention.

[0129] like Figure 3As shown, event triggers are marked in red in the sentence. Checkmarks and crosses indicate whether the detected event is correct or incorrect. For (a) and (b), the ambiguous event trigger words "release" and "offer" have multiple meanings and may correspond to several different event types. However, baseline models typically classify ambiguous event trigger words into the event types they usually belong to. In (a), the semantic features according to the invention show many clues indicating that the sentence belongs to the "Transaction:Transfer-Money" type. In (b), there are insufficient clues in the semantic features to explain that the sentence is of the "Transaction:Transaction-Ownership" type. After the invention intervenes in the trained model, the sentence is classified as "None". For (c), the semantic features of the invention contain some irrelevant words and cannot indicate that the sentence is of the "Justice:Sue" type. Therefore, although the event trigger word "Bankrupt" is the dominant trigger word of the "Declare-Bankrupt" type, it is more consistent with the context, so the invention selects the "Declare-Bankrupt" type. For (d), “detonated” is a high-frequency event trigger word of the “Conflict:Attack” type. In the semantic features of this invention, any word placed within the sentence is fluent, proving that the contextual information of this sentence is insufficient to support the prediction of the baseline model. Therefore, the prediction of the baseline model is abandoned, and the results of this invention are trusted. These four cases demonstrate that this invention performs well under these conditions, reducing errors caused by dominant event trigger words and a lack of understanding of the context.

[0130] In summary, to alleviate the shortcomings of dominant trigger words and poor understanding of contextual information, a semantic-aware method for event detection is proposed. This method focuses more on context to change the dominance of trigger words and delves deeper into the potential connections between trigger words and the context, resulting in a better understanding of contextual information and more semantically meaningful predictions. Furthermore, it is highly portable, requiring no retraining or fine-tuning; it can be easily integrated into any pre-trained event detection model with only lightweight training on a validation set. Extensive experiments validate the effectiveness and rationality of the proposed model.

[0131] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A semantic awareness method for event detection, characterized in that: The method is as follows: Step 1: Prepare the dataset. Preprocess the ACE2005 public dataset and prepare the processed data to be input into the model. Step 2: Input the processed data into the baseline model DMBERT to obtain the probability distribution of trigger words and predictions; Step 3: Input the processed data, the obtained trigger words and probabilities into the semantic awareness plugin SAP. The plugin includes: a semantic representation module and a semantic awareness interaction layer. Step 4: Use similarity metrics to determine the final prediction result; The specific steps of Step 3 are as follows: Step 3.1: Given sequence In the middle, the trigger word Replace the corresponding position with a placeholder. ,get: ； in This represents the text after the mask; Step 3.2: Masked Text The masked language model is used to predict the words that may appear at the masked positions, as shown below: ； ； in It is masked text The hidden state, This represents linear transformation and layer normalization operations. This indicates the probability of predicting a word. Step 3.3: Select words based on their predicted probabilities. Candidate trigger words and to Candidate trigger words and trigger words obtained by the baseline model DMBERT Encoding yields semantic features and trigger word features, represented as follows: ； ； in Returns the probability corresponding to the predicted word. The largest Candidate words for each element, This means that glove embedding transforms each candidate trigger word into a 50-dimensional vector. Represents semantic features, Indicates trigger word features; Step 3.4: Semantic features and trigger word features After concatenation, the semantic representation is obtained, as follows: ； in Represents semantic representation. Indicates a splicing operation; Step 3.5: Sort the predicted probabilities of the baseline model for each event type, and finally select... Event Type and its prediction probability Then, through the semantic-aware interaction layer SAInt, semantic information is integrated into the baseline model DMBERT to influence the prediction results of the baseline model, ultimately obtaining the plugin's prediction results, as shown below: ； ； in These are learnable parameters. The semantic awareness interaction layer SAInt consists of 5 layers of multilayer perceptrons.

2. The semantic awareness method for event detection according to claim 1, characterized in that: The specific steps of Step 1 are as follows: Step 1.1: Download the publicly available dataset ACE2005. This dataset contains 599 documents, with 529 documents in the training set, 30 in the validation set, and 40 in the test set. It also defines 34 event types. Then, insert delimiter symbols into the text using symbols. This means that the processed data will eventually be input into the model.

3. The semantic awareness method for event detection according to claim 1, characterized in that: The specific steps of Step 2 are as follows: Step 2.1: Input the processed data into the baseline model DMBERT. First, it is encoded by BERT, as shown below: ； in It is the hidden state encoded by BERT. Indicates trigger word The hidden state; Step 2.2: When the given sequence contains Each trigger word, the context is divided into Each part is processed using a max-pooling layer to obtain contextual features, as shown below: ； ； ； in It is a contextual feature. This indicates a splicing operation. This indicates the max pooling layer operation. Indicates trigger word Location; Step 2.3: Contextual Features After linear transformation and layer normalization, the probability distribution is obtained, and the loss function is defined as follows: ； ； in This represents linear transformation and layer normalization operations. Represents the probability distribution of the prediction. Distribution of real labels It is the defined loss function.

4. The semantic awareness method for event detection according to claim 1, characterized in that: The specific steps of Step 4 are as follows: Step 4.1: Considering that the baseline model DMBERT's predictions are not all wrong, a similarity metric is used to determine whether to use the baseline model's result or the plugin's result in the final outcome, as shown below: ； ； in These are the predictions from the baseline model. This is the prediction result from the semantic awareness plugin. This represents the similarity threshold used to determine which result to adopt. It is a semantic feature. It is a trigger word feature.

Images