Event knowledge pre-training enhanced cross-language event causal relationship identification method and device

Abstract

The invention relates to an event knowledge pre-training enhanced cross-language event causal relationship identification method and device. According to the method, a multi-language pre-training language model is combined with multi-language alignment event knowledge in an external event common sense knowledge base, and the multi-language event knowledge is integrated into mPLMs through pre-training task design, so that event causal relationship recognition performance in a low-resource language scene is improved. The method comprises the following steps: firstly, extracting event common knowledge of multi-language alignment from ConceptNet; designing a pre-training method based on a parameter sharing adapter, and integrating event node knowledge and relation knowledge between nodes into mPLMs through an event mask pre-training task and a self-supervision link prediction task; and applying the pre-trained mPLMs to a cross-language event causal relationship extraction task. According to the method, the number of parameters of the model is remarkably reduced, the calculation cost is reduced, and the event causal relationship recognition performance is effectively improved.

Description

Technical Field

[0001] The present invention relates to a method and device for cross-language event causal relationship recognition enhanced by event knowledge pre-training, and belongs to the technical field of natural language processing. Background Art

[0002] Event causality identification (ECI) plays an important role in news report analysis and event management. Previous studies have mainly focused on three aspects: event "perception", event "element analysis" and event "core analysis", helping users to discover events from news data, extract event element information, and identify core events that reveal the main content of news documents. However, since news reports are usually limited in length and most reports only focus on a certain stage or hot part of the development of an event, it is difficult for users to fully grasp the overall picture of the event and its internal logic. In order to solve this problem, it is particularly important to conduct event "association analysis" in a timely and effective manner and explore the causal relationship between events. This can not only help users understand the occurrence and development of news events more comprehensively, but also assist relevant managers in making more accurate judgments and decisions when handling events.

[0003] The present invention relates to a cross-language event causal relationship recognition method enhanced by event knowledge pre-training, which extracts multi-language aligned event knowledge from multi-language semantic knowledge bases such as ConceptNet, designs reasonable pre-training tasks, and integrates these knowledge into multi-language pre-training language models, thereby improving the event causal relationship recognition performance in low-resource language scenarios. At present, event causal relationship recognition methods mainly include rule-based, machine learning-based and deep learning-based methods. Rule-based methods rely on a large amount of expert knowledge and preset rules. Although they can achieve certain effects in specific fields, their field adaptability and generalization ability are poor. Machine learning-based methods usually extract event features through feature engineering, but such methods also have the problem of requiring a large amount of labeled data and poor adaptability. With the development of deep learning technology, event causal relationship recognition methods based on deep learning have gradually become a research hotspot. For example, methods based on graph convolutional neural networks (Graph Convolutional Network, GCN) capture the semantic relationship between events by constructing complex graph structures, effectively improving the performance of event causal relationship recognition. Nevertheless, existing deep learning methods usually ignore the use of external event common sense knowledge, such as the causal relationship between earthquakes and tsunamis. Studies have shown that incorporating event common sense from external knowledge bases (such as ConceptNet) into the model can significantly improve the performance of event causal reasoning. However, in the context of multilingual and low-resource languages, how to efficiently extract knowledge suitable for event causal reasoning from external knowledge bases and effectively incorporate it into pre-trained language models (PLMs) remains a challenge. Multilingual pre-trained language models (such as mBERT and XLM-R) provide the possibility for cross-lingual event causal relationship recognition, but the performance of these models on low-resource languages ​​is still limited by problems such as the multilingual curse and language bias. In addition, these models usually have large parameter sizes and high computational costs. Therefore, how to efficiently utilize multilingual pre-trained language models and avoid training from scratch to adapt to cross-lingual event causal relationship recognition tasks is an important direction of current research. Summary of the invention

[0004] The present invention provides a method and device for cross-language event causal relationship recognition enhanced by event knowledge pre-training, so as to solve the problem of whether valuable multilingual aligned event knowledge can be extracted from an external knowledge base, and whether the multilingual aligned event knowledge can be integrated into mPLMs by designing efficient pre-training tasks to improve the event causal relationship recognition performance in low-resource language scenarios.

[0005] The technical solution of the present invention is: in a first aspect, the present invention provides a method for cross-language event causal relationship recognition enhanced by event knowledge pre-training, the method comprising the following steps:

[0006] By designing reasonable pre-training tasks, multilingual aligned event knowledge is integrated into mPLMs, and then mPLMs enhanced with multilingual aligned event knowledge are directly used for cross-language event causal relationship extraction tasks. The specific steps of the cross-language event causal relationship identification method enhanced by event knowledge pre-training are as follows:

[0007] Step 1: Extract multilingual alignment event common sense knowledge from ConceptNet, including nodes and the relationships between nodes;

[0008] Step 2: Based on Step 1, an Adapter pre-training method based on parameter knowledge sharing is proposed, and the event node knowledge and the relationship knowledge between nodes (multilingual alignment relationship and causal relationship) are respectively integrated into mPLMs through the event mask pre-training task and the self-supervised link prediction task;

[0009] Step 3: Based on Step 2, the pre-trained mPLMs are used for the cross-language event causal relationship extraction task.

[0010] Furthermore, the specific steps of Step 1 are:

[0011] Step 1.1, semantic relationship integration: From the original semantic relationships extracted from ConceptNet, delete the relationships that are less relevant to the event causal relationship recognition task. Merge similar relationships into several streamlined semantic relationships. Example relationships include: merging "causes", "causes desire" and "motivated by goal" into "causes".

[0012] Step 1.2, data conversion: Convert the semantic relationship marker string into natural language text, for example, convert "synonyms" into "is a synonym of". Based on the converted semantic relationship text, build a training dataset, which includes inserting the head node and tail node into the corresponding positions of the semantic relationship text. For example, convert "earthquake(en)→(causes)tsunami(en)" into "earthquake(en)causes tsunami(en)". For multilingual alignment event nodes, use the "isA" relationship for data construction.

[0013] Step 1.3, pre-training data generation: Use the random walk method to traverse ConceptNet, set the random walk length to 30, sample the starting and neighboring nodes from the uniform distribution, and generate pre-training data.

[0014] Furthermore, the specific steps of Step 2 are:

[0015] Step 2.1, Adapter architecture: embed parameter-sharing Adapter modules in each layer of mPLMs, freeze the main parameters of mPLMs, and only fine-tune the parameters of the Adapter module; map the input to r dimensions through the linear projection layer, use the Transformer layer to integrate event knowledge and mPLMs language knowledge, and finally divide the output of the mapping layer into two parts: the input of the residual connection layer and the input of the next Adapter layer;

[0016] Step 2.2, pre-training tasks: including event mask pre-training tasks and self-supervised link prediction tasks, where the event mask pre-training task is to mask the event nodes in the text, generate a disturbed text sequence, and predict the masked event nodes based on the output of the Adapter, and calculate the cross entropy loss; the self-supervised link prediction task is to mask certain relationships in the text, train the model to predict the masked relationships; average pooling is performed through the embedding of triples, and a scoring function is defined to calculate the score of each triple. The model classifies the masked edges as positive samples and the randomly selected edges as negative samples. The link prediction loss is calculated based on the difference between the predicted score and the actual result.

[0017] Furthermore, the Step 2.1 includes:

[0018] For the input of the l-th (l∈[1,L]) layer Adapter It is the feature output of the (l-1)-th layer of the Adapter and the feature output of the l-th layer of mPLMs splicing in the vertical direction. Where m and n represent the length of the mPLMs input sequence and the length of the knowledge fragment, respectively, and d represents the size of the hidden layer;

[0019] Step 2.1.1, given Adapter input The Adapter layer first uses a linear projection layer to map it to the r dimension;

[0020] Step 2.1.2, then, through a Transformer layer, the event knowledge is further integrated with the language knowledge of mPLMs;

[0021] Step 2.1.3. Finally, the output of the above mapping layer Divided into two parts: the input of the residual connection layer in mPLMs And the input of the next Adapter layer

[0022] Furthermore, the specific steps of Step 2.2 are:

[0023] Step 2.2.1. Based on Step 2.1, mask pre-training has demonstrated excellent language understanding capabilities in BERT and its variants. However, random masks cannot fully capture event knowledge in generated text. To address this limitation, the present invention improves the traditional mask pre-training method and specifically masks concepts (events) extracted from multilingual knowledge graphs during text generation. Mask event nodes in the text at a certain ratio and replace them with the same number of [MASK] to generate a disrupted text sequence. For example, "earthquake(en) causes[MASK].earthquake(en)isA (da).[MASK]cause tsunami(en). ". Then, we input the disturbed sequence into the Adapter for forward reasoning. Finally, based on the output of the Adapter To predict the Masked event node and calculate its cross entropy loss

[0024] Step 2.2.2, the relationship prediction task helps the Adapter better understand the intrinsic relationship facts in the text and enhance the alignment and causal reasoning capabilities of mPLMs for multilingual events. Therefore, a link prediction task is designed to further enhance the Adapter's understanding of the relationship between event nodes and use graph structure knowledge for self-supervised learning. Specifically, certain relationships are masked in the generated text, and the model is trained to predict these masked relationships. This enhances the model's ability to infer masked relationships using knowledge of concept nodes and their surrounding graph structures. For example, "earthquake(en)[MASK]tsunami(en).earthquake(en)isA (da). (da)[MASK]tsunami(en).” Denote the set of masked hypothetical edges as ε mask ∈ε. For a given triple (h i ,r,t i ), embedding of event nodes (output of Adapter ) to perform average pooling, and finally obtain the corresponding representation of h, r, t. Then define the scoring function Θ(e h ,e t)=<h,r,t> To calculate the score of each triple,<h,r,t> represents the trilinear dot product, r represents the relationship in the text, that is, the MASK that needs to be predicted. The higher the Θ value, the more (e h ,r,e t ) is the correct triplet. The edges in the mask are classified as positive samples, and other randomly selected edges are classified as negative samples. Similarly, the relations in the mask are predicted as positive samples, and other random relations are negative samples. Link prediction loss The definition is as follows:

[0025]

[0026] Where S pos = -logσ s (θ(e h ,e t )+γ) represents the score of the correct triple, γ is the margin, σ s is the sigmoid function, {(e′ h ,r,e′ t )} is the same as the positive triple (e h ,r,e t ) corresponds to n negative triples, is the score of the negative triplet. The final objective function Defined as and A weighted combination of:

[0027]

[0028] where λ is the weight that balances the two losses.

[0029] In Step 3, the cross-language event causal relationship extraction task includes: experimental dataset, evaluation indicators, baseline model, specific implementation details of the experiment, and analysis of experimental results. It should be pointed out that during the experiment, we used the multilingual pre-trained language model XLM-R as the basic model.

[0030] In a second aspect, the present invention provides a cross-language event causal relationship recognition device enhanced by event knowledge pre-training, comprising a module for executing the method described in the first aspect.

[0031] The beneficial effects of the present invention are:

[0032] 1. A pre-training method based on ConceptNet event knowledge proposed in the present invention aims to enhance cross-language event causal relationship recognition by effectively utilizing multilingual aligned event knowledge; specifically, this method utilizes multilingual pre-trained language models (mPLMs) combined with multilingual aligned event knowledge in an external event common sense knowledge base (such as ConceptNet), and integrates multilingual event knowledge into mPLMs through reasonable pre-training task design to improve the event causal relationship recognition performance in low-resource language scenarios.

[0033] 2. The present invention extracts multilingual aligned event common sense knowledge from ConceptNet; then, a pre-training method based on parameter sharing adapter is designed to integrate event node knowledge and relationship knowledge between nodes (multilingual aligned relationships and causal relationships) into mPLMs through event mask pre-training tasks and self-supervised link prediction tasks; the pre-trained mPLMs are applied to the cross-language event causal relationship extraction task; this method significantly reduces the number of model parameters, reduces computational costs, and effectively improves the performance of event causal relationship recognition;

[0034] 3. The present invention verifies the effectiveness of our method on the MECI benchmark dataset. Experiments show that the method PTEKC proposed in the present invention can effectively integrate the multilingual aligned event knowledge extracted from the knowledge graph and the potential causal relationship between multilingual events, significantly improving the performance of the multilingual pre-trained language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is the overall flow chart of the model in the present invention;

[0036] Figure 2 The Adapter model architecture enhanced by the event knowledge of the present invention;

[0037] Figure 3 The ablation experiment results of the present invention on different languages ​​in the monolingual scenario are shown below;

[0038] Figure 4 These are the ablation experiment results on different languages ​​in the cross-language scenario of the present invention. DETAILED DESCRIPTION

[0039] Example 1: Figure 1-Figure 4 As shown in FIG. 1 , a method for cross-language event causal relationship recognition enhanced by event knowledge pre-training is provided. The specific steps are as follows:

[0040] Step 1: Extract multilingual alignment event common sense knowledge from ConceptNet, including nodes and the relationships between nodes;

[0041] Step 1.1, semantic relationship integration: From the original 47 semantic relationships extracted from ConceptNet, remove the relationships with weak relevance to the event causal relationship recognition task. Merge similar relationships into 17 streamlined semantic relationships. Example relationships include: merging "causes", "causes desire" and "motivated by goal" into "causes".

[0042] Step 1.2, data conversion: Convert the semantic relationship marker string into natural language text, for example, convert "synonyms" into "is a synonym of". Based on the converted semantic relationship text, build a training dataset, which includes inserting the head node and tail node into the corresponding positions of the semantic relationship text. For example, convert "earthquake(en)→(causes)tsunami(en)" into "earthquake(en)causes tsunami(en)". For multilingual alignment event nodes, use the "isA" relationship for data construction.

[0043] Step 1.3, pre-training data generation: Use the random walk method to traverse ConceptNet, set the random walk length to 30, sample the starting and neighboring nodes from the uniform distribution, and generate pre-training data. The specific statistical information of the MECI dataset is shown in Table 1

[0044] Table 1 Detailed statistics of the MECI dataset

[0045]

[0046] In order to further verify the generalization of the pre-training method, we refer to the process and standards of the MECI dataset construction and construct Chinese and Vietnamese event causal relationship recognition datasets. The specific statistical information is shown in Table 2.

[0047] Table 2 Detailed statistics of the Chinese and Vietnamese cross-language event causal relationship recognition dataset

[0048]

[0049] Step 2. Based on Step 1, an Adapter pre-training method based on parameter knowledge sharing is proposed, and the event node knowledge and the relationship knowledge between nodes (multilingual alignment relationship and causal relationship) are respectively integrated into mPLMs through the event mask pre-training task and the self-supervised link prediction task.

[0050] Step 2.1, Adapter architecture: embed parameter-sharing Adapter modules in each layer of mPLMs, freeze the main parameters of mPLMs, and only fine-tune the parameters of the Adapter module; map the input to r dimensions through the linear projection layer, use the Transformer layer to integrate event knowledge and mPLMs language knowledge, and finally divide the output of the mapping layer into two parts: the input of the residual connection layer and the input of the next Adapter layer;

[0051] For the input of the l-th (l∈[1,L]) layer Adapter It is the feature output of the (l-1)-th layer of the Adapter and the feature output of the l-th layer of mPLMs splicing in the vertical direction. Where m and n represent the length of the mPLMs input sequence and the length of the knowledge fragment, respectively, and d represents the size of the hidden layer;

[0052]

[0053] Step 2.1.1, given Adapter input The layer first uses a linear projection layer to map it to r dimensions;

[0054] Step 2.1.2, then, through a Transformer layer, the event knowledge is further integrated with the language knowledge of mPLMs;

[0055] Step 2.1.3. Finally, the output of the above mapping layer Divided into two parts: the input of the residual connection layer in mPLMs And the input of the next Adapter layer

[0056] Step 2.2, pre-training tasks: including event mask pre-training tasks and self-supervised link prediction tasks. The event mask pre-training task is to mask the event nodes in the text, generate a disturbed text sequence, and predict the masked event nodes based on the output of the Adapter, and calculate the cross entropy loss. The self-supervised link prediction task masks certain relationships in the text and trains the model to predict the masked relationships. Average pooling is performed through the embedding of triples, and a scoring function is defined to calculate the score of each triple. The model classifies masked edges as positive samples and randomly selected edges as negative samples. The link prediction loss is calculated based on the difference between the predicted score and the actual result.

[0057] Step 2.2.1. Based on Step 2.1, mask pre-training has demonstrated excellent language understanding capabilities in BERT and its variants. However, random masks cannot fully capture event knowledge in generated text. To address this limitation, we improved the traditional mask pre-training method and specifically masked concepts (events) extracted from multilingual knowledge graphs during text generation. We mask event nodes in the text at a certain ratio and replace them with the same number of [MASK] to generate a perturbed text sequence. For example, "earthquake(en) causes[MASK].earthquake(en)isA (da).[MASK]cause tsunami(en). ". Then, we input the disturbed sequence into the Adapter for forward reasoning. Finally, based on the output of the Adapter To predict the Masked event node and calculate its cross entropy loss

[0058] Step 2.2.2, the relationship prediction task helps the Adapter better understand the intrinsic relationship facts in the text and enhance the alignment and causal reasoning capabilities of mPLMs for multilingual events. Therefore, a link prediction task is designed to further enhance the Adapter's understanding of the relationship between event nodes and use graph structure knowledge for self-supervised learning. Specifically, certain relationships are masked in the generated text, and the model is trained to predict these masked relationships. This enhances the model's ability to infer masked relationships using knowledge of concept nodes and their surrounding graph structures. For example, "earthquake(en)[MASK]tsunami(en).earthquake(en)isA (da). (da)[MASK]tsunami(en).” Denote the set of masked hypothetical edges as ε mask ∈ε. For a given triple (h i ,r,t i ), embedding of event nodes (output of Adapter ) to perform average pooling, and finally obtain the corresponding representation of h, r, t. Then define the scoring function Θ(e h ,e t )=<h,r,t> To calculate the score of each triple,<h,r,t> represents the trilinear dot product, r represents the relationship in the text, that is, the MASK that needs to be predicted. The higher the Θ value, the more (e h ,r,e t) is the correct triplet. The edges in the mask are classified as positive samples, and other randomly selected edges are classified as negative samples. Similarly, the relations in the mask are predicted as positive samples, and other random relations are negative samples. Link prediction loss The definition is as follows:

[0059]

[0060] Where S pos = -logσ s (θ(e h ,e t )+γ) represents the score of the correct triple, γ is the margin, σ s is the sigmoid function, {(e′ h ,r,e′ t )} is the same as the positive triple (e h ,r,e t ) corresponds to n negative triples, is the score of the negative triple. The final objective function L is defined as and A weighted combination of:

[0061]

[0062] where λ is the weight that balances the two losses.

[0063] Step 3: Based on Step 2, the pre-trained mPLMs are used for the cross-language event causal relationship extraction task.

[0064] The cross-language event causal relationship extraction task includes: experimental dataset, evaluation indicators, baseline model, specific implementation details of the experiment and analysis of experimental results. It should be noted that in the experiment, the multilingual pre-trained language model XLM-R was used as the basic model. In order to ensure comparability with previous event causal relationship recognition work, precision, recall and F1 value (F1) were used as evaluation indicators. The number of events and the number of causal relationship types in each language are unbalanced. In order to further verify the effectiveness and advancement of the model of the present invention, we also used Area Under the Curve (AUC) as an evaluation indicator in the experiment and performed a significance test. The significance level was set to ρ = 0.01. In the pre-training stage, we adopted a parameter-efficient tuning strategy to solve the challenges of cross-language event causal relationship recognition. In our experimental setting, the dimension reduction factor r of the adapter was configured to 256. Consistent with previous studies, we configured the hidden dimension d of the model to 1024 and specified the number of layers L of the model to 24. The adapter was trained within 30 epochs. The training process is characterized by a batch size of 32, a learning rate of 2e-5, and weights λ selected from {0.1, 0.5}. The dimension reduction parameter r of the knowledge adapter is set to 256.

[0065] In the subsequent fine-tuning and validation phases of downstream tasks, for the ECI model, we follow the hyperparameters specified in their respective original studies to ensure fairness. It is worth noting that we use a multilingual NLP toolkit to generate dependency trees in different languages ​​in the RichGCN framework. In addition, in order to meet the specific needs of various models, a multilingual version of ConceptNet is used.

[0066] In the study of cross-lingual event causal relationship identification, a three-stage training method was adopted. First, in the first stage, we further pre-trained the multilingual pre-trained language model with a multilingual aligned event knowledge base to enhance the model's understanding of events in different languages. Then, in the second stage, we used English, a resource-rich language, and trained the ECI model with the help of the event causal relationship identification dataset to ensure that the model has the ability to accurately identify event causality in a resource-rich language environment. Finally, in the third stage, we migrated the ECI model pre-trained in a resource-rich language to a low-resource language environment, and then fine-tuned the ECI model based on specific low-resource language data, or directly performed zero-shot testing in a low-resource language scenario to verify the cross-lingual applicability and generalization ability of the model.

[0067] According to the concept of the present invention, the present invention also provides a cross-language event causal relationship recognition device enhanced by event knowledge pre-training, which includes the following modules:

[0068] The extraction module is used to extract multilingual alignment event knowledge from ConceptNet, including nodes and the relationships between nodes;

[0069] The training module is used to propose an Adapter pre-training method based on parameter knowledge sharing, which integrates event node knowledge and node relationship knowledge into mPLMs through event mask pre-training tasks and self-supervised link prediction tasks respectively;

[0070] The extraction module is used to use the pre-trained mPLMs for the cross-language event causal relationship extraction task.

[0071] In order to verify the effectiveness and advancement of the pre-training method proposed in this paper, we compared the proposed method with a parameter-efficient fine-tuning method and the current most advanced ECI model.

[0072] Language model fine-tuning method: On the MECI dataset, we compare the proposed method with the currently popular parameter-efficient fine-tuning pre-training method. To ensure a fair comparison, we introduce our extracted external event knowledge into the baseline method. Specifically, we concatenate the extracted training corpus with the original input of the baseline method as the input of the baseline model.

[0073] Bottleneck Adapter: This method was first proposed by Houlsby et al. in NLP tasks. It is an innovative, Adapter-based pre-training method

[0074] Prefix Tuning: To reduce training cost, Prefix Tuning embeds learnable cues in the input and intermediate layers.

[0075] LoRA: In each layer of PLM, LoRA predefines a trainable rank factorization matrix to minimize the number of trainable parameters.

[0076] MAM Adapter: MAM Adapter fully combines the advantages of adapter, prefix tuning and low-rank methods, and designs an adapter-based pre-training method.

[0077] Compacter: Compacter fully combines the advantages of components such as adapters, low-rank optimization, and parameterized hypercomplex multiplication layers to find a balance between task performance and the number of model training parameters.

[0078] Existing ECI models: To further verify the effectiveness of the pre-training method proposed in this paper, we applied the pre-trained XLM-R to the downstream cross-language event causal relationship recognition task and compared and analyzed its performance with the following ECI models:

[0079] RichGCN: Builds an interactive graph of events at the document level and uses a graph convolutional network (GCN) to model the causal relationship between events.

[0080] ERGO: Chen et al. transformed the event causal relationship recognition task into a graph node classification task, and proposed an event causal relationship recognition method based on the Event Relational Graph Transformer (ERGO)

[0081] KEPT: We introduce a hint tuning framework to transform external knowledge into textual narratives and infer causal relationships through a hint-based approach.

[0082] Know: Retrieve relevant concepts and relations of the input events in the ECI task from ConceptNet by conceptualizing event mentions as entities in ConceptNet.

[0083] SemSIn: Utilizes two different semantic frameworks, namely event-centric structure and event-dependent structure, to enhance the effectiveness of ECI solutions.

[0084] CHEER: provides a holistic view of GNN-based DECI modeling and introduces the Event Interaction Graph (EIG) to encapsulate the antecedent event relations and complex interactions while enhancing event centrality through carefully designed features and adopting a multi-task learning strategy.

[0085] GCKAN: incorporates a large amount of event-related background knowledge from knowledge graphs and addresses the noise problem in these graphs.

[0086] DiffusECI: Man et al. proposed a method for event causal relationship identification based on diffusion model.

[0087] DFP: et al. proposed an event causal relationship identification method based on optimal transportation method and background knowledge retrieval.

[0088] GIMC: A zero-shot cross-lingual event causal relationship identification method based on multi-granularity contrast transfer students proposed by He et al.

[0089] To ensure fairness, we compare their performance under the multilingual pre-trained language model (XLM-R) implementation. Each model will be fully evaluated using the MECI dataset under single-language and multi-language learning conditions. The pre-training method proposed in this paper is denoted as PTEKC (Pre-training with Event Knowledge of ConceptNet, PTEKC).

[0090] In order to illustrate the detection effect of the present invention, a baseline method is used to compare the results of the present invention, specifically, the language model fine-tuning performance is compared with the following core method.

[0091] (1) PTEKC consistently outperforms all baseline methods in all languages ​​of the MECI dataset. Compared with the best performing baseline method, PTEKC improves the performance of English, Danish, Spanish, Turkish, and Urdu by 2.3%, 3.2%, 2.3%, 2.1%, and 5.1%, respectively. This experimental result verifies the effectiveness of the PTEKC pre-training method in terms of pre-training performance and event knowledge integration.

[0092] (2) Compared with Bottleneck Adapter, Prefix Tuning, and MAM Adapter, the PTEKC parameter fine-tuning method significantly reduces the number of introduced parameters, but has a significant improvement in the F1 score on the MECI dataset. We attribute this improvement to the efficient parameter sharing strategy in our proposed pre-training method and the successful integration of external event knowledge into mPLMs through two specially designed pre-training tasks.

[0093] (3) Compared with Compacter and LoRA, although the first two parameter fine-tuning methods have fewer parameters, their performance lags significantly behind PTEKC, with a difference of 9.6% and 5.3% in F1 score, respectively. This further shows that the PTEKC pre-training method achieves an excellent balance between the number of model training parameters and ECI performance.

[0094] Table 3 Performance comparison of efficient parameter fine-tuning methods on the MECI dataset

[0095]

[0096] In order to explore the event causal reasoning ability of the PTEKC pre-training method in a single language scenario, we conducted experiments on the MECI dataset (English, Danish, Spanish, Turkish and Urdu) and self-built datasets (Chinese and Vietnamese). Tables 4 and 5 show in detail the performance of the current state-of-the-art ECI model on two multilingual pre-trained language models, XLM-R and PTEKC. Among them, the symbol "Δ" is used to indicate the average performance improvement achieved by the baseline model on all languages ​​compared with XLM-R on PTEKC.

[0097] Table 4 Event causality recognition performance in English, Spanish, Turkish and Chinese

[0098]

[0099] Table 5 Event causality recognition performance on Danish, Urdu and Vietnamese

[0100]

[0101] From the experimental results in Table 4 and Table 5, we can see that:

[0102] (1) In all languages, the baseline model based on PTEKC achieved the best performance compared with XLM-R, with an average F1 value improvement of 3.6%. This shows the effectiveness of the pre-training method proposed in this paper in monolingual scenarios. Incorporating external event knowledge into multilingual pre-trained language models through pre-training tasks can further improve the causal reasoning ability of the ECI model.

[0103] (2) Except for PLMs and DiffusECI, we divide all ECI models listed in Tables 4 and 5 into two categories. The first category of models, such as Know, KEPT, GCKAN, and DFP, effectively improve the causal reasoning ability of ECI models by incorporating external knowledge. The second category of models, such as RichGCN, ERGO, SemSIn, CHEER, and GIMC, use graph neural networks to finely model the association relationship between events, thereby strengthening the causal reasoning ability of ECI models. Although the integration of multilingual event knowledge in the pre-training stage brings significant performance improvement to the ECI model incorporating external knowledge (the F1 value is increased by an average of 3.2%), its performance improvement is relatively small compared with the ECI baseline model represented by RichGCN (the average F1 value increase is 3.9%). After in-depth analysis, we believe that this may be because the self-supervised link prediction pre-training task enhances the model's ability to capture the causal characteristics between events, so that the model represented by RichGCN shows a more obvious advantage in performance.

[0104] This paper aims to further explore the ability of PTEKC pre-training method to transfer cross-language knowledge between different languages ​​in cross-language scenarios. We conducted cross-language knowledge transfer experiments on MECI dataset and self-built dataset. Table 5.8 lists the experimental results on different languages ​​in detail. It should be noted that we specify English as a resource-rich language, i.e., the source language, and other languages ​​as target languages.

[0105] Table 6 Experimental results of event causal relationship recognition in cross-language scenarios

[0106]

[0107]

[0108] From the experimental results in Table 6, we can conclude that:

[0109] The experimental results clearly reveal that there is a significant difference in performance between zero-shot transfer and monolingual fine-tuning. In the ECI task, we found that zero-shot transfer using XLM-R encountered significant challenges. By comparing the experimental results in Table 4 and Table 5, we can further conclude that XLM-R performs significantly better than zero-shot transfer in a monolingual environment, especially when there is target language training data in the ECI task. It is worth noting that the best performing ECI model DiffusECI, after using the PTEKC pre-training method, only has a 6.5%, 6.4% and 8.4% decrease in F1 value for Danish, Spanish and Turkish respectively compared to the monolingual scenario. Interestingly, its F1 value for Urdu and Vietnamese has increased by 0.6% and 0.4%. After in-depth analysis, we speculate that this may be due to the relatively small amount of training data for Urdu and Vietnamese in the dataset, which makes the large-scale English training data perform better when training test data models for these two languages. This phenomenon also further verifies the effectiveness of our proposed pre-training method in solving the problem of cross-language knowledge transfer in low-resource language scenarios.

[0110] By incorporating multilingual aligned event knowledge, the performance of XLM-R in zero-shot cross-language transfer tasks can be significantly improved. Experimental results show that in the zero-shot cross-language transfer scenario, the model using the PTEKC pre-training method, which integrates multilingual aligned event knowledge, has an average performance improvement of 4.7% compared to the ECI model based on the original XLM-R. This result confirms our previous hypothesis that incorporating multilingual aligned event knowledge in pre-training can significantly improve the performance of multilingual pre-trained language models in cross-language event causal relationship recognition tasks.

[0111] This paper also mainly explores the impact of different component implementations in the PTEKC framework on the performance of event causal relationship recognition. In order to achieve the above goals, we set up two sets of experiments, namely the impact of different component implementations on the performance of the ECI model in a monolingual scenario and a cross-language scenario. The PTEKC proposed in this paper is compared with the following variant models, including:

[0112] PTEKC(-SMT): indicates that the event mask pre-training task is deleted in the PTEKC pre-training method.

[0113] PTEKC(-LPT): indicates that the self-supervised link prediction task is removed in the PTEKC pre-training method.

[0114] PTEKC(-SL): Use the traditional random mask pre-training task instead of the event mask pre-training and self-supervised link prediction tasks.

[0115] PTEKC(-Att): Use the traditional nonlinear function to replace the self-attention mechanism in Adapter.

[0116] according to Figure 3 and Figure 4 From the experimental results presented, it can be inferred that removing any component from the PTEKC model will lead to a decrease in the performance of the event causal relationship recognition model, whether in a monolingual environment or a cross-lingual environment. This finding demonstrates the importance and effectiveness of each component in the PTEKC model in improving the overall performance. Further observation shows that the performance of the model decreases particularly significantly when the event mask task and the self-supervised link prediction task are removed. This observation further verifies that the two pre-training tasks we designed have a positive effect in helping the adapter capture the alignment of cross-lingual event knowledge, thereby improving the accuracy of cross-lingual event causal relationship recognition. In addition, PTEKC outperforms PTEKC(-Att) in all tested languages, which fully demonstrates that the self-attention mechanism in the Transformer layer of the adapter module plays a key role in promoting the fusion of multilingual event alignment knowledge and intrinsic knowledge in the language model, thereby strengthening the modeling ability of mPLM for cross-lingual event knowledge.

[0117] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A cross-language event causal relationship recognition method enhanced by event knowledge pre-training, characterized by: The method incorporates multilingual aligned event knowledge into mPLMs by designing pre-training tasks, and then the mPLMs enhanced with multilingual aligned event knowledge are directly used for cross-language event causal relationship extraction tasks.

2. The cross-language event causal relationship recognition method enhanced by event knowledge pre-training according to claim 1 is characterized by: The specific steps of the method are as follows: Step 1: Extract multilingual alignment event knowledge from ConceptNet, including nodes and the relationships between nodes; Step 2: Based on Step 1, an Adapter pre-training method based on parameter knowledge sharing is proposed. Event node knowledge and relationship knowledge between nodes are integrated into mPLMs through event mask pre-training tasks and self-supervised link prediction tasks respectively. Step 3: Based on Step 2, the pre-trained mPLMs are used for the cross-language event causal relationship extraction task.

3. The cross-language event causal relationship recognition method enhanced by event knowledge pre-training according to claim 2 is characterized by: The specific steps of Step 1 are: Step 1.1, semantic relationship integration: from the original semantic relationships extracted from ConceptNet, delete the relationships that are weakly relevant to the event causal relationship recognition task; merge similar relationships into several streamlined semantic relationships; Step 1.2, data conversion: convert the semantic relationship mark string into natural language text, and build a training data set based on the converted semantic relationship text, which includes inserting the head node and tail node into the corresponding positions of the semantic relationship text; for multi-language alignment event nodes, use the "isA" relationship to build data; Step 1.3, pre-training data generation: Use the random walk method to traverse ConceptNet, set the random walk length, sample the starting and neighboring nodes from the uniform distribution, and generate pre-training data.

4. The cross-language event causal relationship recognition method enhanced by event knowledge pre-training according to claim 2 is characterized by: The specific steps of Step 2 are: Step 2.1, build the Adapter architecture: embed the parameter-sharing Adapter module in each layer of mPLMs, freeze the main parameters of mPLMs, and only fine-tune the parameters of the Adapter module; map the input to r dimensions through the linear projection layer, use the Transformer layer to integrate event knowledge and mPLMs language knowledge, and finally divide the output of the mapping layer into two parts: the input of the residual connection layer and the input of the next Adapter layer; Step 2.2, perform pre-training tasks: The pre-training tasks include event mask pre-training tasks and self-supervised link prediction tasks. The event mask pre-training task is to mask the event nodes in the text, generate a disturbed text sequence, and predict the masked event nodes based on the output of the Adapter, and calculate the cross entropy loss; the self-supervised link prediction task masks certain relationships in the text, and trains the model to predict the masked relationships; average pooling is performed through the embedding of triples, and a scoring function is defined to calculate the score of each triple; the masked edges are classified as positive samples, and the randomly selected edges are classified as negative samples. The link prediction loss is calculated based on the difference between the predicted score and the actual result.

5. The cross-language event causal relationship recognition method enhanced by event knowledge pre-training according to claim 3 is characterized by: The Step 2.1 includes the following: For the input of the l-th (l∈[1,L]) layer Adapter It is the feature output of the (l-1)-th layer of the Adapter and the feature output of the l-th layer of mPLMs splicing in the vertical direction; where m and n represent the length of the mPLMs input sequence and the length of the knowledge fragment, respectively, and d represents the size of the hidden layer; Step 2.1.1, given Adapter input The Adapter layer first uses a linear projection layer to map it to the r dimension; Step 2.1.2, then, through a Transformer layer, the event knowledge is further integrated with the language knowledge of mPLMs; Step 2.1.

3. Finally, the output of the above mapping layer Divided into two parts: the input of the residual connection layer in mPLMs And the input of the next Adapter layer 6. The cross-language event causal relationship recognition method enhanced by event knowledge pre-training according to claim 3 is characterized by: The specific steps of Step 2.2 are: Step 2.2.1: Based on Step 2.1, mask the concepts extracted from the multilingual knowledge graph during text generation; Mask the event nodes in the text according to a certain ratio and replace them with the same number of [MASK] to generate a disturbed text sequence; then, input the disturbed sequence into the Adapter for forward reasoning; finally, based on the output of the Adapter To predict the Masked event node and calculate its cross entropy loss Step 2.2.

2. A link prediction task is designed to further enhance the Adapter’s understanding of the relationship between event nodes and use graph structure knowledge for self-supervised learning. Specifically, some relationships are masked in the generated text, and the model is trained to predict these masked relationships. The set of masked hypothetical edges is represented as ε mask ∈ε, for a given triple (h i ,r,t i ), embedding the event node, that is, the output of the Adapter Perform average pooling to finally obtain the corresponding representations of h, r, and t; then define the scoring function Θ(e h ,e t )=<h,r,t> To calculate the score of each triple,<h,r,t> represents the trilinear dot product, r represents the relationship in the text, that is, the MASK that needs to be predicted; the higher the Θ value, the better (e h ,r,e t ) is the correct triplet; the edges in the mask are classified as positive samples, and other randomly selected edges are classified as negative samples; similarly, the relations in the mask are predicted as positive samples, and other random relations are negative samples; the link prediction loss The definition is as follows: Where S pos = -logσ s (θ(e h ,e t )+γ) represents the score of the correct triple, γ is the margin, σ s is the sigmoid function, {(e ′ h ,r,e ′ t )} is the same as the positive triple (e h ,r,e t ) corresponds to n negative triples, is the score of the negative triplet; the final objective function Defined as and A weighted combination of: where λ is the weight that balances the two losses.

7. A cross-language event causal relationship recognition device enhanced by event knowledge pre-training, characterized in that: The method comprises a module for executing the method according to any one of claims 1 to 6.

Images