A document-level event causal relationship identification method based on graph reinforcement learning

Abstract

The application discloses a kind of based on graph reinforcement learning's document-level event causal relationship identification method, comprising the following steps: obtaining text data containing event;Using large language model as external knowledge base and reasoner, generating prior causal graph, obtaining initial graph structure;Using long text pre-training language model to obtain the representation of contextualization event and calculate the correlation degree of event pair, provide prior knowledge;Introduce attention mask and self-attention initialization mechanism, generate the representation of contextualization event, integrate prior causal graph and contextualization event correlation into Transformer model to guide and control the training of model;Using graph reconstruction loss and attention distribution offset regularization loss to train and optimize model;Output the causal relationship between events extracted in gene database. The application introduces the self-attention initialization mechanism based on event correlation, and limits the attention of Transformer in reasonable distribution space.

Description

Technical Field

[0001] This invention belongs to the field of big data technology, and in particular relates to a document-level event causal relationship recognition method based on graph reinforcement learning. Background Technology

[0002] Event causality identification (ECI) is an important research area in Natural Language Processing (NLP), aiming to automatically detect and extract causal relationships between events from textual data and uncover potential logical causal relationships. Clarifying event causal relationships plays a crucial role in explaining events and guiding process analysis, enriching narrative understanding and optimizing information retrieval. In scientific research, causal relationships between events are an important means of exploring natural phenomena and verifying scientific hypotheses. By extracting and analyzing these causal relationships from a large amount of literature, researchers can better understand natural phenomena and discover new scientific laws. Especially in gene-related research and exploration, many problems are often encountered, such as the causal relationship between gene-protein interactions, the causal relationship between gene expression in different tissues and cells and gene structure, and the causal correlation between genes and disease prognosis, etc.

[0003] Depending on whether the events are within the same sentence, ECI tasks can be divided into sentence-level ECI (SECI) and document-level ECI (DECI). Understanding causal relationships, especially those in long texts and document-level events, requires a deep understanding of contextual clues, domain-specific knowledge, and implicit semantic logic.

[0004] Early ECI (Electronic Language Interference) methods discovered causal relationships by identifying specific features or patterns, such as causal triggers, temporal patterns, and co-occurrence patterns. However, these methods could only identify explicit causal relationships and struggled to uncover implicit causal connections implied in semantics. In recent years, neural network-based ECI models have become mainstream research tools due to their powerful ability to learn complex features. Especially with the development and widespread adoption of pre-trained language models (PLMs), more and more PLMs are being used in ECI tasks. Currently, deep learning-based ECI methods mainly include graph-based methods, external knowledge augmentation methods, and PLM fine-tuning methods.

[0005] Graph-based methods treat ECI as a node classification or edge prediction problem, using graph neural networks such as Graph Convolutional Networks (GCNs) for modeling. However, these methods often require starting with a fully connected event-causal graph, which introduces significant noise when applied to long texts with numerous events. Furthermore, these methods require effective event embeddings, but in long texts, event mentions are often obscured by a large amount of irrelevant contextual information, making it difficult to obtain reasonable representations. External knowledge augmentation methods utilize commonsense knowledge bases (such as ConceptNet and WordNet) to find relevant knowledge about event mentions, enriching background knowledge and enhancing training data to assist in the training and inference of neural network models. However, this commonsense knowledge (such as triples) is generated without context, and when external knowledge is inconsistent with the text content, it may cause information interference and confusion. PLM-based fine-tuning methods use standard or cue-based fine-tuning methods to fine-tune the PLM on labeled data, adapting it to the ECI task. This method is highly dependent on data quality and cue templates and performs poorly in cross-topic transfer. Additionally, label imbalance can lead to a large number of missed cases. Summary of the Invention

[0006] To address the limitations of existing methods, this invention proposes a Graph Enhanced Transformer (GET) model. GET is a model based on the Transformer framework, combining the three methods mentioned above. First, a Large Language Model (LLM) is used as an external knowledge base and inferencer to generate a prior causal graph, obtaining a more refined initial graph structure to reduce noise and balance labels. A Longformer pre-trained language model for long texts is used to obtain reasonable contextualized event representations and calculate the relevance of event pairs, providing prior knowledge. Then, by introducing attention masks and self-attention initialization mechanisms, the prior causal graph and contextualized event relevance are integrated into the Transformer model to guide and control model training. The application of LLM and Longformer is end-to-end, requiring no training or fine-tuning steps. Finally, graph reconstruction loss and attention distribution shift regularization loss are designed to train and optimize the model. The GET model framework is as follows: Figure 1 As shown.

[0007] To achieve the above objectives, this application discloses a document-level event causality recognition method based on graph reinforcement learning, comprising the following steps:

[0008] Retrieve text data T containing events from a gene database; for example, the gene database is the Genecards database;

[0009] By using a large language model as an external knowledge base and inference engine, a prior causal graph is generated to obtain an initial graph structure, thereby reducing noise and balancing labels.

[0010] Utilize long-text pre-trained language models to obtain contextualized event representations and calculate the relevance of event pairs, providing prior knowledge;

[0011] Attention masks and self-attention initialization mechanisms are introduced, and Longformer is used to generate contextualized representations of events. Prior causal graphs and contextualized event relevance are integrated into the Transformer model to guide and control the training of the model.

[0012] The model is trained and optimized using graph reconstruction loss and attention distribution shift regularization loss to extract causal relationships between events;

[0013] Output the causal relationships between events identified in the gene database.

[0014] Preferably, a priori causal graph (PCG) is used to limit the number of connections between the input event nodes; for text data T and the event mention set M = {m1, m2, ..., m} therein, ... N ,},m i ∈T, where N is the number of event mentions; the prior causal graph is G. p = {V, E}, where V = {v1, v2, ..., v N ,} represents a set of nodes, where each node v i m corresponds to an event i E = {e1, e2, ..., e} |E|} represents the set of edges, where each edge corresponds to a pair of nodes, i.e., e k =(v i ,v j ), representing m i and m j A causal relationship exists; the prior causal graph is a directed acyclic graph, and the edge information in the graph is used to initialize the connection relationships between event mentions of the input Transformer.

[0015] Preferably, an end-to-end prior causal graph (PCG) is constructed using a large language model based on prompting learning; the construction process is as follows:

[0016] First, a prompt template is designed for LLM, and examples are provided for contextual learning. Then, the mind chain technique is used to guide LLM to reason step by step and determine the causal relationship between each pair of events.

[0017] Specifically, in a single question-and-answer session with an LLM, the prompt word P is input into the large model, and P takes the following form:

[0018] P = [E1; E2; T; (m i ,m j );q],

[0019] Among them, E1 and E2 are examples of positive and negative reasoning, and each example contains three elements: <question, reasoning process, answer>; T and (m i ,m j ) represents the text and target event pair, and q represents the standard format of the causal identification question;

[0020] Then, P is input into the LLM to generate normalized output:

[0021] O = LLM(P) .

[0022] If O = "Yes", then add edge e = (v i ,v j If O = "No", then no edge is added to G; otherwise, no edge is added.

[0023] Preferably, the contextualized representation of the Longformer generated event includes:

[0024] For each text T and event mention set M, add start and end markers [s] and [s / ] to the text, and add special markers [m] and [m / ] to each event mention to identify the position of the event mention in the text;

[0025] The text data T is segmented into multiple tokens w1, w2, ..., w using a tokenizer. |T|+2N+2 , where w1 = w [s] w T+2|N|+2 =w [s / ] Then, use Longformer to encode it into the corresponding embedding vector:

[0026] T = [w1, w2, ..., w |T|+2N+2 ],

[0027] Obtain the embedding vector of the tag [m] corresponding to each event mention, and concatenate it with the overall embedding vector of the text to obtain the contextual embedding of that event mention:

[0028] m i =[w1;w idx([m(i)]) ]

[0029] idx([m(i)]) represents the starting index of the i-th event mention, and ";" indicates vector concatenation;

[0030] For all event mention embeddings, cosine similarity is used to calculate contextualized event relevance:

[0031]

[0032] To avoid the semantic gap reduction problem caused by using high-dimensional vectors to calculate cosine similarity, the baseline value b is subtracted from the generated relevance value to increase its discriminative power; in addition to calculating relevance, m i It is input into the Transformer as the initial embedding vector for events in the model.

[0033] Preferably, the attention masking mechanism includes:

[0034] The attention matrix is ​​calculated using the following formula:

[0035]

[0036] in, Let d be the mask attention weight matrix. k Let Q and K be the embedding dimensions of the Key, where Q and K are the embedding vectors of the Query and Key after linear transformation, and M ∈ R. N×N Let e ​​be the attention mask matrix, if e = (v i ,v j If )∈E, then M ij =0, otherwise M ij Set to -∞; introduce M ij This simplifies the node connection structure by masking irrelevant causal event pairs during model training.

[0037] Preferably, the self-attention initialization mechanism formalizes the relevance of all event pairs into an asymmetric relevance matrix C∈R. N×N To avoid initializing attention weights to a uniform distribution, where C ij The attention weight of i to j is represented by the following formula:

[0038]

[0039] C is input into the Transformer as the attention for the first block, and a masked attention mechanism, i.e., a self-attention mechanism, is used in subsequent blocks as follows:

[0040]

[0041] H (0) H is the initial embedding representation of the model. (1) H is the embedding representation of the model in layer 1. (j) H is the embedding representation of the model in the j-th layer. (j-1) This represents the embedding representation of the model in the (j-1)th layer. This is the parameter matrix for the first layer. Let j be the parameter matrix of the j-th layer. It is a masked attention matrix The j-th row;

[0042] Then, using a multi-head attention mechanism, the outputs of h attention heads are concatenated, and a feedforward network is used for further nonlinear processing, namely:

[0043]

[0044] The output is the result of concatenating multiple heads. FFNN represents a feedforward network, LeakyReLU is the activation function of the neural network unit, and W1 and W2 are learnable parameter matrices.

[0045] Preferably, to construct the graph reconstruction loss function, each event embedding is first concatenated, and its feature relationships are captured using a multilayer perceptron (MLP). Then, the softmax function is used to obtain the predicted label probability p. ij :

[0046] p ij =softmax(MLP[h i h j ]),

[0047] h i and h j These are the node embedding matrices. The i-th and j-th elements, d model It is the embedding dimension of the model;

[0048] The order of splicing corresponds to the order of the event pairs to ensure that causal relationships representing two different directions can be distinguished. These correspond to the probabilities of reverse causality, no causality, and positive causality, respectively.

[0049] Finally, for each training text and its corresponding event mentions and labels, an adaptive focusing loss designed for label imbalance scenarios is used as the reconstruction loss function for the causal graph, i.e.:

[0050]

[0051] Where, α k (j=1,2,3) are the balance parameters, γ is the adjustment factor, and α satisfies j ∈(0,1),α1+α2+α3=1, It is the probability that the i-th node and the j-th node have the k-th relationship, where k = 1, 2, 3 represent reverse causal relationship, no causal relationship, and positive causal relationship, respectively.

[0052] Preferably, the regularization loss of the attention distribution shift is gradually improved by fine-tuning the event relevance matrix C, but the masked attention matrix is ​​not optimized. The deviation from the event relevance matrix C is within a set threshold; the regularization loss of the attention distribution shift is defined as:

[0053]

[0054] D KL Denotes the KL divergence operator, and It is a masked attention matrix The j-th and j-1-th rows;

[0055] The summation of the regularization loss functions for each block is used as the final regularization loss.

[0056] Preferably, the graph reconstruction loss function and the regularization loss of the attention distribution shift are weighted and summed to obtain the overall training loss:

[0057]

[0058] λ is a hyperparameter that controls the weights of the regularization loss; the model is trained using the AdamW optimizer with L2 regularization.

[0059] The beneficial technical effects of the present invention are as follows:

[0060] (1) A graph-enhanced Transformer (GET) model is proposed. GET leverages the powerful semantic capabilities of LLM to generate prior causal graphs and designs reasonable strategies to enhance the task-specific fine-tuning of small-scale pre-trained language models (Transformers). This paradigm is not only applicable to ECI tasks but can also be extended to other downstream tasks that require fine-tuning.

[0061] (2) A prior event causal graph generation method based on LLM is proposed, which uses In-Context Learning (ICL) and Chain of Thought (CoT) to guide large models to generate prior causal graphs based on text and knowledge. An attention masking mechanism is introduced to integrate the prior causal graph into the Transformer model.

[0062] (3) A contextualized event representation and event pair relevance calculation method based on Longformer is proposed. A self-attention initialization mechanism based on event relevance is introduced, and an attention distribution offset regularization loss function is designed to restrict the attention of Transformer to a reasonable distribution space.

[0063] (4) Validation results on the Genecards dataset show that GET outperforms state-of-the-art methods in document-level ECI tasks and exhibits stronger robustness in long text ECI tasks. Attached Figure Description

[0064] Figure 1 Framework diagram of the present invention;

[0065] Figure 2 Contextualized event representation learning and relevance matrix calculation based on Longformer;

[0066] Figure 3 The correlation between causal events changes with the number of training iterations;

[0067] Figure 4 Performance comparison of GET and ERGO on texts of different lengths. Detailed Implementation

[0068] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0069] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0070] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0071] The technical solutions provided in this application involve technologies such as machine learning and natural language processing in artificial intelligence, which are specifically described and illustrated through the following embodiments.

[0072] Before introducing the embodiments of this application, some terms involved in this application will be explained. 1. Graph-based event causality identification: This type of method models the ECI task as a causal graph reasoning problem and uses graph modeling methods to mine causal relationships. Phu and Nguyen captured the connections between event mentions in the input document by constructing an interaction graph by introducing discourse, grammar, and semantic information, and then used GCN to learn an enhanced contextual representation of the document to predict event causality. Graph-based methods require supervised learning on a large amount of annotated data.

[0073] 2. External Knowledge-Enhanced Event Causal Relationship Recognition: Knowledge graphs (KGs) store a large number of relation triples, which can provide causal background knowledge related to event mentions, alleviating the problem of scarce labeled data. In recent years, more and more research has been conducted on ECI (External Knowledge Enhancement) by jointly modeling text and knowledge graphs.

[0074] 3. Event Causality Identification Based on PLM Fine-tuning: This type of method identifies causal relationships by fine-tuning the PLM for ECI tasks.

[0075] refer to Figure 1 The Graph Augmentation Transformer (GET) model proposed in this invention is based on a prior causal graph and uses Transformer to reconstruct a contextualized event causal graph to identify causal relationships in gene database text. GET can be regarded as a graph reasoning model that integrates Graph Convolutional Network (GCN) and Transformer, and uses the Transformer model framework to model the graph. To adapt to the ECI task, GET makes three improvements to Transformer: (1) replaces the original fully connected structure with a prior causal graph and introduces graph knowledge into the Transformer model using an attention masking mechanism; (2) uses Longformer to generate contextualized representations of events and calculates event relevance to initialize attention; (3) designs two loss functions, graph reconstruction loss and attention distribution shift regularization loss, to train the model.

[0076] In ECI tasks, causal connections between events are sparse. Using fully connected structures increases computational complexity and introduces unnecessary noise. Furthermore, long texts often contain a large number of events, and building fully connected graphs can lead to severe data imbalance and a large number of false positives. To address this issue, this invention introduces a Prior Causality Graph (PCG) to reasonably limit the number of connections between input event nodes. For the original text T and the event mention set M = {m1, m2, ..., m...} N ,},m i∈T, where N is the number of event mentions. The prior causal graph is denoted as G. p = {V, E}, where V = {v1, v2, ..., v N ,} represents a set of nodes, where each node v i m corresponds to an event i E = {e1, e2, ..., e} |E|} represents the set of edges, where each edge corresponds to a pair of nodes, i.e., e k =(v i ,v j ), representing m i and m j A causal relationship exists. (PCG) G It is a directed acyclic graph, and the edge information in the graph is used to initialize the connection relationships between the event mentions (nodes) of the input Transformer.

[0077] However, constructing a PCG is a challenging task. First, the PCG needs to include as many correct causal event pairs as possible to improve the upper limit of recall; second, the PCG needs to filter out false causal relationships as much as possible to prevent the introduction of excessive noise and false positives. This invention utilizes a Large Language Model (LLM) based on cue learning for end-to-end PCG construction.

[0078] Benefiting from powerful semantic understanding and context learning capabilities, large language models like ChatGPT can achieve zero-shot ECI tasks and achieve recall (the proportion of true causal event pairs correctly identified) of over 95%. Furthermore, this approach builds PCGs end-to-end, requiring only calls to the model API without access to the model's internal architecture, resulting in high efficiency. However, the inherent causal illusion in LLMs leads to a high number of false positives (accuracy is less than 30%), making them unsuitable for direct use in ECI tasks.

[0079] This invention structures the causal event pairs identified by LLM into a PCG (Programmatical Convolutional Graph). Compared to initializing a fully connected graph, this approach generates a more refined structure, reducing redundant edges in the graph while preserving as many true causal event pairs as possible. The construction process is as follows: First, a prompt template is designed for LLM, and examples are provided for context learning (ICL). Then, the CoT (Cooperative Reasoning) technique guides LLM to reason step by step, determining the causal relationship between each pair of events. Specifically, in a single question-and-answer session with LLM, the prompt word P is input into the large model, and P takes the following form:

[0080] P = [E1; E2; T; (m i ,m j );q],(1)

[0081] Among them, E1 and E2 are examples of positive and negative reasoning, and each example contains three elements: <question, reasoning process, answer>; T and (m i ,m j ) represents the text and target event pair, and q represents the standard format of the causal identification question.

[0082] Then, P is input into the LLM to generate normalized output:

[0083] O = LLM(P).(2)

[0084] If O = "Yes", then add edge e = (v i ,v j If O = "No", then no edge is added to G; otherwise, no edge is added. In actual use, for sentence-level ECI, only the sentence itself is provided as the semantic background, while for document-level ECI, the entire text is provided as the semantic background.

[0085] Contextualized event representation

[0086] In a given text, the information about each event is context-specific; therefore, when learning event representations, it is necessary to generate context-based embeddings. Furthermore, for the ECI task, the degree of association between each pair of event mentions is context-specific, known as context-ualized event correlation (CEC). Event pairs with semantic relationships should have a higher CEC than unrelated event pairs. This semantic correlation is invaluable for identifying causal relationships during the modeling of event mentions.

[0087] To obtain reasonable event contextual representations and semantic relevance, and provide them as prior knowledge to the Transformer, this invention proposes a contextual embedding computation method based on Longformer. Longformer is an upgraded version of Transformer that can efficiently process long texts. Compared to Transformer, it incorporates more contextual information through sliding window dilated convolution and local attention mechanisms, demonstrating a stronger ability to learn contextual embedding representations of specific tags.

[0088] For each text T and event mention set M, first, start and end markers [s] and [s / ] (equivalent to [CLS] and [SEP] in BERT) are added to the text. Each event mention is marked with special markers [m] and [m / ] to identify its position within the text. Then, a tokenizer is used to segment T into multiple markers w1, w2, ..., w |T|+2N+2 , where w1 = w [s] wT+2|N|+2 =w [s / ] Then, use Longformer to encode it into the corresponding embedding vector:

[0089] T = [w1, w2, ..., w |T|+2N+2 ],(3)

[0090] Then, the embedding vector of the tag [m] corresponding to each event mention is obtained and concatenated with the overall embedding vector of the text to form the contextual embedding of that event mention:

[0091] m i =[w1;w idx([m(i)]) (4)

[0092] `idx([m(i)])` represents the starting tag index corresponding to the i-th event mention, and ";" indicates vector concatenation. Finally, for all event mention embeddings, cosine similarity is used to calculate the contextualized event relevance:

[0093]

[0094] To address the potential semantic gap reduction issue caused by using high-dimensional vectors to calculate cosine similarity, a baseline value 'b' is uniformly subtracted from the generated relevance values ​​to increase their discriminative power. Besides calculating relevance, m... i It is also input into the Transformer as the initial embedding vector for events in the model. Figure 4 This demonstrates the process of encoding events and calculating relevance using Longformer.

[0095] It's important to note that GET did not use Longformer as the primary model; it only utilized it to generate initial embeddings and calculate relevance. The rationale for this choice is that subsequent modeling processes are fine-tuning of the ECI task and do not require further processing of long texts. Continuing to use Longformer would necessitate handling more parameters and placing higher demands on computational resources. In contrast, Transformer has a simpler structure and lower computational cost.

[0096] GET's self-attention mechanism

[0097] In the Transformer's self-attention mechanism, the attention weights are calculated by multiplying the inner product of the query and the key, and then obtaining them through scaling and softmax transformation, i.e.:

[0098]

[0099] A is the attention weight matrix, d kLet Q and K be the embedding dimensions of the Key, and Q and K be the linearly transformed embedding vectors of the Query and Key.

[0100]

[0101] H (j) This represents the embedding representation of the model in the j-th layer. Let be the parameter matrix in the j-th layer, and we have:

[0102] H (j) =A (j) V (j) (8)

[0103] Let be the parameter matrix. For ease of representation, all bias terms are omitted from the formulas in this paper.

[0104] To incorporate PCG and CEC into Transformer modeling, GET employs an attention masking mechanism and a self-attention initialization mechanism.

[0105] Attention masking mechanism

[0106] GET uses PCG to improve the model by employing an attention masking (AM) mechanism. With AM introduced, the attention matrix is ​​calculated using the following formula:

[0107]

[0108] Where M∈R N×N Let e ​​be the attention mask matrix, if e = (v i ,v j If )∈E, then M ij =0, otherwise M ij Set to -∞. Introduce M. ij Afterwards, most irrelevant causal event pairs can be masked during model training, simplifying node connection structures, reducing training costs, and effectively mitigating the impact of noise.

[0109] Self-attention initialization mechanism

[0110] In the original Transformer, Q and K are randomly initialized in the first Transformer block. This causes the Transformer to tend to initialize the attention weights as a uniform distribution. When using the Transformer for graph modeling, this approach is not conducive to learning reasonable node relationships and also increases the model's training time.

[0111] To address this issue, GET formalizes the relevance of all event pairs into an asymmetric relevance matrix C∈R. N ×N C ij The attention weight of i to j is represented by the following formula:

[0112]

[0113] C is input into the Transformer as the attention for the first block, and masked attention is used in subsequent blocks. That is, in GET, the self-attention mechanism is as follows:

[0114]

[0115] Then, using a multi-head attention mechanism, the outputs of h attention heads are concatenated, and a feedforward network is used for further nonlinear processing, namely:

[0116]

[0117] This is the output after concatenating multiple headers. Unlike Transformer, GET does not use position encoding because the input event mentions are unordered.

[0118] loss function

[0119] The loss function for GET includes a graph reconstruction loss function and a regularization loss function for attention distribution shift.

[0120] Graph reconstruction loss function

[0121] The modeling goal of ECI is to determine whether a causal relationship exists between each pair of event nodes, i.e., to reconstruct the event causal graph. The final layer output of GET is used as the final node embedding matrix. Each row represents the output embedding of an event node. In the labeled data, determining whether a causal relationship exists between each pair of events is a three-class classification problem, where y∈{-1,0,1} corresponds to reverse causality, no causality, and positive causality, respectively. To construct the loss function, each event embedding is first concatenated, and its feature relationships are captured using a Multi-Layer Perception (MLP). Finally, the softmax function is used to obtain the predicted label probability p. ij :

[0122] p ij =softmax(MLP[h i h j ]),(13)

[0123] The order of splicing corresponds to the order of the event pairs to ensure that causal relationships representing two different directions can be distinguished. These correspond to the probabilities of reverse causality, no causality, and positive causality, respectively.

[0124] Finally, for each training text and its corresponding event mentions and labels, an adaptive focusing loss designed for label imbalance scenarios is used as the reconstruction loss function for the causal graph, i.e.:

[0125]

[0126] Where, α k (j=1,2,3) are the balance parameters, γ is the adjustment factor, and α satisfies j For any ∈(0,1), α1+α2+α3=1. Considering the equivalence of the importance of positive and negative causal relationships, we set α1=α2=α.

[0127] Regularization loss for attention distribution shift

[0128] GET uses the event relevance matrix C to initialize the attention in the first Transformer block, and then uses the masked attention matrix in subsequent training. The essence of the training process is to learn the optimal weights within a meaningful attention distribution space. Therefore, the goal of training should be to gradually improve C through fine-tuning on the ECI task, but... The deviation from C should not be too large.

[0129] To achieve the above objectives, GET uses a regularization loss based on attention distribution shift. It is defined as:

[0130]

[0131] D KL Let KL divergence be the operator. Then, the regularization loss function for each block is summed to obtain the final regularization loss. Finally, the reconstruction loss and the regularization loss are weighted and summed to obtain the overall training loss:

[0132]

[0133] λ is a hyperparameter that controls the weights of the regularization loss. The model is trained using the AdamW optimizer with L2 regularization.

[0134] To verify the superiority of GET in ECI tasks, this paper selects the Genecards dataset and compares the performance of GET with 10 current state-of-the-art ECI methods. Finally, the performance of the GET model on texts of different lengths is analyzed.

[0135] Comparative experiments were conducted using the Genecards dataset. Genecards is a comprehensive, searchable dataset of genes, providing information on almost all known human genes. Genecards automatically integrates resources from approximately 150 gene-centric databases, encompassing information from genomics, transcriptomics, proteomics, genetics, clinical, and functional fields.

[0136] Ten deep learning-based methods were selected as baseline models for comprehensive comparison and evaluation: four graph-based methods (RichGCN, LSIN, GESI, ERGO); three external knowledge enhancement methods (KnowDis, LearnDA, CauSeRL); and three PLM-based fine-tuning methods (Know-MMR, DPJL, KEPT). These methods are all state-of-the-art approaches proposed within the last three years.

[0137] Of these baseline models, RichGCN, GESI, and ERGO are applicable to both SECI and DECI tasks, while the other models are only applicable to the SECI task.

[0138] Precision (P), recall (R), and F1-score (F1) are chosen as evaluation metrics, and their definitions are as follows:

[0139] P = |C p ∩C l | / |C p |,(17)

[0140] R = |C p ∩C l | / |C l |,(18)

[0141] F1=2PR / (P+R),(19)

[0142] C p and C l Let P and R represent the sets of predicted and labeled causal pairs of events, respectively. P measures the false positive rate, R measures the false negative rate, and F1 is the harmonic mean of P and R, reflecting the overall performance of the model. Following the practice of previous research, only texts with at least one causal event pair are considered when evaluating the test set. Macro-F1 is used for evaluation, that is, P and R are calculated for each test sample, and then the average is taken to calculate F1, which is the reported model output.

[0143] On the Genecards dataset, evaluations were conducted by distinguishing between “Document-Level Causality” (DECI), “Sentence-Level Causality” (SECI), and “Overall Performance”.

[0144] (1) Environment configuration: Windows operating system, one NVIDIA GeForce RTX 3090Ti graphics card, Python version 3.10, PyTorch version 2.0.0, CUDA version 11.7.

[0145] (2) Model Selection: The main model for GET is implemented using the PyTorch version of the transformers package released by huggingface. To align with the embedding of Longformer, the embedding dimension is set to 1024. The openai package in Python is used to call the APIs of GPT-4, GPT-3.5, and GPT-3 to perform LLM-based PCG generation. The open-source longformer-large-4096 is used for Longformer, with the default parameters.

[0146] (3) Hyperparameter settings: The batch size for training was set to 16. The learning rate of the Transformer was set to 2e-5, with linear decay. Hyperparameters were optimized using a grid search method: the number of attention heads h was selected from {1,2,4,8}; the regularization weight parameter λ was selected from {0.1,0.2,0.3,0.4,0.5}; the number of FFN layers was 2; droppout was selected from {0.1,0.2,0.3}; and the baseline value b = 0.9. For LLM, the model parameter temperature was set to 0 to reduce output uncertainty. The parameters for the adaptive focusing loss were set to α = 0.4 and γ = 2.

[0147] For context-based learning prompt templates, both the paradigm of the prompt words and the number of examples affect the quality of the generated PCG. This invention selects the optimal "causal relationship" question paradigm and the example quantity of "4 positive examples + 2 negative examples" for experiments.

[0148] In the training set, all causal labels are known. Considering that the ratio of causal event pairs generated by PCG to real causal event pairs is between 20 and 50, and to control the number of spurious causal samples, during model training, spurious causal pairs are constructed by random selection and replacement at a ratio of 10 times that of real causal pairs, and together with real causal pairs, they constitute the input PCG.

[0149] After optimizing the hyperparameters by dividing the training set into a 15% validation set, the following hyperparameters were selected for the test set: attention head number h = 8, regularization weight λ = 0.2, and dropput probability of 0.2.

[0150] Table 1 compares the SECI performance metrics of six baseline models, two versions of LLM, and GET on the ESC / CTB dataset. Table 2 compares the SECI, DECI, and overall performance metrics of three baseline models and GET on the ESC dataset. For clarity, all evaluation values ​​in the tables are obtained by multiplying the original results by 100. Bold numbers in the tables represent the best metric values ​​among the experimental models, underlined numbers represent the second-best metric values, and * represents the third-best metric values. " / " in the tables indicates that the source literature for this method did not report relevant metrics and did not open-source the code for reproduction.

[0151] First, a horizontal comparison of the GET performance of PCG generation based on different LLMs shows that in both the SECI and DECI tasks, the performance ranking is gpt4 > gpt3.5 > davinci-002 > davinci-003. The precision of the four GET models is similar, but the recall differs significantly, and this difference in recall directly leads to the difference in overall performance (F1 score).

[0152] Table 1. Comparison of SECI performance of GET with 6 baseline models and 2 versions of the large language model.

[0153]

[0154]

[0155] Table 2 compares the SECI, DECI, and overall performance of GET with three baseline models on the Genecards dataset.

[0156]

[0157] As shown in Table 1, in the SECI task, GET+gpt4 ranked 3rd in F1 score, second only to ERGO and DPJL. Apart from the four LLMs, GET+gpt4 ranked first in recall, and GET+gpt3.5 ranked second, showing significant improvement compared to other baseline models: on the Genecards dataset, GET+gpt4 and GET+gpt3.5 outperformed the second-place ERGO by 2.8% and 1.2%, respectively.

[0158] As shown in Table 2, in the DECI task of the Genecards dataset, GET+gpt4 ranked first in F1 score (52.5%) compared to the baseline model, with an F1 score improvement of 1.4% compared to the second-place GESI. It also ranked second in recall, exceeding the third-place RichGCN by 9.6%. GET+gpt4's overall evaluation metrics, F1 score and recall, were both higher than the baseline model, exceeding the second-place model by 1.6% (ERGO) and 8.2% (RichGCN), respectively. GET+gpt3.5 achieved F1 scores of 51.6% and recall of 60.2%, ranking first in recall, exceeding the second-place model (RichGCN) by 8.9%.

[0159] Comparing GET models based on different LLMs, we found that the precision difference when LLMs are used individually for ECI tasks has a small impact on GET performance, but the recall difference has a significant impact. The reasons for this are as follows: First, the precision of GET models built with PCG enhancements using these LLMs is not significantly different. This is because the GET model training does not use the PCG generated by the LLMs. Therefore, false positives (spurious causality) on the test set are only related to a small number of event pairs generated in the PCG input to the test set that are "easily misidentified," and are not directly affected by the false positives of the LLMs themselves. Conversely, the recall value of LLMs used directly for ECI tasks reflects the number of true causal pairs in the PCGs identified and generated by the LLMs. Those true causal pairs not in the PCGs (edges not initialized by the PCGs) will never be identified. Therefore, the recall performance difference of LLMs has a significant impact on the recall value of the GET model.

[0160] In the SECI task, the GET model has a certain advantage in recall compared to all baseline methods (except LLM). This is because GET performs graph inference on the PCG generated by LLM, and the simpler initial graph structure effectively reduces noise during inference, thereby improving the ability to identify causal relationships. Another reason is that since LLM identifies almost all labeled causal pairs while filtering out a large number of non-causal pairs, the training sample labels are more balanced, which is beneficial for the model to learn causal and non-causal patterns.

[0161] To verify the above analytical conclusions, 10 documents containing at least two labeled causal pairs were selected from the Genecards dataset. The ratio of non-causal event pairs to causal event pairs was calculated after initialization using a fully connected graph and PCG. The results are as follows: Figure 3 As shown in the figure, the statistical results demonstrate that PCG has a significant effect on balancing labels, reducing negative samples (non-causal pairs) by nearly 90%.

[0162] In the SECI task, none of the four GET models achieved the best F1 score, performing worse than the ERGO and DPJL models, primarily due to the lower precision of the GET models. This is because GET only considers the context during the initial embedding, at which point intra-sentence events are highly relevant. However, during model training, the semantic background of the context no longer plays a role. Limited by self-attention initialization and attention distribution shift loss, some intra-sentence event pairs maintain a consistently high degree of correlation, leading the model to identify more spurious intra-sentence causal event pairs.

[0163] In the DECI task, the recall metrics and F1 scores of the four GET models significantly outperformed the baseline methods, with the advantage in recall being particularly pronounced. Analysis suggests that, besides PCG providing most of the correct document-level causal structures, experiments also revealed that the vast majority of causal events have higher (mutual) attention weights compared to non-causal event pairs. Although this difference is not significant initially, it is amplified after training, making it easier to identify correct causal event pairs given PCG as prior knowledge.

[0164] To verify the above analysis, 50 labeled document-level causal event pairs were randomly sampled from Genecard's training set, and the average ranking of attention weights was recorded as the number of training iterations increased. For each event pair (A, B), the attention weights for "B to A" and "A to B" were recorded separately. The attention weight ranking for B to A refers to the ranking of the attention weight between B and A, which have a causal relationship with a specific event A, among all the attention weights between A and all events (MA) associated with A. The higher this ranking, the more likely the model is to identify the correct causal relationship, i.e., the stronger the model's performance.

[0165] Initially, the attention weights for most genuine causal event pairs ranked in the top 7 of all weights. However, as training progressed, their average ranking gradually decreased, eventually stabilizing at around 3.5. This validates the effectiveness of using initial attention and regularized loss to guide training. However, from another perspective, the way relevance is calculated leads to a small number of spurious causal event pairs having high relevance, resulting in a relatively poor precision metric for the GET model and limiting its overall performance.

[0166] To verify the performance of GET in handling document-level event causality relationships in long texts, the text on the Genecards test set was divided into four intervals according to length: [0, 256], [256, 512], [512, 1024], and [1024, 2048]. The performance of GET was compared with that of the state-of-the-art graph-based ECI model ERGO (open source) in each interval. The results are as follows: Figure 4 As shown. From Figure 4 As can be seen, GET performs slightly worse than ERGO in shorter texts, but as the text length increases, ERGO's performance drops significantly, while GET's performance drops relatively less. This verifies that the GET model is more robust to text length in the DECI task and has the ability to identify causal relationships between events in longer texts.

[0167] This invention proposes a graph-augmented Transformer model for event causality recognition. Addressing the challenges of difficult context representation and noisy data in existing methods, this model first utilizes a large language model to construct a prior causal graph, reducing noise and balancing data introduced during graph modeling. Then, a Longformer is used to generate contextualized event representations and relevance matrices, acquiring initial event representations and prior attention knowledge. Next, attention masking and self-attention initialization mechanisms are introduced into the Transformer, integrating the prior causal graph and attention knowledge into the model to guide training. Finally, reconstruction loss and attention distribution shift regularization loss are designed to train the Transformer. Experimental results on two public datasets demonstrate that the proposed model exhibits performance advantages in event causality recognition tasks, particularly in document-level causality and long-text event causality recognition.

[0168] As used herein, the term "preferred" is meant as an example, illustration, or illustration. Any aspect or design described herein as "preferred" need not be construed as being more advantageous than other aspects or designs. Rather, the use of the term "preferred" is intended to present the concept in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusionary "or." That is, unless otherwise specified or clear from the context, "X uses A or B" naturally includes either of the permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0169] Furthermore, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components (e.g., elements, etc.), the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure performing the functions in the exemplary implementations of this disclosure shown herein. Moreover, although specific features of this disclosure have been disclosed with respect to only one of several implementations, such features may be combined with one or more features of other implementations that may be desirable and advantageous for a given or particular application. Furthermore, with regard to the use of the terms “comprising,” “having,” “containing,” or variations thereof in the Detailed Description or claims, such terms are intended to be included in a manner similar to the term “including.”

[0170] The functional units in this invention embodiment can be integrated into a processing module, or each unit can exist physically separately, or multiple units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. The aforementioned devices or systems can execute the storage methods in the corresponding method embodiments.

[0171] In summary, the above embodiments are one implementation of the present invention, but the implementation of the present invention is not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made that deviate from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A document-level event causal relationship recognition method based on graph reinforcement learning, characterized in that, Includes the following steps: Retrieve text data T containing the event from the gene database; By using a large language model as an external knowledge base and inference engine, a prior causal graph is generated to obtain an initial graph structure, thereby reducing noise and balancing labels. Utilize long-text pre-trained language models to obtain contextualized event representations and calculate the relevance of event pairs, providing prior knowledge; Attention masks and self-attention initialization mechanisms are introduced, and Longformer is used to generate contextualized representations of events. Prior causal graphs and contextualized event relevance are integrated into the Transformer model to guide and control the training of the model. The model is trained and optimized using graph reconstruction loss and attention distribution shift regularization loss to extract causal relationships between events; Output the causal relationships between events identified in the gene database; Specifically, an end-to-end prior causal graph (PCG) is constructed using a large language model based on prompting learning; the construction process is as follows: First, a prompt template is designed for LLM, and examples are provided for contextual learning. Then, the mind chain technique is used to guide LLM to reason step by step and determine the causal relationship between each pair of events. Specifically, in a single question-and-answer session with an LLM, the prompt words will be... Input into the large model, The format is as follows: in, and This is a set of positive and negative reasoning examples, each containing three elements: <question, reasoning process, answer>. and For text and target event pairs, A standard format for questions related to causality identification; Then, The input is fed into the LLM to generate normalized output: if So add an edge arrive In the middle, and vice versa Then no edges are added, nodes A mention corresponding to an event .

2. The document-level event causal relationship recognition method based on graph reinforcement learning according to claim 1, characterized in that, Use a priori cause-effect graph (PCG) to limit the number of connections between input event nodes; for text data and the set of events mentioned within it , The number of times the event is mentioned; the prior cause-effect graph is... ,in Represents a set of nodes, each node A mention corresponding to an event ; This represents a set of edges, where each edge corresponds to a pair of nodes. ,express and A causal relationship exists; the prior causal graph is a directed acyclic graph, and the edge information in the graph is used to initialize the connection relationships between event mentions of the input Transformer.

3. The document-level event causal relationship recognition method based on graph reinforcement learning according to claim 2, characterized in that, The contextualized representation of the Longformer generated event includes: For each text And the event mention set M, add start and end markers to the text. and Add a special tag to each event mention. and To indicate the location of the event mention in the text; Use a word segmenter to divide text data Divide into multiple markers ,in , Then, use Longformer to encode it into the corresponding embedding vector: Get the tag corresponding to each event mention The embedding vector is concatenated with the overall embedding vector of the text to serve as the contextual embedding for the event mention: Indicates the first Each event mentions the starting index of the corresponding marker, and ";" indicates vector concatenation; For all event mention embeddings, cosine similarity is used to calculate contextualized event relevance: To avoid the semantic gap reduction problem caused by using high-dimensional vectors to calculate cosine similarity, the baseline value is subtracted from the generated relevance value. To increase its discriminative power; in addition to calculating relevance, It is input into the Transformer as the initial embedding vector for events in the model.

4. The document-level event causal relationship recognition method based on graph reinforcement learning according to claim 3, characterized in that, The attention masking mechanism includes: The attention matrix is ​​calculated using the following formula: in, For the mask attention weight matrix, For the embedding dimension of the key, and It is an embedding vector of the query and key after linear transformation. Let be the attention mask matrix, if ,So Otherwise Set as ; Introduction This simplifies the node connection structure by masking irrelevant causal event pairs during model training.

5. The document-level event causal relationship recognition method based on graph reinforcement learning according to claim 4, characterized in that, The self-attention initialization mechanism formalizes the relevance of all event pairs into an asymmetric relevance matrix. To avoid initializing attention weights to a uniform distribution, where express right The attention weights are calculated using the following formula: The attention input into the Transformer is used as the first block's attention, and subsequent blocks use a masked attention mechanism, i.e., a self-attention mechanism, as follows: H (0) H is the initial embedding representation of the model. (1) This represents the embedding representation of the model in layer 1. For the model in the first Embedded representation in layers, H (j-1) For the model in the first Embedded representation in layer -1 This is the parameter matrix for the first layer. Let j be the parameter matrix of the j-th layer. It is a masked attention matrix The j-th row; Then, using a multi-head attention mechanism, the pieces are stitched together. The output of each attention head is then further processed nonlinearly using a feedforward network, namely: The output is the result of concatenating multiple heads. FFNN represents a feedforward network, LeakyReLU is the activation function of the neural network unit, and W1 and W2 are learnable parameter matrices.

6. The document-level event causal relationship recognition method based on graph reinforcement learning according to claim 5, characterized in that, To construct the graph reconstruction loss function, each event embedding is first concatenated, and its feature relationships are captured using a multilayer perceptron (MLP). Then use the softmax function to obtain the predicted label probability. : h i and h j These are the node embedding matrices. The i-th and j-th elements, d model It is the embedding dimension of the model; The order of splicing corresponds to the order of the event pairs to ensure that causal relationships representing two different directions can be distinguished. These correspond to the probabilities of reverse causality, no causality, and positive causality, respectively. Finally, for each training text and its corresponding event mentions and labels, an adaptive focusing loss designed for label imbalance scenarios is used as the reconstruction loss function for the causal graph, i.e.: in, It is a balance parameter. It is an adjustment factor that satisfies... , It is the probability that the i-th node and the j-th node have the k-th relationship, where k=1,2,3 represent reverse causal relationship, no causal relationship, and positive causal relationship, respectively.

7. The document-level event causal relationship recognition method based on graph reinforcement learning according to claim 6, characterized in that, The regularization loss for the attention distribution shift is fine-tuned to progressively improve the event relevance matrix. However, this makes the mask attention matrix... Event Relevance Matrix The deviation is within a set threshold; the regularization loss of the attention distribution shift is defined as: Denotes the KL divergence operator, and It is a masked attention matrix The j-th and j-1-th rows; The summation of the regularization loss functions for each block is used as the final regularization loss.

8. The document-level event causal relationship recognition method based on graph reinforcement learning according to claim 7, characterized in that, The overall training loss is obtained by weighted summing the graph reconstruction loss function and the regularization loss of the attention distribution shift: These are hyperparameters that control the weights of the regularization loss; the model is trained using the AdamW optimizer with L2 regularization.

Images