Fusing coreference relations and external knowledge enhanced causal inference method

Abstract

The application relates to the technical field of natural language processing, and proposes a causal inference method fusing co-reference relations and external knowledge enhancement, which comprises the following steps: extracting event words in a text as mask target words and performing a mask operation on non-event words, using an improved BERT model to predict the masked event words by context reconstruction in a pre-training stage; extracting co-reference event pairs of cause events and result events to obtain co-reference features of the co-reference events; extracting potential indirect events associated with the cause / result events from an external knowledge base to construct an external knowledge base causal graph, constructing a structured semantic graph, and performing node feature fusion; performing nonlinear transformation on the co-reference features and the fused node features; fusing the context semantic features and the external knowledge base features to obtain comprehensive feature representation; and inputting the comprehensive feature representation into a prediction layer to obtain the final prediction result of the event pair relation.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of natural language processing, and in particular to a causal inference method fusing co-reference relations and external knowledge enhancement. BACKGROUND

[0002] Event Causality Identification (ECI) is a key and challenging task in the field of natural language processing (NLP), aiming to identify the causal relationship between events from text, which is of great significance for understanding the deep semantics of text, constructing knowledge graph, question and answer system, text summarization and other applications.

[0003] Document-Level Event Causality Identification (D-ECI) is a highly challenging task in the field of natural language processing. The construction of document-level event relationships usually relies on external knowledge base to enhance the coherence of causal chain, but existing methods often only take the target event that needs to be identified as the starting point and ending point of the causal chain, which not only limits the depth of utilization of external knowledge base, but also weakens the full mining of text semantic features. This limitation leads to the construction of causal chain that cannot fully reflect the complex logical relationship between events, and cannot maximize the utilization of potential information of text and knowledge base. SUMMARY

[0004] In view of the above defects, the purpose of the application is to provide a causal inference method fusing co-reference relations and external knowledge enhancement, aiming to realize accurate semantic mapping of external knowledge and document events, and improve the accuracy and robustness of causal relationship identification.

[0005] To achieve this purpose, the application adopts the following technical solutions:

[0006] The causal inference method fusing co-reference relations and external knowledge enhancement comprises the following steps:

[0007] S1: Extract event words in the text of the document level as mask target words and randomly mask non-event words, and use an improved BERT model to predict the masked event words by context reconstruction in the pre-training stage, so that the event words fully learn the context semantic features;

[0008] S2: Randomly extract event pairs from the text and label them as cause events and result events. Extract the co-referenced event pairs of cause events and result events. Obtain the labeled positive co-referenced event samples from an external knowledge base. At the same time, generate non-co-referenced event pairs as negative samples. Use a contrastive learning mechanism to design a contrastive loss function to narrow the semantic distance between co-referenced event pairs and obtain the co-referenced features of co-referenced events.

[0009] S3: Extract potential indirect events associated with cause / effect events from an external knowledge base to construct an external knowledge base causal graph. Construct a structured semantic graph with events as nodes and causal relationships as edges from the external knowledge base causal graph. Use graph attention mechanism and graph convolutional neural network to fuse node features. The potential indirect events include indirect events between cause events and result events.

[0010] S4: Fuse the coreference features from step S2 and the fused node features from step S3, and perform nonlinear transformation through a fully connected layer to obtain the external knowledge base features of the cause event and the result event.

[0011] S5: A multi-head attention mechanism is used to fuse the contextual semantic features and external knowledge base features to obtain a comprehensive feature representation. The comprehensive feature representation is then input into the prediction layer to obtain the final prediction result of the event pair relationship.

[0012] Preferably, in step S1, the following relationship is satisfied:

[0013]

[0014] Where X = [x1, x2, ..., x i ,…x N ] represents the word sequence of the input text, x i Let N represent the i-th word, E represent the set of event words, and N represent the set of event words. text P represents the length of the word sequence in the text. mask (x i ) represents the probability of the i-th word being masked, where α and β are the masking probabilities. This represents the contextual semantic features of the i-th word, i.e., the embedding vector of the event word.

[0015] Preferably, step S2 includes:

[0016] By designing a contrastive loss function using a contrastive learning strategy, the semantic distance between causal events and their co-referenced events, and between result events and their co-referenced events, is narrowed, while the semantic distance between non-co-referenced event pairs is widened.

[0017] Construct positive sample coreference pairs (e) using external knowledge bases and text events. i ,e j)∈C, randomly combined from non-coreference events The projection layer is introduced to map events to a new semantic space, and the cosine similarity is used to calculate the context relevance of coreference events, satisfying the relationship:

[0018]

[0019] where W and b represent the weight matrix and bias of the projection layer, respectively, represents the context semantic feature of the ith word, z i represents the projection vector of the embedding vector h i . is the loss function of contrastive training, s(z i , z j ) represents the similarity between z i and z j , and τ represents the temperature coefficient.

[0020] Preferably, in step S3, potential indirect events associated with cause / effect events are extracted from an external knowledge base, and entity linking and relationship extraction techniques are used to identify and extract causal relationships between events;

[0021] The causal relationship is serialized and then encoded using an EMBert model with parameter freezing, satisfying the relationship:

[0022] Input=[CLS]e0[indirect]e1[indirect]…[indirect]e n [effect];

[0023]

[0024] where Input represents the input sequence, [indirect] represents the insertion symbol of indirect causal relationship between events, [effect] represents the final result event of the causal chain, represents the embedding vector of event e i in sequence k.

[0025] Preferably, in step S3, the embedding vector of the insertion symbol between event nodes of the causal graph is calculated and combined with the attention weight to fuse the weights of the same nodes and edges in different sequences, satisfying the relationship:

[0026]

[0027]

[0028] where, denotes the event node e in sequence k i to event node e j the weight W g is a learnable matrix, denotes the event node e in sequence k i and e j the attention weight v T is a learnable vector, is the weight between global event nodes.

[0029] Preferably, in step S3, it includes adopting mean pooling to fuse the features of the same event nodes in multiple different sequences into unified global features, satisfying the relationship:

[0030]

[0031] wherein denotes the embedding vector of event e i in the sequence, denotes the number of sequences with the same event node e i , and is the global feature representation of each sequence event node.

[0032] Preferably, in step S3, it includes calculating the attention coefficient between each node and its adjacent node according to the graph attention mechanism, fusing the features of the adjacent nodes, while adopting the graph convolutional neural network to fuse the edge weight information into the node features, satisfying the relationship:

[0033]

[0034] wherein, denotes the attention coefficient between adjacent nodes, A is the attention coefficient matrix, H (0) denotes the node feature of the initial layer, W GAT denotes a learnable feature transformation matrix, H (1) denotes the feature fused by the first layer of graph attention mechanism, H (2) denotes the feature fused by the second layer of graph convolutional neural network, denotes the degree matrix, denotes the normalized adjacent matrix of the graph, W GCN denotes the weight between global event nodes, each element is

[0035] Preferably, step S4 satisfies the relationship:

[0036]

[0037] f = [y1; y2;...; y n ];

[0038] wherein denotes the node feature of the structured semantic graph, z i denotes the coreference feature of the event node, y i denotes the fusion result of the node feature and the coreference feature of the structured semantic graph, f denotes the feature splicing vector of each coreference event of the cause event or the result event.

[0039] Preferably, step S5 satisfies the relationship:

[0040]

[0041] M = [head1;... ; head h ]W O ;

[0042] wherein, C cause and C effect denote the spliced cause event and result event feature vectors respectively, W i Q , denote the weight matrices of Q, K, and V respectively, W O is the weight matrix of the multi-head attention output mapping, M is the comprehensive feature representation of the comprehensive cause event and result event, denotes the context semantic feature of the cause event in the text, denotes the context semantic feature of the result event in the text, f cause denotes the external knowledge base feature of the cause event, f effect denotes the external knowledge base feature of the result event.

[0043] Preferably, step S5 includes inputting the comprehensive feature representation into a prediction layer, determining whether a cause-effect relationship exists between the event pair, calculating the loss thereof, and obtaining the final prediction result of the event pair relationship, satisfying the relationship:

[0044] p i = sigmoid(WM + b);

[0045]

[0046] wherein, p i denotes the prediction probability, y i denotes the true label, N denotes the sample number of the event pair, denotes the binary cross-entropy loss function, is the comprehensive loss, M is the comprehensive feature representation of the comprehensive cause event and result event, b denotes the bias of the projection layer, and λ denotes the loss function weight coefficient, denotes the loss function of contrast training.

[0047] One of the above technical solutions has the following advantages or beneficial effects:

[0048] The application optimizes the pre-training process through the event mask strategy, improves the parsing accuracy of event semantics, and helps to enhance the extraction efficiency of event semantic features in the context environment; the contrast learning mechanism is used to narrow the semantic distance of the coreference event pair, effectively capture the coreference relationship between events, improve the deep mining ability of the model to the text semantics, and deeply consider the relevance of the coreference event to the target event pair in the text context. Further, by integrating the coreference event nodes in the causal graph constructed in the external knowledge base, and incorporating the external knowledge associated with these coreference events into the causal inference process, the semantic information of the causal graph can be enriched, and the comprehensive analysis ability of the event causal relationship can be significantly improved by strengthening the logical reasoning between events; the coreference feature and the external knowledge base feature are fused through full connection layer nonlinear transformation to obtain a more comprehensive external knowledge base feature representation; a multi-head attention mechanism is used to fuse the context semantic features and the external knowledge base features to generate high-quality comprehensive feature representation, and improve the accuracy of causal relationship prediction. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0050] Figure 1 is a flowchart of the causal inference method enhanced by fusing coreference relationship and external knowledge provided by the embodiment of the present application;

[0051] Figure 2 is an implementation framework diagram of the causal inference method enhanced by fusing coreference relationship and external knowledge provided by the embodiment of the present application;

[0052] Figure 3 is a schematic diagram of extracting causal chains and constructing coreference samples by the causal inference method enhanced by fusing coreference relationship and external knowledge provided by the embodiment of the present application;

[0053] Figure 4 is a schematic diagram of fusing coreference features and causal graph node features of the causal inference method enhanced by fusing coreference relationship and external knowledge provided by the embodiment of the present application. DETAILED DESCRIPTION

[0054] Embodiments of the present application are described below in detail with reference to the accompanying drawings, wherein the same or similar components are denoted by the same or similar reference numerals throughout the drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be understood as limiting the present application.

[0055] In the description of the present application, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "axial", "radial", "circumferential" and the like are based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0056] In addition, the terms "first" and "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features referred to. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0057] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be a fixed connection, or a detachable connection, or an integral connection; it can be directly connected, or indirectly connected through an intermediate medium, or the internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0058] The fusion of co-reference relationship and external knowledge enhanced causal inference method, as shown in Figure 1 A preferred embodiment of the present application, the causal inference method proposed by the present application can be executed by a corresponding causal inference model or system, and the causal inference method comprises the following steps:

[0059] S1: Extract event words in the text at the document level as mask target words and randomly mask non-event words, and use an improved BERT model to predict the masked event words by context reconstruction in the pre-training stage, so that the event words fully learn the context semantic features;

[0060] Specifically, in the field of natural language processing, event words, as the carriers of key actions or state information in text, contain rich semantic information and contextual associations, while document-level text often has complex semantic structures and long dependencies. Step S1 preprocesses the document-level text, first extracts event words from the text and masks them as target words, and randomly masks non-event words. The improved BERT model is used to predict the masked event words by reconstructing the context in the pre-training stage, which enables the event words to fully learn the context semantic features on a large scale of unsupervised data, thereby obtaining more accurate context embedding representation, which aims to improve the understanding of event semantics and context awareness, laying a solid foundation for subsequent causal inference tasks and enhancing the effectiveness of capturing potential causal relationships between events in text.

[0061] Among them, the improved BERT model is a deep language model optimized on the basis of the original BERT architecture to solve the problems of traditional BERT models (inaccurate semantic feature extraction of event words, difficulty in fully capturing the logical association between event words), with stronger context encoding and semantic understanding capabilities, thus requiring the use of an improved BERT model;

[0062] S2: Randomly extract event pairs in the text and label them as cause events and result events, extract co-reference event pairs between cause events and result events, obtain labeled co-reference event positive samples from an external knowledge base, and generate non-co-reference event pairs as negative samples. A contrast learning mechanism is used to design a contrast loss function to narrow the semantic distance between co-reference event pairs, obtaining co-reference features of co-reference events;

[0063] In text data, the causal relationship between event pairs is often implicit and complex and variable, and co-reference events, as different expressions of the same event at different positions in the text, contain rich semantic associations and contextual clues, which can provide strong support for causal inference. Step S2 randomly extracts event pairs in the text and labels them as cause events and result events, extracts co-reference event pairs between them, and obtains co-reference event positive samples from an external knowledge base and generates non-co-reference event pairs as negative samples. A contrast learning mechanism is used to design a contrast loss function to narrow the semantic distance between co-reference event pairs, thereby obtaining co-reference features of co-reference events. The purpose is to mine the co-reference information contained in the text, strengthen the understanding and modeling of semantic associations between co-reference events, so that the potential causal logic between event pairs can be more accurately captured, the accuracy of causal inference can be improved, and the effectiveness of deep mining and utilization of text semantics can be enhanced.

[0064] Among them, the event pair is a combination of two events with potential causal association in the text, which is the core analysis object of the causal inference task; the cause event and the result event respectively represent the events that cause and are caused by the results in the event pair, and clear labeling of the two helps to focus on the causal logic direction; the co-reference event pair refers to different forms of expression of the same event in the text, which have high consistency in semantics. By extracting co-reference event pairs, the understanding of event semantics can be enriched, and multi-angle clues can be provided for causal inference; the external knowledge base is a structured data set that stores a large amount of domain knowledge or common sense knowledge, which contains rich co-reference event annotation samples and causal relationship information, which can be used to supplement the missing semantic information in the text and enhance the knowledge coverage; the contrast learning mechanism is an effective machine learning strategy, which makes similar samples closer in semantic space and dissimilar samples farther away through the design of contrast loss function, thereby strengthening the discriminant ability of sample features; the contrast loss function is a key component in contrast learning, which is used to measure the semantic similarity difference between sample pairs and guide parameter optimization, promote the convergence of co-reference event pair semantic embedding, and enable more accurate distinction between co-reference and non-co-reference event pairs.

[0065] S3: Extracting potential indirect events associated with cause / result events from the external knowledge base to construct an external knowledge base causal graph, constructing a structured semantic graph with events in the external knowledge base causal graph as nodes and causal relationships as edges, and using graph attention mechanism and graph convolutional neural network for node feature fusion, the potential indirect events include indirect events between cause events and result events;

[0066] The external knowledge base contains a large amount of domain knowledge and causal relationship information, which complements each other with the events in the text, and can provide a more comprehensive semantic perspective and knowledge background for causal inference. The purpose of step S3 is to integrate the causal relationship links in the external knowledge base, form a structured causal knowledge representation, fully mine the semantic information in the external knowledge base through the feature fusion ability of the graph neural network, and effectively integrate it into the modeling process of the target event pair, improve the knowledge depth and logical coherence of causal inference, and enhance the reasoning ability of complex causal logic between events.

[0067] The potential indirect event refers to an intermediate event having an indirect causal correlation with the cause event and the result event, which plays a key role in bridging the causal link and can expand the depth and complexity of the causal relationship. The causal graph is a semantic graph structure taking events as nodes and causal relationships as edges, which directly shows the causal logic and correlation path between events, and is an important basis for knowledge fusion and reasoning. The graph attention mechanism is a graph neural network technology that dynamically aggregates adjacent node features by calculating the attention coefficients between nodes, highlights the influence of important nodes on the target node, and improves the relevance and effectiveness of feature fusion. The graph convolutional neural network is a deep learning based on graph structure data, which can effectively fuse node features and edge weight information, perform feature learning and information propagation of graph structure data, and make full use of the structured semantic information of the causal graph.

[0068] S4: The co-reference features in step S2 and the fused node features in step S3 are fused, and a fully connected layer is used for nonlinear transformation to obtain external knowledge base features of the cause event and the result event;

[0069] The co-reference features are semantic features extracted from the co-reference event pair in the text through a contrast learning mechanism, which reflects the semantic consistency correlation of the event in the text and provides clues for the event semantics in the text. The node features are features fused from the external knowledge base causal graph through a graph neural network, which contain event semantics and causal relationship information in the external knowledge base, and can supplement rich background knowledge. The fully connected layer is a basic component of a neural network, which realizes nonlinear transformation and fusion of features through a fully connected structure of neurons, can learn complex correlations and interaction patterns between features, and makes the fused features more discriminative and expressive. The external knowledge base features are feature representations after fusion, which integrate the information of co-reference features and external knowledge base node features, and are an important feature basis for subsequent causal inference tasks, which determines the comprehensive understanding of event pair semantics and knowledge by the causal inference method.

[0070] S5: The context semantic features and the external knowledge base features are fused by using a multi-head attention mechanism to obtain comprehensive feature representations, and the comprehensive feature representations are input into a prediction layer to obtain the final prediction result of the event pair relationship.

[0071] In causal inference models, contextual semantic features and external knowledge base features provide rich feature descriptions for event pairs from the perspectives of semantic information within the text and external knowledge, respectively, but they emphasize different semantic levels and knowledge dimensions. A multi-head attention mechanism is employed to fuse contextual semantic features and external knowledge base features, resulting in a comprehensive feature representation. This mechanism captures multi-dimensional relationships and interactions between features, highlighting the impact of key information on causal inference. The comprehensive feature representation is then input into the prediction layer, where classification or regression calculations yield the final prediction result of the event pair relationship. The aim is to integrate the advantages of internal textual semantics and external knowledge to form a more discriminative and interpretable feature representation, enhancing the causal inference model's comprehensive judgment ability regarding the causal relationship between event pairs. This provides accurate prediction results for the final causal inference, meeting the accuracy and reliability requirements for causal relationship identification in practical applications.

[0072] Among them, contextual semantic features refer to the contextual semantic information of event pairs extracted from text through pre-trained models (such as improved BERT), reflecting the lexical, grammatical, and semantic environment features of events in the text, and are the key basis for causal inference models to understand the internal semantics of text; external knowledge base features refer to the features after fusion in step S4, covering coreference features and event semantics and causal relationship information in the external knowledge base, providing causal inference models with knowledge supplementation and logical support outside the text; multi-head attention mechanism is a variant of attention model, which can capture different semantic levels and interaction patterns between features by calculating the association between features in parallel through multiple attention heads, thereby improving the effect of feature fusion and the expressive power of features; comprehensive feature representation is the final feature form after fusing contextual semantic features and external knowledge base features, including comprehensive information of internal and external knowledge, and is the direct feature basis for causal inference models to perform causal inference; prediction layer is the output layer of the model, which may include components such as classifiers or regressors, used to calculate the probability or strength of causal relationship between event pairs based on comprehensive feature representation, and output the final prediction result, which is the key link for causal inference models to transform features into practical application value.

[0073] Preferably, in step S1, the following relationship is satisfied:

[0074]

[0075] Where X = [x1, x2, ..., x i ,…x N ] represents the word sequence of the input text, x i Let N represent the i-th word, E represent the set of event words, and N represent the set of event words. text P represents the length of the word sequence in the text. mask (x i) represents the probability of the i-th word being masked, and a and b are the mask probabilities, represents the context semantic feature of the i-th word, i.e., the embedding vector of the event word.

[0076] Specifically, first, the event words are extracted from the text as the mask target words, and the non-event words are randomly masked. In this process, an improved BERT model (EM-Bert) is used for context reconstruction training to predict the masked event words, which can fully learn the context semantic features and obtain more accurate context embedding representation.

[0077] In the embodiment, as shown in Figure 2 The obtained document-level text is: Kenneth Dorsey says the woman accused of killing two co-workers and critically injuring a third at the Kraft plant in Northeast Philly is a good person. And so were the two women she’s accused of gunning down with a.357 Magnum. The causal inference method strictly controls the proportion of mask words ([MASK]) to be 15% when processing the input text, extracts event keywords through a label recognition mechanism and masks them, and uses a random masking strategy to process the remaining non-event words. In this embodiment, the event words are killing, accused, injuring, and gunning down. These words are replaced with [MASK] in the pre-training process. The priority masking strategy of event words enables the model to focus on the logical association between events and enhances the extraction of semantic features of event words from the context. The random masking of non-event words enhances the model’s generalization ability to text semantics while maintaining the masking proportion.

[0078] Preferably, in step S2, the following steps are included:

[0079] By designing a contrast loss function through a contrast learning strategy, the semantic distance between the cause event and its co-reference event, the result event and its co-reference event is narrowed, while the semantic distance between non-co-reference event pairs is widened.

[0080] By constructing a positive sample co-reference pair (e i ,e j )∈C, from non-co-reference events, randomly combining A projection layer is introduced to map events to a new semantic space, and cosine similarity is used to calculate the relevance of core-referenced events to the context, satisfying the following relation:

[0081]

[0082] Where W and b represent the weight matrix and bias of the projection layer, respectively. z represents the contextual semantic features of the i-th word. i Represents the embedding vector h i The projection vector, To compare the loss function used in training, s(z) i ,s j ) represents z i With z j The similarity between them, where τ represents the temperature coefficient.

[0083] Positive sample coreference pairs are coreference event pairs constructed from external knowledge bases and original text events, such as (e i ,e j )∈C, where e i and e j These are co-referenced events, and negative sample pairs are event pairs randomly combined from non-co-referenced events, for example... Where e i e k Instead of coreference events, the projection layer is a neural network layer that maps events to a new semantic space, using linear transformations and nonlinear activations through a weight matrix W and a bias b.

[0084] like Figure 3 The lower half (constructing coreference sample pairs based on an external knowledge base) is shown below. First, the events in the text are used to construct target event pairs (e i ,e k The event is divided into causal events and result events. The killing-injuring pair is selected as the target event pair. Then, the external knowledge base MRVEN-ERE is introduced. Events with coreference relationships are grouped as positive coreference pairs, while events without coreference relationships are randomly grouped to obtain negative coreference pairs. The coreference events for killing are murdering and demonstrate, and the coreference events for injuring are harming and damage. When a projection layer is introduced, the events are mapped to a new semantic space. The contextual semantic features of the events are obtained through the weight matrix W and bias b of the projection layer. It is transformed into a new projection vector z i , To embed vectors, the linearly transformed features are non-linearly processed using a sigmoid activation function. This mapping can enhance the model's ability to represent semantic features and improve its discriminative performance. When calculating similarity, cosine similarity is used to calculate the similarity between co-reference event pairs, such as calculating the cosine similarity of z i 、z j to evaluate their semantic relevance. Cosine similarity measures can more accurately reflect the directional similarity between vectors, improving the accuracy of similarity calculation. Subsequently, the semantic distance of co-reference event pairs is minimized through a contrastive loss function, while the semantic distance of non-co-reference event pairs is maximized. For example, using a temperature coefficient τ to control the distribution of similarity, the introduction of the contrastive loss function can effectively optimize the training process of the causal inference model, enabling the model to better learn the semantic relationships between events and enhancing its generalization ability when processing complex texts.

[0085] Preferably, in step S3, potential indirect events associated with the cause / effect events are extracted from an external knowledge base, and entity linking and relation extraction techniques are used to identify and extract causal relationships between events;

[0086] The causal relationships are sequentially processed, and then the EMBert model with parameter freezing is used to perform semantic encoding of the causal relationships, satisfying the relationship:

[0087] Input=[CLS]e0[indirect]e1[indirect]…[indirect]e n [effect];

[0088]

[0089] where Input represents the input sequence, [indirect] represents the insertion symbol of indirect causal relationships between events, [effect] represents the final result event of the causal chain, and e i is the embedding vector of event e in sequence k.

[0090] As Figure 3As shown in the upper half part (constructing the external knowledge base causal chain), the causal graph containing the coreference relationship is constructed through the external knowledge base CauseNet. In order to extract the potential indirect events related to the cause event and the result event from the external knowledge base, the present application specifically does the following: starting from the target cause event, the subsequent events related to the cause event are explored step by step by traversing the event network in the knowledge base. In each step of the search, the current event is regarded as a new cause event, and the downstream exploration is continued until the target result event is successfully located. At the same time, the indirect causal relationship between the coreference events is included, so that the complete causal relationship link is constructed. The causal relationship link is part of the causal graph, and each link is a path in the causal graph. For example, for the target event pair killing—injuring, the causal chain constructed through the external knowledge base CauseNet is as follows: [CLS] killing[indirect] protest[indirect] clash[indirect] injuring[effect], [CLS] demonstrate[indirect] protest[indirect] escalation[indirect] injury[effect]. The serialization process is the process of converting the causal relationship link into a linear sequence in order to input and process the model. The serialization process usually introduces specific markers such as [indirect] and [effect] to identify the indirect causal relationship between events and the final result event of the causal chain. The parameter frozen EMBert model is an EMBert model with fixed parameters after pre-training, which is used for semantic encoding of the serialized causal relationship link. Parameter freezing can maintain the stability of the model and the general semantic information learned in the pre-training stage, while reducing the computational overhead in the training process.

[0091] Preferably, in step S3, the embedding vector of the caret inserted between the event nodes of the causal graph is included The weights of the same nodes and edges in different sequences are combined with attention weights, satisfying the relationship: The weights of the same nodes and edges in different sequences are combined with attention weights, satisfying the relationship:

[0092]

[0093]

[0094] wherein, represents the weight of the event node e i to the event node e j in the sequence k, W g is a learnable matrix, represents the weight of the event node e i and e jAttention weights between, v T For learnable vectors, This represents the weights among global event nodes.

[0095] Specifically, this is achieved by calculating the embedding vectors of the caret symbols between event nodes in the causal graph. and The attention weights are combined to fuse the weights of the same nodes and edges in different sequences. Specifically, the learnable matrix W is first used... g A linear transformation is performed on the embedding vector of the caret and the embedding vector of the final event in the causal chain, and the event node e in sequence k is calculated using the LeakyReLU activation function. i To event node e j weight Then, by calculating attention weights The weights of the same event nodes in different sequences are weighted and fused to obtain the weights among global event nodes. This process aims to dynamically adjust the contributions of weights among event nodes in different sequences through an attention mechanism, enabling the model to more accurately capture causal relationships between events and improve the accuracy and robustness of causal inference. The learnable matrix W... g A matrix used to linearly transform the embedding vectors of the caret and the embedding vectors of the final event in the causal chain. It learns to adjust the vector representation to better capture the weight relationships between event nodes; the LeakyReLU activation function is an activation function used to introduce non-linearity, enabling the model to learn more complex feature representations; attention weights. This indicates that in sequence k, event node e i With E j Attention weights between nodes reflect the relative importance of pairs of the same event nodes in different sequences.

[0096] Therefore, the causal inference model can more accurately calculate the weights between event nodes, dynamically adjust the contribution of the weights between event nodes in different sequences, and obtain the weight representation between global event nodes. This makes the causal inference model perform well in handling complex text and multi-hop inference tasks, and can be more reliably applied to practical causal analysis scenarios. It can provide strong support for causal inference tasks in the field of natural language processing.

[0097] Preferably, step S3 includes using mean pooling to fuse the features of the same event nodes from multiple different sequences into a unified global feature, satisfying the following relation:

[0098]

[0099] in Indicates event e iembedding vectors in the sequence, representing the number of sequences with the same event node e i , fusing the global feature representation of each sequence event node.

[0100] Specifically, mean pooling is adopted to fuse the features of the same event node in multiple different sequences into a unified global feature, and the global feature representation is obtained by calculating the average of the embedding vectors of the same event node in multiple sequences. This mean pooling operation aims to integrate feature information from different sequences, reduce the dimension of features, and at the same time enhance the robustness and generalization ability of the causal inference model to event node features, providing higher quality input features for subsequent graph neural network processing.

[0101] Preferably, in step S3, the attention coefficients between each node and its adjacent nodes are calculated according to the graph attention mechanism, the features of adjacent nodes are fused, and at the same time the edge weight information is fused into the node features by using the graph convolutional neural network, satisfying the relationship:

[0102]

[0103] wherein, represents the attention coefficient between adjacent nodes, A is the attention coefficient matrix, H (0) represents the node feature of the initial layer, W GAT represents a learnable feature transformation matrix, H (1) represents the feature after the first layer of graph attention mechanism fusion, H (2) represents the feature after the second layer of graph convolutional neural network fusion, represents the degree matrix, represents the normalized adjacent matrix of the graph, W GCN represents the weight between global event nodes, each element is

[0104] The attention coefficients between each node and its adjacent nodes are calculated by the graph attention mechanism (GAT), the features of adjacent nodes are fused, and at the same time the edge weight information is fused into the node features by using the graph convolutional neural network (GCN). Specifically, first, the learnable feature transformation matrix W GAT is used to transform the node feature H (0) of the initial layer, and then the attention coefficients between each pair of nodes are calculated by the attention mechanism These attention coefficients can reflect the relative importance between nodes and are used to weight and fuse the features of adjacent nodes to obtain the feature H (1)Next, the edge weight information is fused into the node features by using a graph convolutional neural network (GCN), and the propagation of the graph structure information is performed through the normalized adjacency matrix and the degree matrix to obtain the second-layer graph convolutional neural network fused features H (2) This process aims to fully utilize the structured semantic information of the causal graph through the feature fusion capability of the graph neural network, improve the reasoning capability of the causal inference model for the complex causal logic between events, and enhance the accuracy and robustness of causal inference.

[0105] Preferably, step S4 satisfies the relationship:

[0106]

[0107] f = [y1; y2;...; y n ];

[0108] wherein represents the node features of the structured semantic graph, z i represents the coreference features of the event nodes, y i represents the fusion result of the node features and the coreference features of the structured semantic graph, and f represents the feature concatenation vector of each coreference event of the cause event or the result event.

[0109] In step S4, in order to fuse the node features extracted in step S3 and the coreference features extracted in step S2, a multi-layer perceptron (MLP) is used to perform nonlinear transformation on the features of each event node. Specifically, the node features of the structured semantic graph and the coreference features z i of the event nodes are concatenated to form a joint feature vector, and then nonlinear transformation is performed by the MLP to obtain the fused feature representation y i . Finally, the fused features y i of all event nodes are concatenated into a feature vector f, which represents the feature concatenation vector of each coreference event of the cause event or the result event. This process aims to integrate the node features and the coreference features of the structured semantic graph to form a more comprehensive and richer feature representation, providing higher quality input features for subsequent causal inference. By fusing the node features and the coreference features of the structured semantic graph, the feature representation is more comprehensive and richer, which can better reflect the semantic information and coreference relationship of the event nodes; and through nonlinear transformation by the multi-layer perceptron (MLP), the complex relationships between features can be more accurately captured, improving the effect of feature fusion; the fused feature representation y i and the feature concatenation vector f can more effectively support subsequent causal inference tasks, improving the accuracy and robustness of the causal inference model.

[0110] Preferably, step S5 satisfies the relationship:

[0111]

[0112] M = [head1;... ; head h ]W O ;

[0113] wherein C cause and C effect represent the concatenated cause event and result event feature vectors respectively, W i Q , represent the weight matrices of Q, K, V respectively, W O is the weight matrix of the multi-head attention output mapping, and M is the comprehensive feature representation of the cause event and the result event, represents the context semantic feature of the cause event in the text, represents the context semantic feature of the result event in the text, f cause represents the external knowledge base feature of the cause event, and f effect represents the external knowledge base feature of the result event.

[0114] In step S5, in order to fuse the feature vectors of the cause event and the result event, a multi-head attention mechanism is adopted. Specifically, first, the context semantic feature of the cause event is concatenated with the external knowledge base feature f cause to form a feature vector C cause , and the context semantic feature of the result event is concatenated with the external knowledge base feature f effect to form a feature vector C effect . Then, the outputs of multiple attention heads are calculated through the multi-head attention mechanism, and these outputs are concatenated and passed through an output mapping matrix W O to obtain the final comprehensive feature representation M. This process aims to capture various interaction relationships between the cause event and the result event through the multi-head attention mechanism, and to improve the comprehensive analysis ability of the causal inference model on the causal relationship between events.W i Q , respectively represent the query (Query), key (Key), and value (Value) weight matrices of the i-th attention head, which are used to linearly transform the feature vector into different representation spaces. By concatenating the context semantic features and the external knowledge base features, the information of the internal text and the external knowledge base can be integrated, making the feature representation more comprehensive and richer. Through the multi-head attention mechanism, various interaction relationships between the cause event and the result event can be captured, improving the understanding ability of the causal inference model for causal relationships. Through the nonlinear transformation of the multi-head attention mechanism, the complex relationships between features can be more accurately expressed, improving the feature expression ability of the causal inference model and the accuracy of causal inference.

[0115] Further, step S5 includes inputting the integrated feature representation into a prediction layer, determining whether a causal relationship exists between the event pair, and calculating the loss thereof to obtain the final prediction result of the event pair relationship, satisfying the relationship:

[0116] p i = sigmoid(WM + b);

[0117]

[0118] wherein, p i represents the predicted probability, y i represents the true label, N represents the number of samples of the event pair, represents the binary cross-entropy loss function, is the integrated loss, M is the integrated feature representation of the integrated cause event and result event, b represents the bias of the projection layer, and λ represents the loss function weight coefficient, represents the loss function of contrast training.

[0119] In step S5, the integrated feature representation M is input into the prediction layer, and the prediction layer is used to determine whether a causal relationship exists between the event pair and calculate the corresponding loss function to optimize the model. Specifically, the prediction layer uses the sigmoid() function to map the integrated feature representation to a probability value p i , which represents the possibility of a causal relationship between the event pair. The difference between the predicted probability and the true label is measured by the binary cross-entropy loss function , and the loss function of contrast learning is combined to construct the integrated loss to comprehensively optimize the prediction ability and contrast learning ability of the causal inference model. The purpose of this process is to improve the recognition accuracy and generalization ability of the model for causal relationships by minimizing the integrated loss.

[0120] To this end, the binary cross-entropy loss function and the loss function of contrast training The combination of the two can make the causal inference model more accurately predict the causal relationship between the event pairs, and improve the prediction accuracy; through the optimization of the comprehensive loss, the causal inference model not only learns the ability to predict the causal relationship in the training process, but also can enhance the sensitivity to the semantic similarity between the event pairs, and improve the generalization ability; the introduction of the loss function of the contrast training makes the causal inference model be able to distinguish the co-reference event pairs and the non-co-reference event pairs, and can enhance the discrimination ability of the model, and help to improve the robustness. The introduction of the loss function of the contrast training makes the causal inference model be able to distinguish the co-reference event pairs and the non-co-reference event pairs, and can enhance the discrimination ability of the model, and help to improve the robustness.

[0121] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the exemplary description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0122] Although the embodiments of the present application have been shown and described, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and purposes of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A causal inference method that integrates coreference and external knowledge enhancement, characterized in that, The causal inference method includes the following steps: S1: Extract event words from the document-level text as masking target words and randomly mask non-event words. Use the improved BERT model to predict the masked event words through context reconstruction during the pre-training stage, so that the event words can fully learn the contextual semantic features. S2: Randomly extract event pairs from the text and label them as cause events and result events. Extract the co-referenced event pairs of cause events and result events. Obtain the labeled positive co-referenced event samples from an external knowledge base. At the same time, generate non-co-referenced event pairs as negative samples. Use a contrastive learning mechanism to design a contrastive loss function to narrow the semantic distance between co-referenced event pairs and obtain the co-referenced features of co-referenced events. S3: Extract potential indirect events associated with cause / effect events from an external knowledge base to construct an external knowledge base causal graph. Construct a structured semantic graph with events as nodes and causal relationships as edges from the external knowledge base causal graph. Use graph attention mechanism and graph convolutional neural network to fuse node features. The potential indirect events include indirect events between cause events and result events. S4: Fuse the coreference features from step S2 and the fused node features from step S3, and perform nonlinear transformation through a fully connected layer to obtain the external knowledge base features of the cause event and the result event. S5: A multi-head attention mechanism is used to fuse the contextual semantic features and external knowledge base features to obtain a comprehensive feature representation. The comprehensive feature representation is then input into the prediction layer to obtain the final prediction result of the event pair relationship. Step S3 includes extracting potential indirect events associated with cause / effect events from an external knowledge base, and identifying and extracting causal relationships between events using entity linking and relationship extraction techniques; The causal relationship is serialized, and then the EMBert model with parameter freezing is used to semantically encode the causal relationship to satisfy the following relation: ； ； in, Represents the input sequence. The caret symbol indicates an indirect causal relationship between events. This represents the event that is the final result of a causal chain. Indicates an event In sequence Embedded vectors in; Step S3 includes calculating the embedding vector of the inter-event caret in the causal graph. and By combining attention weights and fusing the weights of the same nodes and edges in different sequences, the following relationship is satisfied: ； ； ； in, Represents a sequence event nodes To the event node The weight, For learnable matrices, Represents a sequence event nodes and Attention weights between them For learnable vectors, The weights between global event nodes; Step S5 satisfies the following relation: ； ； ； ； in, and These represent the concatenated feature vectors of the causal event and the resultant event, respectively. , , They represent The weight matrix, The weight matrix is ​​the output mapping for multi-head attention. This represents the combined characteristics of causal and consequential events. This indicates the contextual semantic features of the causal event in the text. This indicates the contextual semantic features of the resulting event in the text. External knowledge base features representing causal events External knowledge base features representing the resulting event.

2. The causal inference method according to claim 1, characterized in that, In step S1, the following relation is satisfied: ； ； ； in, Represented as a sequence of words in the input text. Indicates the first One word, Represents a set of event words. Indicates the length of the word sequence in the text; Indicates the first The probability of masking each word and For mask probability, Indicates the first The contextual semantic features of each word, i.e., the embedding vector of the event word.

3. The causal inference method according to claim 1, characterized in that, Step S2 includes: By designing a contrastive loss function using a contrastive learning strategy, the semantic distance between causal events and their co-referenced events, and between result events and their co-referenced events, is narrowed, while the semantic distance between non-co-referenced event pairs is widened. Construct positive sample coreference pairs using external knowledge bases and text events. Randomly combine from non-co-referenced events A projection layer is introduced to map events to a new semantic space, and cosine similarity is used to calculate the relevance of core-referenced events to the context, satisfying the following relation: ； ； ； in, and These represent the weight matrix and bias of the projection layer, respectively. Indicates the first The contextual semantic features of each word Represents the embedding vector The projection vector, To compare the loss function used during training, express and Similarity between them This represents the temperature coefficient.

4. The causal inference method according to claim 1, characterized in that, Step S3 includes using mean pooling to fuse the features of the same event nodes from multiple different sequences into a unified global feature, satisfying the following relation: ； in Indicates an event Embedded vectors in a sequence Indicates nodes with the same event the number of sequences, To fuse the global feature representations of each sequence event node.

5. The causal inference method according to claim 4, characterized in that, Step S3 includes calculating the attention coefficient between each node and its neighboring nodes according to the graph attention mechanism, performing feature fusion on the neighboring nodes, and simultaneously using a graph convolutional neural network to fuse edge weight information into the node features, satisfying the following relationship: ； ； ； in, This represents the attention coefficient between adjacent nodes. This is the attention coefficient matrix. Represents the node characteristics of the initial layer. This represents the learnable feature transformation matrix. This represents the features fused by the attention mechanism in the first layer graph. This represents the features fused by the second-layer graph convolutional neural network. Degree matrix, This represents the normalized adjacency matrix of the graph. This represents the weight between global event nodes, with each element being... .

6. The causal inference method according to claim 1, characterized in that, Step S4 satisfies the following relation: ； ； in Node features representing structured semantic graphs The coreference characteristic representing event nodes. This represents the fusion result of node features and coreference features in a structured semantic graph. A feature concatenation vector representing the common references of the causal or resultant events.

7. The causal inference method according to claim 1, characterized in that, Step S5 includes inputting the comprehensive feature representation into the prediction layer, determining whether a causal relationship exists between event pairs, calculating the loss, and obtaining the final prediction result of the event pair relationship, satisfying the following relation: ； ； ； in, Represented as predicted probability, Represented as a real label, Indicates the number of samples for each event pair. Represented as a binary cross-entropy loss function, To account for the overall losses, This represents the combined characteristics of causal and consequential events. Indicates the bias of the projection layer. Represents the weight coefficients of the loss function. This represents the loss function for comparative training.

Images