Identification method and device of causal relationship, electronic equipment, storage medium and computer program product
By constructing a heterogeneous graph interactive network and multi-grained comparison learning, the problem of causal relationship recognition in low-resource languages is solved, cross-language causal relationship recognition is realized, and the transferability and recognition performance of causal knowledge are improved.
Patent Information
- Application Number
- CN202411072445.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art has difficulty identifying document-level event causality in low-resource languages, especially due to shortage of training data and the problem of cross-language causal knowledge alignment.
By building a heterogeneous graph interactive network, using contrast learning to train the model in the source language, and applying it to the target language, realizing document-level causal relationship recognition with zero-sample cross-language, combining multi-grained comparison transfer learning and dependency structure analysis, improves Cross-lingual transferability of causal knowledge.
Under limited language resources, causal knowledge is effectively transferred from the source language to other languages, improving the causal recognition performance in low-resource languages, realizing cross-language causal recognition, and making stronger adaptability.
Smart Images

Figure CN118917313A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of natural language processing, and in particular to a method, device, electronic device, storage medium and computer program product for identifying causal relationships. Background Art
[0002] Event causality identification (ECI) is an important task in natural language processing (NLP), which can facilitate the realization of various applications, such as explainable question-answering systems, intelligent search, and complex reasoning. Related technologies focus on the sentence level based on English corpus.
[0003] However, a large number of causal relationships are expressed by multiple sentences. For example, about 68.7% of the causal relationships in the English corpus are attributed to event pairs between sentences. Therefore, it is necessary to identify event causal relationships at the document level. Moreover, training an ECI model usually relies on a large amount of data, especially for the document level, which makes it difficult to adapt to low-resource languages. In addition, low-resource languages face a serious shortage of training data, making it challenging to improve the document-level ECI performance of language models. Summary of the invention
[0004] The causal relationship identification method, device, electronic device, storage medium and computer program product provided by the exemplary embodiments of the present disclosure can at least solve the above technical problems and other technical problems not mentioned above.
[0005] According to one aspect of the present disclosure, a method for identifying causal relationships is provided, comprising: obtaining text data to be identified in a target language; identifying causal relationships between events in the text data to be identified based on a heterogeneous graph interaction network trained with text data in a source language, wherein the target language includes a language different from the source language; wherein the training step of the heterogeneous graph interaction network comprises: obtaining text data in the source language; generating a heterogeneous graph interaction network based on the text data in the source language, wherein the heterogeneous graph interaction network comprises: statement nodes corresponding to statements in the text data in the source language, wherein the statements are sentences containing two events; and The method comprises the following steps: obtaining an initial feature vector of each node in the heterogeneous graph interaction network through a model; based on the initial feature vector, using a graph attention network to learn an embedding representation of each node in the heterogeneous graph interaction network by optimizing a loss function to train the heterogeneous graph interaction network, wherein the loss function is used to maximize the similarity between positive samples and minimize the similarity between negative samples through comparative learning between positive and negative samples, the positive samples include multilingual statement texts obtained by copying statements containing causal events in the text data of the source language, and the negative samples include multilingual statement texts obtained by copying statements not containing causal events in the text data of the source language.
[0006] Optionally, the heterogeneous graph interaction network also includes: event pair nodes corresponding to event pairs in the text data of the source language; the positive samples also include: event pairs and contexts in statements of causal events in the text data of the source language; the negative samples also include: event pairs that do not contain causal events and contexts unrelated to the event pairs in the text data of the source language.
[0007] Optionally, the loss function is also used to calculate the cross entropy loss of the concatenation of the statement representation and the event pair representation, wherein the statement representation is the embedding representation corresponding to the statement node, and the event pair representation is the embedding representation corresponding to the event pair node.
[0008] Optionally, generating a heterogeneous graph interaction network based on the text data of the source language includes: parsing the dependency structure of sentences in the text data of the source language to obtain a corresponding dependency tree; extracting information phrases in the dependency tree and dependency relationships corresponding to the information phrases to obtain a simplified dependency tree with the information phrases; generating the heterogeneous graph interaction network based on the simplified dependency tree and the text data of the source language, wherein the heterogeneous graph interaction network also includes information phrase nodes corresponding to the information phrases.
[0009] Optionally, the heterogeneous graph interaction network obtained by training based on the text data of the source language identifies the causal relationship between events in the text data to be identified, including: concatenating the statement representation and the event pair representation in the heterogeneous graph interaction network obtained by training based on the text data of the source language, and identifying the causal relationship between events in the text data to be identified.
[0010] Optionally, the obtaining of the initial feature vector of each node in the heterogeneous graph interaction network based on a pre-trained language model includes: using the pre-trained language model to convert each word in the text data of the source language into an embedding vector; obtaining the initial feature vector of each node in the heterogeneous graph interaction network based on the embedding vector of each word, wherein the initial feature vector of the information phrase node is the mean of the embedding vectors of the words related to the current information phrase node plus the embedding vector of the role information of the current information phrase, the initial feature vector of the statement node is the mean of the embedding vectors of the words related to the current statement node, and the initial feature vector of the event pair node is the connection of the embedding vectors of the words corresponding to the events related to the current event pair node.
[0011] According to another aspect of the present disclosure, a causal relationship recognition device is also provided, comprising: a data acquisition module, configured to: acquire text data to be recognized in a target language; a causal recognition module, configured to: recognize the causal relationship between events in the text data to be recognized based on a heterogeneous graph interaction network trained with text data of a source language, wherein the target language includes a language different from the source language; wherein the training step of the heterogeneous graph interaction network comprises: acquiring text data of the source language; generating a heterogeneous graph interaction network based on the text data of the source language, wherein the heterogeneous graph interaction network comprises: statement nodes corresponding to statements in the text data of the source language, wherein the statements are statements containing two The method comprises the steps of: obtaining an initial feature vector of each node in the heterogeneous graph interaction network based on a pre-trained language model; learning an embedded representation of each node in the heterogeneous graph interaction network by optimizing a loss function based on the initial feature vector, so as to train the heterogeneous graph interaction network, wherein the loss function is used to maximize the similarity between positive samples and minimize the similarity between negative samples through comparative learning between positive and negative samples, the positive samples include multilingual statement texts obtained by copying statements containing causal events in the text data of the source language, and the negative samples include multilingual statement texts obtained by copying statements not containing causal events in the text data of the source language.
[0012] According to another aspect of an embodiment of the present disclosure, there is also provided an electronic device, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute any of the causal relationship identification methods described above.
[0013] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is also provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute any of the causal relationship identification methods described above.
[0014] According to another aspect of an embodiment of the present disclosure, a system is also provided, comprising at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute any of the causal relationship identification methods described above.
[0015] According to another aspect of an embodiment of the present disclosure, a computer program product is further provided, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method for identifying a causal relationship as described in any one of the above items is implemented.
[0016] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0017] According to the causal relationship identification method, device, electronic device, storage medium and computer program product disclosed in the present invention, by constructing a heterogeneous graph interaction network and performing multilingual replication on statements in the source language containing event pairs and statements not containing event pairs, the cross-language transferability of causal knowledge learned from the source language can be improved using contrastive learning at the statement level; the ECI model (heterogeneous graph interaction network) trained with labeled data in the source language can be directly applied to the target language to achieve zero-sample cross-language document-level ECI; under limited language resources, causal relationship knowledge can be effectively transferred from the source language to any other language (e.g., low-resource / less-researched languages, etc.) to better adapt to low-resource languages.
[0018] In addition, by combining contrastive learning at the aspect-level causal pattern level, multi-granularity contrastive transfer learning can be used to further improve the cross-lingual transferability of causal knowledge learned from the source language.
[0019] In addition, after using multilingual natural language tools to parse the dependency structure of sentences to obtain information phrases in the document, a heterogeneous graph interaction network is constructed based on the information phrases, which can model the long-distance dependencies between events scattered in the document based on the global information of the document. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0021] Figure 1 A schematic diagram showing the use of a multilingual ECI model for document-level causal relationship prediction in the related art;
[0022] Figure 2 A schematic diagram showing document-level causal relationship prediction in related technologies;
[0023] Figure 3 A flowchart showing a method for training a heterogeneous graph interaction network in an exemplary embodiment of the present disclosure;
[0024] Figure 4 A schematic diagram showing a method of simulating long-distance dependencies between events by using information phrases in an exemplary embodiment of the present disclosure;
[0025] Figure 5 A schematic diagram showing a method of obtaining a dependency tree in an exemplary embodiment of the present disclosure;
[0026] Figure 6 A flow chart showing a method for identifying causal relationships in an exemplary embodiment of the present disclosure;
[0027] Figure 7 A flowchart showing a method for identifying a causal relationship in an exemplary embodiment of the present disclosure is provided;
[0028] Figure 8 A schematic diagram showing a method for identifying a causal relationship in an exemplary embodiment of the present disclosure;
[0029] Fig. 9 A block diagram showing a device for identifying causal relationships in an exemplary embodiment of the present disclosure;
[0030] Fig.10 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0031] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.
[0033] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0034] Although pre-trained language models have demonstrated significant causal recognition capabilities in various tasks, they still have difficulties in multilingual settings, as has been demonstrated in related research. Even the powerful ChatGPT (ChatGenerative Pre-trained Transformer) shows quite limited performance in multilingual causal relationship prediction tasks, with an average accuracy of only about 37%. Especially for low-resource languages, the model performs even worse, for example, only reaching an accuracy of about 6.3% in Urdu.
[0035] Related Art Despite significant efforts in the monolingual setting, two key challenges arise when applying monolingual ECI to the zero-shot cross-lingual document-level setting:
[0036] (1) Language-agnostic causal knowledge alignment. Each language has its own idiosyncrasies. A multilingual ECI model (MECI) trained in the source language inevitably tends to learn language-specific knowledge rather than purely language-agnostic knowledge (i.e., causal knowledge). As a result, a trained ECI model may only perform well in the source language, e.g., the distance distributions between causal events in different languages are significantly different. Figure 1 A schematic diagram showing the use of a multilingual ECI model for document-level causal relationship prediction in related technologies, see Figure 1, the principal distance between causal events in English documents is usually larger than the principal distance between causal events in Urdu documents. According to statistics, the principal distance between causal events in the English corpus is 40 words, while in Turkish, this distance is 33 words, and in Danish, this distance is 23 words. In addition, based on the error analysis of the original pre-trained language model, when the model trained on English was tested on Danish, more than half (53%) of the false positives had a distance between causal events exceeding 23 words, which means that the ECI model trained on the English corpus tends to predict a causal relationship between two events that are far apart, which has a negative impact on the prediction of Danish;
[0037] (2) Dispersion of causal events. It requires a holistic understanding of the document to capture the long-distance dependencies between events. Figure 2 A schematic diagram showing document-level causal relationship prediction in related technologies, see Figure 2 ,The related techniques can predict the intra-sentence causal relationship between event 201 and event 202. In contrast, event 201 and event 203 are located in sentence 1 and sentence 5 respectively, which belong to long-distance events.,The interference of irrelevant events such as event 204 and event 205 makes it difficult to directly simulate the long-distance,dependency between the mentioned events.
[0038] In order to solve the above problems, the present disclosure provides a causal relationship identification method, device, electronic device, storage medium and computer program product. After constructing a heterogeneous graph interaction network and performing multilingual replication on statements containing event pairs and statements not containing event pairs in the source language, contrastive learning is used at the statement level to improve the cross-language transferability of causal knowledge learned from the source language; the ECI model (heterogeneous graph interaction network) trained with labeled data in the source language can be directly applied to the target language to achieve zero-sample cross-language document-level ECI; under limited language resources, causal relationship knowledge can be effectively transferred from the source language to any other language (for example, low-resource / less-studied languages, etc.) to better adapt to low-resource languages.
[0039] Next, we will refer to Figures 3 to 10 The present invention specifically describes the causal relationship identification method, device, electronic device, storage medium and computer program product.
[0040] Figure 3 A flowchart of a method for training a heterogeneous graph interaction network in an exemplary embodiment of the present disclosure is shown.
[0041] Reference Figure 3 According to an exemplary embodiment of the present disclosure, the training steps of the heterogeneous graph interaction network may include but are not limited to the following steps:
[0042] In step 301, text data in a source language may be obtained.
[0043] According to an exemplary embodiment of the present disclosure, a language with rich corpus resources can be used as a source language. The source language may contain one or more language types. The text data of the source language may be long text data at the document level.
[0044] In step 302, a heterogeneous graph interaction network may be generated based on the text data in the source language, wherein the heterogeneous graph interaction network may include but is not limited to: statement nodes corresponding to statements in the text data in the source language, where the statement is a sentence containing two events.
[0045] According to an exemplary embodiment of the present disclosure, natural language processing (NLP) technology can be used to extract required information from text data of the source language, including different types of entities (such as names, etc.) and events, etc.; then the nodes and edges of the heterogeneous graph interaction network can be defined based on the extracted information. For example, a sentence containing two events can be used as a statement to define a statement node, and sentence nodes, etc. can also be defined, and each node can be assigned a unique identifier. Different types of edges can be defined to represent the relationship between nodes. For example, sentence-sentence edges, sentence-statement edges, etc. can be defined. A graph data structure (such as an adjacency matrix, an adjacency list, or a graph database) can be used to represent a heterogeneous graph, and the defined nodes and edges can be added to the graph to form a complete heterogeneous graph interaction network.
[0046] According to an exemplary embodiment of the present disclosure, generating a heterogeneous graph interaction network based on text data of a source language may include but is not limited to the following steps: the dependency structure of sentences in the text data of the source language may be parsed to obtain a corresponding dependency tree; information phrases in the dependency tree and dependency relationships corresponding to the information phrases may be extracted to obtain a simplified dependency tree with information phrases; a heterogeneous graph interaction network may be generated based on the simplified dependency tree and the text data of the source language, wherein the heterogeneous graph interaction network also includes information phrase nodes corresponding to the information phrases.
[0047] Figure 4 A schematic diagram showing simulation of long-distance dependency between events by information phrases in an exemplary embodiment of the present disclosure.
[0048] Reference Figure 4 According to an exemplary embodiment of the present disclosure, due to Figure 2The dispersion of causal events in the example will lead to interference from irrelevant events, making it difficult to directly simulate the long-distance dependencies between the mentioned events. Therefore, the information phrases (i.e., information phrases 401 and information phrases 402) related to the dispersed causal events event 201 and event 203 can be efficiently obtained by processing the dependency structure and heuristic rules; since these information phrases are similar and can serve as an intermediate bridge connecting the two events, this shows the importance of information phrases in simulating long-distance dependencies between events.
[0049] So, just as an example, for a given sentence in the text data of the source language, we can use the multilingual NLP toolkit Trankit to obtain Figure 5 The dependency tree shown in (a).
[0050] Figure 5 A schematic diagram showing how to obtain a dependency tree in an exemplary embodiment of the present disclosure.
[0051] Furthermore, the dependency structure of the dependency tree can be processed using heuristic rules based on the semantics of the dependency relationship to obtain a simplified dependency tree with information phrases, such as Figure 5 As shown in (b) in the figure. First, all dependency relationships and their subtypes in the dependency tree can be analyzed, and semantically rich and indicative dependency relationships (edges) can be retained, for example, nsubj (noun subject), obj (object), obl (oblique case noun), etc. 19 types. It is understood that the number of types does not constitute a limitation to the present disclosure.
[0052] Specifically, for example, the simplification process may include the following steps: first, Trankit may be used to obtain the sentence dependency structure graph (a), and then the 19 selected dependency relationships are retained. If the two words have dependency relationships other than the 19, they may be directly concatenated together (for example, "November" and "on", "25", "2019" in (a)). In the above manner, information phrases and their corresponding dependency relationships may be extracted, and these dependency relationships may indicate the relevant arguments of the event of interest (for example, the noun subject (nsubj) of event 201 is the information phrase 401, etc.).
[0053] After obtaining the simplified dependency tree, a heterogeneous graph interaction network including information phrase nodes can be generated based on the simplified dependency tree and text data of the source language, so that long-distance dependency relationships between events scattered in the document can be modeled through the heterogeneous graph interaction network.
[0054] Return to reference Figure 3 , in step 303, an initial feature vector of each node in the heterogeneous graph interaction network can be obtained based on the pre-trained language model.
[0055] According to an exemplary embodiment of the present disclosure, a heterogeneous graph interaction network including information phrase nodes, sentence nodes, statement nodes, and event pair nodes may be constructed.
[0056] According to an exemplary embodiment of the present disclosure, obtaining an initial feature vector for each node in a heterogeneous graph interaction network based on a pre-trained language model may include but is not limited to the following steps: each word in the text data of the source language may be converted into an embedding vector using a pre-trained language model; the initial feature vector for each node in the heterogeneous graph interaction network may be obtained based on the embedding vector of each word, wherein the initial feature vector of an information phrase node may be the mean of the embedding vectors of words related to the current information phrase node plus the embedding vector of the role information of the current information phrase, the initial feature vector of a statement node may be the mean of the embedding vectors of words related to the current statement node, and the initial feature vector of an event pair node may be the concatenation of the embedding vectors of words corresponding to events related to the current event pair node.
[0057] According to an exemplary embodiment of the present disclosure, after each word is converted into an embedding vector using a pre-trained language model, for each information phrase node whose role information needs to be fused, the embedding vector of the information phrase node can be initialized by taking the average of the embeddings of the relevant words and adding the embedding of the role information. It is understandable that the role information can be obtained in the process of dependency parsing of the above sentence to obtain a dependency tree, for example, the 19 retained edge relationships, such as "subject", "object", etc.
[0058] For each sentence node, its embedding vector can be initialized by taking the average of the embeddings of the words in the sentence. A statement can be defined as a sentence containing two events to accommodate the case where the two events span sentences; for example, Figure 2 The positions shown in are event 201 and event 202 in the same sentence, or the connection of two sentences to include event pairs across sentences, for example, Figure 2 The event 201 in the first sentence and the event 203 in the fifth sentence are shown in . The embedding vector of the statement node can be initialized by taking the average of the word embeddings in the statement. For an event pair (i, j) with a representation (representation refers to a vector, i and j are the vectors (representations) of event i and event j respectively), their embedding vectors can be initialized as the concatenation of the two event representations:
[0059]
[0060] Among them, W v is a trainable parameter, || means concatenating two vectors, A vector representing events on nodes, e i and e j are vectors representing two events respectively.
[0061] To capture the interactions between these nodes, six types of edges can be introduced: phrase-phrase edge, sentence-phrase edge, phrase-event pair edge, sentence-event pair edge, statement-event pair edge, and event pair-event pair edge.
[0062] The phrase-phrase edge can be obtained after performing dependency tree parsing based on the sentence, and other edges can be customized. During the training process, these edges can transfer information from different nodes to update the representation of different nodes.
[0063] After constructing the heterogeneous graph (initial heterogeneous graph interaction network), Graph Attention Networks (GATv2) can be applied to simulate global interactions.
[0064] For each edge (j, i) in a heterogeneous graph, the importance of neighbor node j to the current node i can be expressed by the scoring function f:
[0065] f(h i ,h j )=a T LeakyReLU(W·[h i ||h j ])
[0066] Among them, h i 、h j represents the node vector, a and W are both trainable parameters, · represents the dot product, and LeakyReLU is a neural network activation function.
[0067] Then, we can define the attention function:
[0068]
[0069] Among them, h j′ is the embedding representation of the neighbor nodes, is the number of neighbor nodes, α ij Represents the attention score.
[0070] Then, the normalized attention scores can be used to calculate the weighted average of the neighbor node conversion features as the new embedding representation of node i (that is, the updated embedding representation is obtained):
[0071]
[0072] Among them, W r is a trainable parameter, h′ i h i Updated embedding representation.
[0073] We can use K independent attention heads and concatenate their outputs as the output of node i:
[0074]
[0075] Among them, W o are trainable parameters, is the embedding representation of node i output by the kth attention head, output h′ i represents the embedded representation of the updated node. The output of each attention head is different. ‖ represents concatenating the K outputs together and then adding them to the trainable W. o Matrix multiplication for dimensionality reduction.
[0076] Return to reference Figure 3 In step 304, based on the initial feature vector, a graph attention network can be used to learn the embedding representation of each node in the heterogeneous graph interaction network by optimizing the loss function to train the heterogeneous graph interaction network, wherein the loss function is used to maximize the similarity between positive samples and minimize the similarity between negative samples through comparative learning between positive and negative samples, the positive samples include multilingual statement texts obtained by copying statements containing causal events in the text data of the source language, and the negative samples include multilingual statement texts obtained by copying statements that do not contain causal events in the text data of the source language.
[0077] According to an exemplary embodiment of the present disclosure, a graph attention network can be used to encode each node in a heterogeneous graph interaction network to obtain an event pair representation. The embedding representation of each node can be updated during the training process.
[0078] When obtaining the representation of an event pair, the statement containing the event pair can be first encoded, and then the representation of the tokens of the two events can be obtained and then averaged as the representation of the event pair.
[0079] In order to generalize causal patterns to multilingual spaces, contrastive learning can be used to align cross-lingual causal statement representations. For contrastive learning of causal patterns at the statement level, the causal event pair representations in the statements can be aligned across languages. Since the expression of causality may be different in different languages, after encoding statements with causal event pairs using multilingual pre-trained models, they can be considered to be in the same vector space, so contrastive learning can be performed based on this. Alignment means that the encoded vectors of statements with causal event pairs are as similar as possible.
[0080] Appropriate positive and negative samples can be selected for the anchored statements: (1) Positive samples: For each anchored causal statement, that is, a statement containing a causal event, a multilingual mixed statement can be generated as a positive sample using tools such as the MUSE bilingual dictionary. The phrases in the statement can be operated on by randomly replacing one word at a time. Specifically, for a given phrase, a bilingual dictionary (e.g., English-Danish) can be first randomly selected, and then each word in the phrase can be replaced one by one using the bilingual dictionary, which is expected to intuitively maintain more complete semantic information; (2) Negative samples: In order to make negative examples more recognizable, non-causal statements in the same document that do not have any textual overlap with the anchor statement, that is, statements that do not contain a causal event, can be selected as negative examples. In addition, since the expected model focuses on cross-language transfer, the negative sample list can be increased by generating multilingual negative examples through code conversion.
[0081] Afterwards, contrastive transfer learning can be used to make statements as close as possible to causal patterns in a multilingual space, and as far away as possible from statements without causal knowledge. That is, during training, the representations of statements containing causal events are as far away as possible from the representations of statements without causal events, and the representations of statements containing causal events are as close as possible to each other.
[0082] According to an exemplary embodiment of the present disclosure, for statement-level causal pattern contrastive learning, context-specific causal patterns can be learned from causal statements, so a statement-level causal pattern contrastive learning loss Aligning the representation of the anchored causal statement with the generated positive examples.
[0083] Formally, this can be formulated as:
[0084]
[0085] Among them, h CLS is the representation of the current statement, A representation that represents a statement that does not contain a causal event, represents the representation of statements containing causal events, K represents the number of negative samples, j represents the number of positive samples, s() represents the vector product, and n represents the total number of positive samples.
[0086] It can be understood that the event pairs mentioned in the present disclosure mean the pairwise combination of all events in the document; the statement means a sentence containing two events. If the two events are in one sentence, the statement is one sentence; if the two events are in different sentences, the statement is the concatenation of two sentences containing the two events respectively; the causal statement means a statement containing causal events.
[0087] According to an exemplary embodiment of the present disclosure, a heterogeneous graph interaction network may also include, but is not limited to: event pair nodes corresponding to event pairs in the text data of the source language; positive samples may also include, but are not limited to: event pairs and contexts in statements containing causal events in the text data of the source language; negative samples may also include, but are not limited to: event pairs that do not contain causal events in the text data of the source language and contexts unrelated to the event pairs.
[0088] According to an exemplary embodiment of the present disclosure, for aspect-level causal pattern contrastive learning, representations between causal patterns and statements are aligned in a fine-grained manner.
[0089] Using event-agnostic, context-specific patterns for causal recognition, causal patterns can be divided into two aspects, namely, two granularities of aspect-level comparison: event pairs and event-agnostic context (also called event-shielded context, event-pair-independent context, both of which refer to the context after removing event pairs from the statement). Specifically, each statement can be divided into event pairs and event-agnostic context, and representations of these two types of aspects, event pairs and event-agnostic context, can be obtained after encoding through a multilingual pre-trained language model.
[0090] We can further introduce aspect-level causal pattern contrastive learning loss and
[0091]
[0092] in, is the aspect-level contrastive learning loss for event pairs, h AspE is the representation of event pairs, is a representation of event pairs in statements involving causal events, is a representation of event pairs that do not contain causality.
[0093]
[0094] in, is the event-pair-independent contrastive learning loss, h AspC is the representation of the context, is a representation of the context in statements involving causal events, Represents an event pair irrelevant to the context.
[0095] According to an exemplary embodiment of the present disclosure, the loss function can also be used to calculate the cross entropy loss of the concatenation of the statement representation and the event pair representation, wherein the statement representation can be the embedding representation corresponding to the statement node, and the event pair representation can be the embedding representation corresponding to the event pair node.
[0096] According to an exemplary embodiment of the present disclosure, an event pair representation and a statement representation may be connected.
[0097]
[0098] in, Indicates the probability that the current sample is causal / non-causal, is the event pair representation, h CLS For the statement, W p is a trainable parameter matrix and softmax() is a normalized exponential function.
[0099] Connection is concatenation. In one example, concatenating two 5-dimensional vectors can produce a 10-dimensional vector, which can then be passed through the trainable parameter matrix W. p Then it was reduced to 5 dimensions.
[0100] In training, cross entropy can be used as the loss function
[0101]
[0102] Among them, E s represents the set of all event pairs, is the embedding representation of the event pair.
[0103] Finally, the loss can be calculated as follows
[0104]
[0105] Figure 6 A flow chart showing a method for identifying a causal relationship in an exemplary embodiment of the present disclosure.
[0106] Reference Figure 6 In step 601, text data to be recognized in a target language may be obtained.
[0107] According to an exemplary embodiment of the present disclosure, the text data to be recognized may be long text data at a document level.
[0108] In step 602, a causal relationship between events in the text data to be identified may be identified based on a heterogeneous graph interaction network trained based on text data in a source language, wherein the target language may include a language different from the source language.
[0109] According to an exemplary embodiment of the present disclosure, when the target language is a low-resource, zero-sample language, efficient recognition of causal relationships in the target language can also be achieved based on the heterogeneous graph interaction network obtained by training the source language. The exemplary embodiment of the present disclosure can also perform causal recognition on other target languages outside the training set.
[0110] According to an exemplary embodiment of the present disclosure, a heterogeneous graph interaction network obtained by training based on text data of a source language is used to identify the causal relationship between events in the text data to be identified, which may include but is not limited to the following steps: the causal relationship between events in the text data to be identified may be identified by concatenating statement representations and event pair representations in a heterogeneous graph interaction network obtained by training based on text data of a source language.
[0111] According to an exemplary embodiment of the present disclosure, after obtaining the updated event pair representation on the heterogeneous graph and the statement representation after contrastive learning, these two feature representations can be spliced together to obtain the final representation, and based on this final representation, cross-language event causal relationship recognition can be performed.
[0112] It can be understood that the final representation is a vector, based on which classification can be performed, for example, causal relationship / no causal relationship, etc.
[0113] The input of the heterogeneous graph interaction network model can be a document in the target language, in which the locations of different events can be marked, and the output can be the causal relationship between the given event pairs in the document, that is, the classification prediction of whether there is a causal relationship between the event pairs.
[0114] Figure 7 A flowchart illustrating a method for identifying a causal relationship in an exemplary embodiment of the present disclosure is shown.
[0115] Reference Figure 7 According to an exemplary embodiment of the present disclosure, heterogeneous graph contrast transfer learning can be used to model cross-language event causal relationship recognition, so that the model can use cross-language causal knowledge alignment to improve the performance of zero-shot language event causal relationship recognition tasks.
[0116] In an example, the causal relationship identification method can mainly include four steps: information phrase extraction, heterogeneous graph interaction network, multi-granularity contrast transfer learning, and feature fusion.
[0117] For information phrase extraction, multilingual natural language tools can be used to parse the dependency structure of sentences in document text data to obtain information phrases in the document, that is, sentence dependency tree parsing; for heterogeneous graph interaction networks, heterogeneous graph interaction networks can be constructed to model long-distance dependency relationships between events scattered in the document, that is, the full text is constructed as a heterogeneous graph; for multi-granularity contrastive transfer learning, the cross-language transferability of causal knowledge learned from the source language can be improved based on causal knowledge contrastive learning at the statement level and causal knowledge contrastive learning at the causal pattern level; finally, the updated event pair representation and causal statement representation can be feature fused to form the final representation for causal relationship identification between events, that is, result prediction.
[0118] Contrastive transfer learning can be composed of two parts: statement-level contrastive learning of causal patterns to align causal representations of statements across languages, and aspect-level contrastive learning of causal patterns to align fine-grained causal patterns and representations between statements.
[0119] It can be understood that the multi-granularity mentioned in the present disclosure may refer to the granularity at the statement level and the aspect level, and the aspect level also includes event pair patterns and event pair-independent context patterns.
[0120] Figure 8 A schematic diagram showing a method for identifying a causal relationship in an exemplary embodiment of the present disclosure.
[0121] Reference Figure 8 According to an exemplary embodiment of the present disclosure, a multilingual pre-trained language model (Multilingual PLM, MPLM) can be used to encode the entire document and obtain different nodes. The node information in the multi-level graph structure can include information phrases, sentences, statements, event pairs, etc. in the document; then, it can be combined into a multi-level graph structure (heterogeneous graph interaction network) composed of different nodes, and multi-granular causal pattern contrast learning is performed at the statement level and aspect level while using the graph attention network to update the nodes, which can also be used to update the embedded representation of the node. In the process of contrastive learning, for anchor statements, multi-language positive samples can be generated through code conversion; for contexts unrelated to anchor statements, they can be used as negative samples.
[0122] Fig. 9 A block diagram showing a device for identifying causal relationships in an exemplary embodiment of the present disclosure.
[0123] Reference Fig. 9 The exemplary embodiment of the present disclosure further provides a causal relationship identification device 900 , which may include but is not limited to a data acquisition module 901 and a causal identification module 902 .
[0124] The data acquisition module 901 can acquire text data to be recognized in the target language.
[0125] The causal identification module 902 can identify the causal relationship between events in the text data to be identified based on the heterogeneous graph interaction network obtained by training the text data of the source language, wherein the target language includes a language different from the source language; wherein the training steps of the heterogeneous graph interaction network may include but are not limited to the following steps: obtaining the text data of the source language; generating a heterogeneous graph interaction network based on the text data of the source language, wherein the heterogeneous graph interaction network includes: statement nodes corresponding to statements in the text data of the source language, wherein the statements are sentences containing two events; obtaining an initial feature vector of each node in the heterogeneous graph interaction network based on a pre-trained language model; based on the initial feature vector, using a graph attention network to learn the embedded representation of each node in the heterogeneous graph interaction network by optimizing a loss function to train the heterogeneous graph interaction network, wherein the loss function is used to maximize the similarity between positive samples and minimize the similarity between negative samples through comparative learning between positive and negative samples, wherein the positive samples include multilingual statement texts obtained by copying statements containing causal events in the text data of the source language, and the negative samples include multilingual statement texts obtained by copying statements not containing causal events in the text data of the source language.
[0126] It is understandable that in the exemplary embodiment of the causal relationship identification device 900 described above, the specific implementation process is roughly the same as the exemplary embodiment of the causal relationship identification method described above, and will not be repeated here. The causal relationship identification device 900 can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, these devices may correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. In addition, one or more functions implemented by these devices may also be uniformly executed by components in physical entity devices (for example, processors, clients, or servers, etc.).
[0127] Fig.10 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.
[0128] Reference Fig.10 The electronic device 1000 includes at least one memory 1001 and at least one processor 1002, wherein the at least one memory 1001 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 1002, the method for identifying a causal relationship according to an exemplary embodiment of the present disclosure is executed.
[0129] As an example, the electronic device 1000 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1000 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 1000 may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.
[0130] In the electronic device 1000, the processor 1002 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.
[0131] The processor 1002 may execute instructions or codes stored in the memory 1001, wherein the memory 1001 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.
[0132] The memory 1001 may be integrated with the processor 1002, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1001 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1001 and the processor 1002 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1002 can read files stored in the memory.
[0133] In addition, the electronic device 1000 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1000 may be connected to each other via a bus and / or a network.
[0134] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the above-mentioned causal relationship identification method.
[0135] Examples of computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers. It should be noted that the instructions can also be used to execute additional steps in addition to the above steps or to perform more specific processing when executing the above steps. These additional steps and further processing contents have been mentioned in the description of the relevant methods, so they will not be repeated here to avoid repetition.
[0136] Another embodiment of the present disclosure relates to a system including at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute the above-mentioned causal relationship identification method.
[0137] It should be noted that the system according to the exemplary embodiments of the present disclosure can completely rely on the execution of computer programs or instructions to realize corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., lib library) to realize the corresponding functions.
[0138] On the other hand, when the above-mentioned system is implemented in software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.
[0139] According to an exemplary embodiment of the present disclosure, the storage device may be integrated with the computing device, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage device may include an independent device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage device and the computing device may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the computing device can read instructions stored in the storage device.
[0140] Another embodiment of the present disclosure relates to a computer program product, including a computer program / instruction, which implements the causal relationship identification method as described in any one of the above when the computer program / instruction is executed by a processor.
[0141] According to the causal relationship identification method, device, electronic device, storage medium and computer program product provided by the present invention, by constructing a heterogeneous graph interaction network and performing multilingual replication on statements in the source language containing event pairs and statements not containing event pairs, the cross-language transferability of causal knowledge learned from the source language can be improved by using contrastive learning at the statement level; the ECI model (heterogeneous graph interaction network) trained with labeled data in the source language can be directly applied to the target language to achieve zero-sample cross-language document-level ECI; under limited language resources, causal relationship knowledge can be effectively transferred from the source language to any other language (e.g., low-resource / less-researched languages, etc.) to better adapt to low-resource languages.
[0142] In addition, by combining contrastive learning at the aspect-level causal pattern level, multi-granularity contrastive transfer learning can be used to further improve the cross-lingual transferability of causal knowledge learned from the source language.
[0143] In addition, after using multilingual natural language tools to parse the dependency structure of sentences to obtain information phrases in the document, a heterogeneous graph interaction network is constructed based on the information phrases, which can model the long-distance dependencies between events scattered in the document based on the global information of the document.
[0144] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.
Claims
1. A method for identifying causal relationships, characterized in that: include: Obtaining text data to be recognized in the target language; Identifying the causal relationship between events in the text data to be identified based on a heterogeneous graph interaction network trained with text data in a source language, wherein the target language includes a language different from the source language; The training steps of the heterogeneous graph interaction network include: Acquiring text data in the source language; Generating a heterogeneous graph interaction network based on the text data of the source language, wherein the heterogeneous graph interaction network includes: statement nodes corresponding to statements in the text data of the source language, the statement being a sentence containing two events; Acquire an initial feature vector of each node in the heterogeneous graph interaction network based on a pre-trained language model; Based on the initial feature vector, a graph attention network is used to learn the embedded representation of each node in the heterogeneous graph interaction network by optimizing the loss function to train the heterogeneous graph interaction network, wherein the loss function is used to maximize the similarity between positive samples and minimize the similarity between negative samples through comparative learning between positive and negative samples, the positive samples include multilingual statement texts obtained by copying statements containing causal events in the text data of the source language, and the negative samples include multilingual statement texts obtained by copying statements that do not contain causal events in the text data of the source language.
2. The method for identifying causal relationships according to claim 1, characterized in that: The heterogeneous graph interaction network also includes: event pair nodes corresponding to event pairs in the text data of the source language; the positive samples also include: event pairs and contexts in statements containing causal events in the text data of the source language; the negative samples also include: event pairs that do not contain causal events and contexts unrelated to the event pairs in the text data of the source language.
3. The method for identifying causal relationships according to claim 2, characterized in that: The loss function is also used to calculate the cross entropy loss of the concatenation of the statement representation and the event pair representation, wherein the statement representation is the embedding representation corresponding to the statement node, and the event pair representation is the embedding representation corresponding to the event pair node.
4. The method for identifying causal relationships according to claim 3, characterized in that: The generating a heterogeneous graph interaction network based on the text data of the source language includes: Parsing the dependency structure of sentences in the text data of the source language to obtain a corresponding dependency tree; Extracting information phrases in the dependency tree and dependency relationships corresponding to the information phrases to obtain a simplified dependency tree with the information phrases; The heterogeneous graph interaction network is generated based on the simplified dependency tree and the text data of the source language, wherein the heterogeneous graph interaction network further includes information phrase nodes corresponding to the information phrases.
5. The method for identifying causal relationships according to claim 4, characterized in that: The heterogeneous graph interaction network obtained by training the text data of the source language identifies the causal relationship between events in the text data to be identified, including: Based on the splicing of the statement representation and the event pair representation in the heterogeneous graph interaction network obtained by training the text data of the source language, the causal relationship between events in the text data to be identified is identified.
6. The method for identifying causal relationships according to claim 5, characterized in that: The obtaining of the initial feature vector of each node in the heterogeneous graph interaction network based on the pre-trained language model includes: Using the pre-trained language model, convert each word in the text data of the source language into an embedding vector; Based on the embedding vector of each word, an initial feature vector of each node in the heterogeneous graph interaction network is obtained, wherein the initial feature vector of the information phrase node is the mean of the embedding vectors of the words related to the current information phrase node plus the embedding vector of the role information of the current information phrase, the initial feature vector of the statement node is the mean of the embedding vectors of the words related to the current statement node, and the initial feature vector of the event pair node is the connection of the embedding vectors of the words corresponding to the events related to the current event pair node.
7. A causal relationship identification device, characterized in that: include: The data acquisition module is configured to: acquire text data to be recognized in a target language; A causal identification module is configured to: identify the causal relationship between events in the text data to be identified based on a heterogeneous graph interaction network trained with text data in a source language, wherein the target language includes a language different from the source language; The training steps of the heterogeneous graph interaction network include: Acquiring text data in the source language; Generating a heterogeneous graph interaction network based on the text data of the source language, wherein the heterogeneous graph interaction network includes: statement nodes corresponding to statements in the text data of the source language, the statement being a sentence containing two events; Acquire an initial feature vector of each node in the heterogeneous graph interaction network based on a pre-trained language model; Based on the initial feature vector, a graph attention network is used to learn the embedded representation of each node in the heterogeneous graph interaction network by optimizing the loss function to train the heterogeneous graph interaction network, wherein the loss function is used to maximize the similarity between positive samples and minimize the similarity between negative samples through comparative learning between positive and negative samples, the positive samples include multilingual statement texts obtained by copying statements containing causal events in the text data of the source language, and the negative samples include multilingual statement texts obtained by copying statements that do not contain causal events in the text data of the source language.
8. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the causal relationship identification method according to any one of claims 1 to 6.
9. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the causal relationship identification method according to any one of claims 1 to 6.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the causal relationship identification method as described in any one of claims 1 to 6 is implemented.
