Method for identifying relationship between events in unstructured data

Abstract

The invention relates to a method for identifying a relationship between events in unstructured data, and belongs to the technical field of big data processing and data mining. In the template creation step, template sentences required in the model training step and the relationship inference step are created according to rules; in the model training step, a building template is used for finely adjusting the BART model; and inferring the relationship between events by using the model in the model training step and the template sentence in the template creating step. According to the method, the recognition accuracy is improved, multiple event relationships can be recognized at the same time, and when the event relationships are recognized, the accuracy, the recall rate and the comprehensive score of the method are superior to those of a traditional method and a corresponding model thereof; moreover, the method can be easily expanded to the recognition of various relationships by modifying the template under the condition that the format of the output layer is not changed.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing and data mining, and specifically relates to a method for identifying relationships between events in unstructured data. Background Art

[0002] An event is one or more specific actions performed by one or more participants at a specific time and place. There are many kinds of relationships between events, such as cause-effect relationships, sequential relationships, and composition relationships. In recent years, event-based technologies such as machine reading comprehension, process extraction, event graphs, and event prediction have developed rapidly. The relationship between events plays a role in connecting events in these tasks. Text, as one of the important carriers of human information transmission, contains rich event information. Therefore, accurately identifying the relationship between events contained in text has become an important task in the field of natural language processing.

[0003] Chinese patent "CN 113642321 A Causal Relationship Extraction Method and System for the Financial Field". Obtain relevant corpus from unstructured data in the financial field, and annotate sentences in the corpus as "causal" and "no causal relationship"; use annotated sentences to train the BERT model, and expand the central word list by extracting central words; after correcting and processing the central word list, use the central word classification to identify causal relationships, and realize the identification of four types of causal relationships (one cause and one effect, one cause and multiple effects, multiple causes and one effect, and multiple causes and multiple effects).

[0004] Chinese patent "CN 112463970 A A method for extracting causal relationships contained in text based on time relationships". Build a BI-LSTM-GCN neural network model; use BI-LSTM to encode the input text to obtain the hidden features of the text; input the hidden features into GCN to identify the causal relationship between events in the text.

[0005] The Chinese patent "CN 113642321 A Method and system for extracting causal relationships in the financial field" uses the BERT model to find the central word and identify the causal relationship based on the position of the central word. This method ignores the existence of implicit causal relationships where the causal clue trigger words are not obvious. The Chinese patent "CN 112463970A A method for extracting causal relationships contained in text based on time relationships" only trains the model with a small training set data, without introducing external knowledge for knowledge enhancement.

[0006] Currently, no patent has proposed a method for identifying the relationship between events in unstructured data, considering that the relationship between events often lacks clear clues and the existing relevant data sets for this task are small. At the same time, the current methods all build a model to identify one of the relationship between events, causal relationship or sequential relationship, and no patent can use the same model to simultaneously identify multiple event relationships. Summary of the invention

[0007] 1. Technical issues to be resolved

[0008] The technical problem to be solved by the present invention is how to provide a method for identifying the relationship between events in unstructured data, so as to solve the problem of simultaneously identifying multiple event relationships using the same model.

[0009] (II) Technical solution

[0010] In order to solve the above technical problems, the present invention proposes a method for identifying relationships between events in unstructured data, the method comprising the following steps:

[0011] Step 1, "Template creation step": create the template sentences required in the model training step and the relationship inference step according to the rules;

[0012] Step 2, "Model training step": fine-tune the BART model using the template created in the template creation step;

[0013] Step 3, "relationship inference step": use the model in the model training step and the template sentences in the template creation step to infer the relationship between events. This step ends when all candidate event pairs are inferred.

[0014] (III) Beneficial effects

[0015] The present invention proposes a method for identifying the relationship between events in unstructured data. The present invention designs a method for identifying the relationship between events in unstructured data, which improves the recognition accuracy and can simultaneously identify multiple event relationships.

[0016] When the present invention performs event relationship recognition, its precision (P), recall (R) and comprehensive score (F1) scores are better than those of traditional methods and their corresponding models; and the present invention can be easily extended to the recognition of multiple relationships by modifying the template without changing the output layer format. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Creating a flowchart of steps for the template of the present invention;

[0018] Figure 2 This is a flow chart of the training model steps of the present invention;

[0019] Figure 3 This is a training model step example of the present invention;

[0020] Figure 4 It is a flow chart of the relationship inference steps of the present invention;

[0021] Figure 5 This is a case of relationship inference for the present invention;

[0022] Figure 6 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0024] At present, there are three common methods for identifying the relationship between events:

[0025] (1) Artificially formulate some causal trigger words such as "because", "so", "cause" to identify the causal relationship of events. However, this method is not applicable to implicit causal relationships without clear causal clues.

[0026] (2) Use neural network models to identify relationships between events. Due to the lack of existing corpus data, it is difficult to construct an advanced deep learning model without prior knowledge.

[0027] (3) Use knowledge enhancement methods to introduce external knowledge to identify relationships between events. This method constructs an event descriptive knowledge graph induction module and an event relationship graph induction module, and introduces two types of external knowledge, event descriptive knowledge and relationship knowledge, to identify causal relationships. However, when using this method to simultaneously identify multiple event relationships, the external knowledge that needs to be introduced is too complex.

[0028] The present invention proposes a method based on language sequence generation model and template matching to complete the event relationship recognition task, taking the unstructured data containing events as the input sequence and the target sequence filled with candidate event pairs composed of the contained events as the output sequence, so as to train the Bidirectional Autoregressive Transformer (BART) model. Subsequently, the trained BART model is used for the inference process, and the relationship is inferred according to the matching scores between the two events in the candidate event pair and the relationship between each event.

[0029] The present invention specifically solves the following three problems.

[0030] (1) Due to the powerful generalization ability of the BART model, the BART model can be fine-tuned through manually formulated templates, thereby transferring the knowledge that the BART model has learned during the pre-training process to improve the accuracy of event relationship recognition.

[0031] (2) When the present invention is used for event relationship recognition, its precision (P), recall (R) and comprehensive score (F1) scores are better than the above three traditional methods and their corresponding models. Compared with the previous most advanced model, the latent structure induction network model (LSIN), the F1 of the present invention is improved by 0.35.

[0032] (3) Compared with traditional methods, the present invention can be easily extended to the recognition of multiple relationships by modifying the template without changing the output layer format.

[0033] The technical solution of the present invention:

[0034] Event relationship recognition is one of the most challenging tasks in the field of natural language processing. In natural language, there is usually a lack of clear clue trigger words between two events with a relationship. Secondly, this task currently lacks a large amount of annotated data to train the model, resulting in the inability of existing methods to solve this problem well. In response to these two difficulties, the present invention proposes a method based on a language sequence generation model and template matching, which converts the event relationship recognition task into a language sequence generation problem in a sequence-to-sequence framework. A text containing an event is used as an input sequence, and a template sentence filled with candidate event pairs and capable of expressing event relationships is output, so as to fine-tune the BART model. When performing causal relationship inference, the output result of the BART model is matched with the corresponding template, and the candidate event pairs are classified according to the score.

[0035] The following is a detailed introduction to events, event texts, and the relationships between events.

[0036] Event. An event is a specific action taken by a participant at a specific time and place. Each sentence in the unstructured data contains one or more events. The set of events contained in it is defined as E = {e1, e2, …, e m},e i refers to the i-th event, and the number of events is m. Each event consists of words that can represent this event, that is, e i ={t p ,…,t q}.

[0037] Event text, the event text in the present invention uses unstructured data. A piece of unstructured data contains a piece of text. This piece of text consists of multiple sentences separated by punctuation marks. A piece of unstructured data is defined as a sentence set S = {s1, s2, ..., s n}. Among them, s iIt refers to the i-th sentence in the text, and n is the number of sentences. Each sentence can be regarded as a set of multiple characters T = {t1, t2…, t |T|}, where t i It refers to the i-th character in the text, and the number of characters is |T|.

[0038] Relationship between events. Relationship between events can be divided into causal relationship, sequential relationship, composition relationship, etc. It shows the specific development order and internal logic between two events. The event relationship set is defined as R = {r1, r2, ..., r |R|}, the number of relationship categories is |R|.

[0039] The method proposed by the present invention has three steps:

[0040] Step 1, "Template creation step", is the step of creating the template sentences required in the model training step and the relationship inference step according to the rules. After the template is created, this step ends, including the attached Figure 1 ;

[0041] Step 2, "Model training step", is the step of fine-tuning the BART model using the template created in the template creation step. This step ends after the model training is completed. Figure 2 , attached Figure 3 ;

[0042] Step 3, “relationship inference step”, is the step of inferring the relationship between events using the model in the model training step and the template sentences in the template creation step. This step ends when all candidate event pairs are inferred. Figure 4 , attached Figure 5 .

[0043] The relationship between these three steps is as follows: First, in the "template creation step", the candidate event pairs consisting of events contained in the unstructured data are filled into the template according to the rules to form a target sequence. In the subsequent step "model training step", the target sequence created in the "template creation step" is used to fine-tune the BART model, and finally the "relationship inference step" is performed to determine the relationship between events. Figure 6 .

[0044] Here is the detailed process in three steps:

[0045] (1) Template creation steps

[0046] The template sentence of the present invention is composed of candidate event pairs {e i , e j} and the relationship category r between events k Fill in. Among them, different event relations R={r1,r2,…,r |R|} corresponds to different natural characters Y = {y1, y2, ..., y |R|}, for example: r k = "Causality" corresponds to y k = "cause". For the candidate event pair {e i , e j}, using y k Construct the corresponding event relation r k Positive example template And the negative template Outout that represents no relationship between candidate event pairs - . For "e i is they k of e j ”,Outout - For "e i has no relationship with e j ”.

[0047] Take the sentence "During Lohan's stay at the famed Betty Ford Center in 2010, Lohan was accused of attacking a female staffer." as an example. First, construct a positive example. The positive template sentence indicating that "attacking" and "accused" have a "causal relationship" is "'Attacking' is the cause of'accused'". Similarly, the event relationship r k = "successive relationship", the corresponding y k = "next", the template sentence can be set to "(e j )is the nextof(e i )"; event relationship r k = "inclusion relationship", the corresponding y k = "part", the template sentence can be set to "(e i )is the part of(e j )”.

[0048] At the same time, we construct the negative example Outout - , the template sentence indicating that “attacking” and “accused” have no relationship is “'Attacking' has no relationship with'accused'”.

[0049] Finally, we can get a set of template sentences Each template sentence can be represented as a set of characters. i ={o1,o2,…,o d}, the number of characters is d.

[0050] The present invention uses the above method to easily create template sentences for a variety of event relationships, and the rest of the model and algorithm remain unchanged. In addition, since the output of the present invention is a natural sentence rather than a specific annotation sequence. Therefore, the different natural texts Y corresponding to different event relationship categories R are all vocabulary subsets of the pre-trained model BART. Therefore, the method can easily and continuously fine-tune the BART model, transferring label knowledge in a low-cost but efficient way, and has a good transfer learning effect.

[0051] (2) Model training steps

[0052] The present invention uses the labeled data to create the target sequence required for training. Assume that the event pair {e i , e j} has a causal relationship, then use {e i , e j} and y k filling As the target sequence. Similarly, if the event {e i , e j} does not have a relationship, then use {e i , e j Fill Out - As the target sequence. The present invention constructs positive data samples based on all event pairs marked as having a relationship in the data set, and constructs negative data samples by randomly selecting event pairs that are not marked as having a relationship. In order to avoid an imbalanced data set, the ratio of the number of positive and negative samples is 1:1.

[0053] Given an event pair {e i , e j}, the sentences they are in are If the two events are in different sentences, the source sequence is composed of the two sentences. If the two events are in the same sentence, use e i The sentence is used as the source sequence:

[0054]

[0055] First, input the source sequence Input into the BART encoder to obtain the hidden representation H of the sentence:

[0056] H=ENCODER(Input)

[0057] In the Lth step of the decoder, H l and the previous output word o 1:l-1 As input, the attention mechanism is used to produce the representation of the Lth step:

[0058] H l =DECODER(H,o 1:l-1 )

[0059] Generate word o l The conditional probability of is defined as:

[0060]

[0061] in represents the vocabulary size of the pre-trained BART, u h Represents the hidden layer dimension of the encoder. Use the decoder output and the target sequence Output i The cross entropy between is used as the loss function:

[0062]

[0063] (3) Relationship inference steps

[0064] For a piece of text to be detected, first combine all the events in it into candidate event pairs. Then, for each candidate event pair {e i , e j}, fill it in the prepared template. Since the number of events in each document in the dataset is usually less than 100, the set of candidate event pairs to be inferred is not large. Figure 5 As shown, the template sentence used is "e i is the y k ofe j " indicates that the candidate event pair has y k Relationship, "e i has no relationship with e j " indicates that there is no relationship. In the present invention, BART is used as a pre-trained generative language model, and BART fine-tuned by the method in the model training step is used to output each template sentence. i ={o1,o2,…,o d} to score:

[0065]

[0066] By the template sentence set By scoring all the sentences in , we can get the template sentence score of each relationship type. and the score of the template sentence that does not have a relationship f(Outout - ). The relation type corresponding to the template sentence with the highest score is the candidate event pair {e i , e j}The relationship type between them.

[0067] The key points of the present invention are:

[0068] The present invention designs a method for identifying the relationship between events in unstructured data, which improves the recognition accuracy and can simultaneously identify multiple event relationships. There are two problems with the current event relationship recognition task: first, the causal relationship between events often lacks clear causal clues; second, the existing relevant data sets for this task are small. Therefore, the present invention designs a method based on sequence generation model and template matching to complete the event relationship recognition task, and solves the above two problems at the same time.

[0069] Effect of the present invention:

[0070] The present invention verifies the effectiveness of the method by comparing it with the most advanced methods. In the experiment of the present invention, the precision (P), recall (R) and F1 score (F1) of each model on the same data set are used as evaluation indicators.

[0071] The dataset used in this paper is Event Story Line, which contains 258 documents and a total of 7275 events. The "PLOT_LINK" mark is used in Event Story Line to mark the generalized causal relationship between events, that is, the occurrence of one thing causes or proves the occurrence of another thing. There are 5,519 "PLOT LINK" in this dataset, including 117 clear and explicit causal relationships.

[0072] In order to verify the effect on the multi-category event relationship recognition task, the present invention also uses the EEOK dataset that annotates the relationships between multiple events. This dataset annotates the Baidu Encyclopedia event entry corpus in the fields of explosion, fire, geological disaster, traffic accident, personal injury, etc., including a total of 221 documents, 1532 event pairs with sequential relationships, and 381 event pairs with causal relationships.

[0073] Table 1 Experimental results of different models and the present invention on the Event Story Line dataset

[0074]

[0075]

[0076] The proposed method is verified on the Event Story Line dataset and compared with other models. The experimental results are shown in Table 1. The proposed method can maintain a precision rate P of 0.8876 while the recall rate R reaches 0.8765, and the final F1 score is 0.8758. The effectiveness of the present invention can be verified by comparing with other models.

[0077] First, among all the models, the recall rate R of the present invention is second only to the OP model, indicating that it can find causal event pairs more comprehensively. This is because all candidate event pairs are traversed and judged during the inference process.

[0078] At the same time, after comparing with the BiLSTM and ITP models, the present invention outperforms these two deep neural network-based models in all three evaluation indicators. This shows that the present invention is better than the deep neural network model trained only by existing training data. It also shows that the present invention can effectively use the knowledge contained in the BART model to improve the accuracy of identifying event relationships.

[0079] Moreover, the present invention is better than the three knowledge enhancement-based methods, which proves that the template-based method proposed in the present invention can make good use of the knowledge contained in the pre-trained model BART, thereby better transferring knowledge from the outside, thereby improving the effect on the event causal relationship recognition task.

[0080] Table 2 Sentence templates corresponding to different event relations

[0081] Relationship Category template causation <![CDATA[(e i )isthecauseof(e j )]]> Succession relationship <![CDATA[(e j )isthenextof(e i )]]> No relationship <![CDATA[(e i )hasnorelationshipwith(e j )]]>

[0082] Table 3 Evaluation index scores of each relationship category when simultaneously identifying causal relationships and sequential relationships

[0083] Relationship Category P R F causation 0.90090090 0.90909091 0.90497738 Succession relationship 0.89772727 0.86813187 0.88268156 No relationship 0.93421053 0.93729373 0.93574959

[0084] Table 4 Evaluation index scores of each relationship category when only causal relationships are identified

[0085] Relationship Category P R F causation 0.91251171 0.91205212 0.91214506 No relationship 0.92976589 0.91749175 0.92358804

[0086] In order to verify the feasibility of using the present invention to identify multi-category event relationships, the present invention conducted experiments on multi-category event relationship identification tasks. The experimental results of simultaneously identifying causal relationships and sequential relationships are shown in Table 3, and the experimental results of only identifying causal relationships are shown in Table 4. The experimental results show that when performing multi-category event relationship identification, the performance of the present invention is not much different from the performance when only "causal relationships" are identified. Moreover, when simultaneously identifying "causal relationships" and "sequential relationships", the recognition effect of "unrelated" event pairs is improved. This may be because when only "causal relationships" are identified, the model will identify some event pairs with "sequential relationships" as "unrelated" event pairs, which will obviously have a bad impact on the final classification results.

[0087] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for identifying relationships between events in unstructured data, characterized in that: The method comprises the following steps: Step 1, "Template creation step": Create the template sentences required in the model training step and the relationship inference step according to the rules; Step 2, "Model training step": fine-tune the BART model using the template created in the template creation step; Step 3, "relationship inference step": use the model in the model training step and the template sentences in the template creation step to infer the relationship between events. This step ends when all candidate event pairs are inferred.

2. The method for identifying relationships between events in unstructured data according to claim 1, characterized in that: The step one comprises: The template sentence consists of candidate event pairs {e i , e j } and the relationship category r between events k Fill; where different event relations R = {r1, r2, ..., r |R| } corresponds to different natural characters Y = {y1, y2, ..., y |R| }; For the candidate event pair {e i , e j }, using y k Construct the corresponding event relation r k Positive example template and the negative template Outputi representing no relationship between candidate event pairs - ; For "e i is the y k of e j ”,Output - For "e i has no relationship with e j ”; Finally, we get a set of template sentences Each template sentence is represented as a set of characters. Output i ={o1,o2,…,o d }, the number of characters is d.

3. The method for identifying relationships between events in unstructured data according to claim 2, characterized in that: In step 2, the target sequence required for training is created using the labeled data; assuming that the event pair {e i , e j } has a causal relationship, then use {e i , e j } and y k filling as the target sequence; similarly, if the event {e i , e j } does not have a relationship, then use {e i , e j } Fill Output - as the target sequence.

4. The method for identifying relationships between events in unstructured data according to claim 3, characterized in that: Positive data samples are constructed based on all event pairs marked as having a relationship in the dataset, and negative data samples are constructed by randomly selecting event pairs that are not marked as having a relationship.

5. The method for identifying relationships between events in unstructured data according to claim 4, characterized in that: The ratio of the number of positive and negative samples is 1:

1.

6. The method for identifying relationships between events in unstructured data according to claim 3, characterized in that: Given an event pair {e i , e j }, the sentences they are in are If the two events are in different sentences, the source sequence is composed of the two sentences; if the two events are in the same sentence, e is used. i The sentence is used as the source sequence:

7. The method for identifying relationships between events in unstructured data according to claim 6, characterized in that: In the step 2, the fine-tuning process includes: First, input the source sequence Input into the BART encoder to obtain the hidden representation H of the sentence: H=ENCODER(Input) In the Lth step of the decoder, H l and the previous output word o 1:l-1 As input, the attention mechanism is used to produce the representation of the Lth step: H l =DECODER(H,o 1:l-1 ) Generate word o l The conditional probability of is defined as: Where |V| represents the vocabulary size of the pre-trained BART, u h Represents the hidden layer dimension of the encoder; uses the decoder output and the target sequence Output i The cross entropy between is used as the loss function:

8. The method for identifying relationships between events in unstructured data according to any one of claims 3 to 7, characterized in that: In the step 3, for a piece of text to be detected, firstly, all the events therein are grouped into candidate event pairs, and then, for each candidate event pair {ei, ej}, it is filled in the prepared template; The template sentence used is "e i is the y k of e j " indicates that the candidate event pair has y k Relationship, i has no relationship with e j " indicates no relationship.

9. The method for identifying relationships between events in unstructured data according to claim 8, characterized in that: In step 3, BART is used as a pre-trained generative language model, and BART fine-tuned by the method in the model training step is used to output each template sentence. i ={o1,o2,…,o d } for scoring:

10. The method for identifying relationships between events in unstructured data according to claim 9, characterized in that: By the template sentence set Score all the sentences in the template to get the score of each relationship type. and the score of the template sentence that does not have a relationship f(Outout - ), the relation type corresponding to the template sentence with the highest score is the candidate event pair {e i , e j }The relationship type between them.

Images