A zero-shot event detection method based on a text entailment recognition model

Abstract

The application discloses a zero sample event detection method based on a text entailment recognition model, and comprises the following steps: S1, obtaining initial text data from a NYT corpus and performing pretreatment, and obtaining separate sentence texts after the treatment; S2, using a WordNet to expand seed keywords; S3, using the expanded keyword set to screen the text data, when a text sentence contains keywords corresponding to a certain event type, the sentence has a considerable probability to express the event type; S4, using a text entailment recognition model to mark the data screened in the step S3; S5, further using the text entailment recognition model to screen the data obtained in the step S4; S6, using the data obtained in the step S5 to train the text entailment recognition model, and using the trained model to perform event detection on an ACE data set. The application focuses on optimizing the application of the text entailment recognition model in zero sample event extraction.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of natural language processing, and particularly relates to a zero-shot event detection method based on a text entailment recognition model. BACKGROUND

[0002] The event extraction method based on deep learning is currently the mainstream and the best performing method, however, these methods are highly dependent on high-quality labeled data, and manual labeling data needs to consume expensive costs, at the same time, the data labeled by experts in various fields is often only applicable to the event extraction task in a single field, and is difficult to apply to cross-domain event extraction tasks. In view of the dependence of the existing event extraction method on the labeled data and the current situation that manual labeling data needs to consume expensive costs, the zero-shot event detection method based on zero-shot learning does not use the labeled data of the existing data set, so as to reduce the high cost of manual labeling data and be applicable to research in different fields, and has great development prospect and application value. The technology of the application belongs to the field of natural language processing.

[0003] The event detection research under the event extraction task has high application value for mining key information in the current social big data. It can be found from the comprehensive analysis of the current zero-shot event detection research status that some event detection methods do not use complete sentence semantic information in terms of the application degree of text information, and some methods using knowledge base information to obtain more labeled data are obviously limited by the knowledge base. The existing zero-shot event detection method has a large gap compared with the supervised method on the ACE data set. The present method combines the development trend of the current event detection research, mainly aims at the dependence of the current supervised event detection method on high-quality labeled data, and the problem that manual labeling data needs to consume expensive costs and is difficult to solve the cross-domain problem, focuses on optimizing the application idea of the text entailment recognition model in the zero-shot event extraction research, and explores and proposes a new zero-shot event detection method based on a text entailment recognition model.

[0004] Qing Lyu et al. (Lyu Q, Zhang H, Sulem E, et al. Zero-shot event extraction via transfer learning: Challenges and insights [A]. Proceedings of ACL [C]. 2021: 322-332.) from University of Pennsylvania first used semantic role labeling (SRL) as a preprocessing step to divide the sentence into text segments, and designed a description text for each event type of ACE, and then applied a text entailment recognition model, taking each text segment of the sentence as the premise and the description text designed for each event type as the hypothesis, and inputting them into the text entailment recognition model. The model will output the probability of the premise entailing the hypothesis. If the probability output by the text entailment recognition model exceeds the set threshold, the text segment as the premise will be regarded as the trigger of the event type corresponding to the hypothesis. The results of the role semantic labeling model have a great influence on this method. In order to identify the trigger of the sentence, the method splits the sentence into text segments, which damages the integrity of the sentence semantics. Moreover, only the semantic information at the text segment level is used, and the semantic role labeling method used will cause cascading errors.

[0005] Yubo Chen et al. (Chen Y, Liu S, Zhang X, et al. Automatically labeled data generation for large scale event extraction [A]. Proceedings of ACL [C]. 2017: 409-419.) from University of Chinese Academy of Sciences proposed to use knowledge bases such as Freebase, Wikipedia, and FrameNet to automatically generate large-scale labeled data. The rule of automatically labeled data is that any sentence containing all the arguments in the knowledge base is considered to contain the event corresponding to the argument in the knowledge base. Since the event definitions of the knowledge base and the event extraction dataset are inconsistent, the event types of the labeled data generated by the knowledge base may not cover all the event types of the event extraction dataset. The event type categories that can be labeled by this method will be limited by the knowledge base itself.

[0006] Ying Lin et al. (Lin Y, Ji H, Huang F, et al. A joint neural model for information extraction with global features [A]. Proceedings of ACL [C]. 2020: 7999-8009.) from the University of Illinois at Urbana-Champaign proposed a joint neural framework OneIE to perform end-to-end information extraction with global constraints, which does not use local classifiers to predict individual knowledge elements, but extracts the globally optimal information network for the input sentence. This method is the best existing supervised event extraction method, but it relies on high-quality labeled data.

[0007] A comprehensive analysis of the current status of existing zero-shot event detection methods can find that some event detection methods do not use complete sentence semantic information in terms of the degree of application to text information, and some methods that use knowledge base information to obtain more labeled data are obviously limited by the knowledge base, as shown in Figure 1 The application mode of the existing zero-shot event detection method to the text entailment recognition model, these studies simply use the text entailment recognition model as a part of their method, and do not fully utilize the text understanding capabilities of the text entailment recognition model. The existing zero-shot event detection method performs relatively generally on the ACE dataset and has a large gap compared with the supervised method. SUMMARY

[0008] To solve the above problems, the present application provides a zero-shot event detection method based on a text entailment recognition model. Combining the development trend of current event detection research, the present application mainly aims at the dependence of the supervised event detection method on high-quality labeled data, and the need for expensive cost and the difficulty in solving the cross-domain problem of manually labeled data. The present application can automatically obtain relatively high-quality labeled data from large-scale unlabeled data, focuses on optimizing the application of the text entailment recognition model in the zero-shot event extraction research, and explores and proposes a new zero-shot event detection method that can achieve the best effect of the existing zero-shot method on the ACE dataset, which is the main dataset of event detection, and forms a certain competition with the supervised method. As long as a set of hypothesis templates can be designed for the event types in the corresponding field, the present application can be applied to event detection tasks in any background (such as medical treatment, finance, etc.), and has good application value.

[0009] The present application is realized by at least one of the following technical solutions.

[0010] A zero-shot event detection method based on a text entailment recognition model, comprising the following steps:

[0011] S1, obtaining initial text data from the NYT corpus and preprocessing, obtaining separate sentence texts after processing;

[0012] S2, using WordNet to expand seed keywords;

[0013] S3, using the expanded keyword set to filter text data, when a text sentence contains keywords corresponding to a certain event type, this sentence has a considerable probability of expressing a certain event type;

[0014] S4, using a text entailment recognition model to label the data filtered in step S3;

[0015] S5, further using a text entailment recognition model to filter the data obtained in step S4;

[0016] S6, using the data obtained in step S5 to train a text entailment recognition model, and using the trained model to perform event detection on the ACE dataset.

[0017] Further, the zero-shot event detection dataset is a text dataset obtained from the NYT corpus.

[0018] Further, the preprocessing step includes reading data from XML format files and dividing segmented text into individual sentences.

[0019] Further, the preprocessing is performed using the Spacy text processing library.

[0020] Further, the input of the text entailment recognition model includes premise and hypothesis, and the text entailment recognition model is used to label the text sentence by taking the text sentence as the premise and the hypothesis templates corresponding to the event types to which all keywords contained in the text sentence belong as the hypothesis.

[0021] Further, the hypothesis template is designed by artificially referring to the definition of the 33 event types of ACE.

[0022] Further, the input of the text entailment recognition model is the text sentence and the hypothesis template corresponding to a certain event type, and when any one of the text sentence and the hypothesis in the hypothesis template is input, the text entailment recognition model outputs an entailment probability exceeding a set threshold, then the sentence is labeled as the corresponding event type.

[0023] Further, WordNet expands keywords in three ways:

[0024] The first is the word in the same synonym set as the seed keyword, which means the near-synonym with similar semantics as the keyword;

[0025] The second is a hyponym set of the synonym set to which the seed keyword belongs;

[0026] The third is an instance hyponym set of the synonym set to which the seed keyword belongs.

[0027] Further, in step S5, the text sentence containing the certain keyword is input into the text entailment recognition model with different word meanings of the keyword, wherein the word meaning most similar to the event type definition to which the keyword belongs is input as the first input, and when the text sentence has the highest entailment probability with the first input word meaning of the keyword, the labeling data is retained, otherwise it is filtered out.

[0028] Further, the text entailment recognition model is obtained by training a DeBERTa V2 xlarge model from the MNLI data set.

[0029] Compared with the prior art, the present application has the following beneficial effects:

[0030] The text entailment recognition model is used as a weakly supervised source to label large amounts of unlabeled data for the first time, and the overall data labeling process is optimized by effectively using keyword screening and expanding keywords and other strategy methods, and corresponding measures are taken to further enhance the ability of the text entailment recognition model to screen data. The overall data labeling method proposed can obtain relatively high-quality labeled data from a corpus containing a large amount of text data. Moreover, the method can be applied to event detection tasks in any background (such as medical treatment, finance, etc.) as long as a set of hypothesis templates is designed for the event types in the corresponding field, which will effectively help the development of zero-shot event detection and even event extraction tasks in different fields.

[0031] In addition to using the text entailment recognition model to label a large amount of unlabeled text, the method also differs from the use mode of the text entailment recognition model in previous studies, and effectively enhances the processing capability of the model in the event detection task based on the fine-tuning of the text entailment recognition model based on the constructed data set. The data form for fine-tuning the text entailment recognition model is reasonably designed, and the performance of the zero-shot event detection method on the ACE data set is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Flowchart of the text entailment recognition model of the prior art zero-shot event detection method for the embodiment;

[0033] Figure 2 Flowchart of the application mode of the text entailment recognition model for the embodiment;

[0034] Figure 3A flowchart is constructed for the example data set. DETAILED DESCRIPTION

[0035] In order to make the purposes, technical features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in combination with the drawings and examples.

[0036] It should be noted that, unless otherwise defined, all technical and scientific terms used in the present application have the same meanings as commonly understood by those skilled in the art to which the present application belongs. The terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0037] As Figures 2 and 3 , the zero-shot event detection method based on the text entailment recognition model of the embodiment aims to maximize the event detection capability of the text entailment recognition model. For the first time, the text entailment recognition model is used as a weak supervision source to label large-scale unlabeled data. In this process, the overall data labeling process is optimized by effectively using keyword screening, expanding keywords and other strategy methods. At the same time, by analyzing the labeled data obtained and taking corresponding measures, the ability of the text entailment recognition model to screen data is further enhanced. The overall data labeling method proposed can obtain relatively high-quality labeled data from a corpus containing a large amount of text data. The data set constructed according to this method is the premise and basis for the subsequent event detection and trigger word recognition task research. At the same time, the method of the present application is not only applicable to the related fields involved in the ACE data set. In fact, as long as a set of hypothesis templates can be designed for the event types in the corresponding field, the method of the present application can be applied to event detection tasks in any background (such as medical care, finance, etc.) in theory. This will effectively help the development of zero-shot event detection and even event extraction tasks in different fields, which specifically includes the following steps:

[0038] S1, constructing a zero-shot event detection data set: obtaining initial text data from the NYT corpus and preprocessing, obtaining single sentence text after processing; The contents of preprocessing include removing part of the messy information, dividing the segmented text into single sentences, etc. The text processing library such as Spacy is used in the embodiment.

[0039] As Figure 3 shown, first, the text data obtained, the embodiment reads the initial text from the NYT (The New York Times Annotated Corpus) corpus.

[0040] Because the text data in the NYT corpus is stored in XML format, the first step is to preprocess the format file to obtain natural segmented text. In addition to the text information of news reports, the corpus also contains a lot of messy information, so the initial text read out also needs to be preprocessed.

[0041] The content of the preprocessing step includes reading data from the XML format file (at this time the whole paragraph of text data is obtained, and the xml.dom.minidom package in python is used to read the relevant program), dividing the segmented text into individual sentences (Spacy text processing library is used).

[0042] S2, using WordNet to expand the seed keywords, using the expanded keyword set to filter the text data, when a text sentence contains keywords corresponding to the A type event, the sentence has a considerable probability of expressing A event type. The extended seed keywords are used to solve the problem of reducing the knowledge base caused by keyword expansion. Because using keywords to filter means that text sentences that do not contain keywords will not be selected into the data set.

[0043] The text data can be obtained from the NYT corpus by the preprocessing. In fact, the large amount of text data contained in the NYT corpus is mixed with a lot of text without substantive information. If the text entailment recognition model is directly applied to label these data at this time, not only will it cause waste of computing resources, but also it will consume a lot of time, and the quality of the labeled data obtained may also have a large fluctuation. Therefore, in order to obtain high-quality labeled data more efficiently, the application considers applying keywords to help filter text data. As long as a word in a text sentence contains any keyword, the corresponding text sentence will be retained. It can be understood that when a text sentence contains keywords corresponding to the A type event, the sentence has a considerable probability of expressing the A event type.

[0044] Using keywords to pre-screen text data can improve the efficiency of screening data, but the use of keywords will lead to the reduction of knowledge base. If a text sentence does not contain any keyword, this sentence will never be selected into the data set, and in fact, there are still many sentences related to the required event type in the large amount of text data screened out without containing keywords, that is, keywords help to screen text data, but also limit the range of available text in the data set, which brings waste of resources and incomplete information sources, so the application applies WordNet to help expand keywords, WordNet is a large English vocabulary database, nouns, verbs, adjectives and adverbs are combined into synsets in WordNet.

[0045] Seed keywords are obtained from the article (Zhang H, Wang H, Roth D. Zero-shot Label-Aware Event Trigger and Argument Classification [A]. Findings of ACL [C]. 2021: 1331-1340.) (The problem of knowledge base reduction caused by keyword expansion is that using keywords to screen means that text sentences that do not contain keywords will not be selected into the data set.

[0046] Specifically, each seed keyword is mapped to a synset in WordNet, and then the keywords are expanded in three ways.

[0047] The first is the word in the same synset as the seed keyword, which means the near-synonyms with similar semantics as the keyword;

[0048] The second is the hyponyms synset of the synset to which the seed keyword belongs;

[0049] The third is the instance hyponyms synset of the synset to which the seed keyword belongs.

[0050] Expanding keywords through WordNet to obtain multiple new keywords can largely solve the problem of reducing the knowledge base caused by using keywords to screen data.

[0051] S3, using a text entailment recognition model to label the data screened in step S2;

[0052] After the preprocessing step and the keyword screening, a series of text sentences containing keywords are obtained, but these data cannot be directly used to fine-tune the pre-trained language model because these data still lack event type annotation information. The present application applies a text entailment recognition model to label these data.

[0053] As an embodiment, the text entailment recognition model used in the present embodiment is microsoft / deberta-v2-xlarge-mnli (open source). The model is obtained by fine-tuning the DeBERTa V2 xlarge model (pre-trained language model, open source) on the MNLI dataset. The MNLI (The Multi-Genre Natural Language Inference Corpus) is a multi-type natural language inference database, and its training set contains 392702 samples.

[0054] The input of the text entailment recognition model includes premise and hypothesis. The text sentence is used as the premise, and the hypothesis template corresponding to the event type to which all the keywords contained in the text sentence belong is used as the hypothesis. The input of the text entailment recognition model is the text sentence and the hypothesis template corresponding to a certain event type. When any one of the text sentence and the hypothesis in the template is input, if the entailment probability output by the text entailment recognition model exceeds the set threshold value, the sentence is labeled as the corresponding event type and added to the OTEL (Our Textual Entailment Labeled Dataset) dataset. The dataset is used to fine-tune the text entailment recognition model, and the ACE dataset is only used to detect the event detection ability of the fine-tuned model, i.e., the dataset constructed by the present application is used to train the model, and the ACE dataset is used to detect the performance of the model. The threshold value set in the present embodiment is 0.9.

[0055] The hypothesis template is designed by artificially referring to 33 event types of ACE (Automatic Content Extraction, a general dataset in the field of event extraction, reference literature: Doddington G R, Mitchell A, Przybocki M A, et al. The automatic content extraction (ACE) program-tasks, data, and evaluation [A]. Proceedings of LREC [C]. 2004: 837-840.).

[0056] As an embodiment, it is assumed that the text sentence is Jack and Mary got married in New York last year, witnessed by their relatives and friends. (Jack and Mary got married in New York last year, witnessed by their relatives and friends after 5 years of love.) The text sentence as assumed above contains the keyword married, which belongs to the keyword set corresponding to the Life:MARRY event type, and the Life:MARRY event type has a total of 8 different assumption templates. In the process of labeling by using the text entailment recognition model, the sentence will be input into the text entailment recognition model respectively, and 8 entailment probabilities will be output. As long as one of the entailment probabilities exceeds the set threshold, the text sentence is labeled as the Life:MARRY event type, and the text sentence is added to the OTEL (Our Textual Entailment Labeled Dataset) dataset.

[0057] S5, because the quality of the labeled data obtained in the S4 step still has room for improvement, the text entailment recognition model is further used to screen the data obtained in the S4 step for the problem of multiple word senses of a word.

[0058] After labeling part of the data, the quality of the data is analyzed. Specifically, it is judged by humans whether the event type labeled by the model is the event actually expressed by the corresponding text sentence. For example, the model labels the data as an A event type, but the human judges believe that the data does not express an A event type, and the labeling of the data is considered to be wrong.

[0059] The specific process is that after a part of the data (about 6500 text sentences) is labeled by using the text entailment recognition model, the quality of the data is analyzed by human judgment. The specific method is to randomly select 20 sentences from the data of each event type for analysis (if there are less than 20, select all the data of the event type), but the quality of the obtained data still has a large room for improvement, and many labeling errors may be caused by the multiple word senses of the keywords. Therefore, the method of the application further screens the labeled data obtained by using the text entailment recognition model. Because the input of the text entailment recognition model is two texts of premise and assumption, the text sentence is taken as the premise, and the corresponding assumption is replaced by the multiple word senses of the keyword. Thus, it can be judged which word sense of the keyword has the highest entailment probability with the text sentence. The analysis of the quality of the obtained data shows that the data labeled by the above steps has good quality.

[0060] The application uses the text entailment recognition model twice, the first time is to directly label the data using the model, and the second time is to filter the data that has been labeled in combination with different word meanings of the keywords.

[0061] The process of further filtering the labeled data obtained by using the text entailment recognition model the second time: the text sentence containing a certain keyword will be input into the text entailment recognition model respectively with different word meanings of the keyword, wherein the word meaning most similar to the event type definition to which the keyword belongs will be the first input, and when the text sentence has the highest entailment probability with the first input word meaning of the keyword, the labeled data will be retained, otherwise it will be filtered out.

[0062] For example, for the word close in the keyword set, through the query in the Oxford Advanced Learner's Dictionary, it mainly contains the four word meanings shown in Table 1.

[0063] Table 1: Word meanings of the word close

[0064]

[0065]

[0066] The word close belongs to the keyword set corresponding to the Business:End-Org event type, and the meaning of this event type is to mean the bankruptcy of a company or organization. This means that a text sentence expressing the Business:End-Org event type, when close is the keyword of the sentence, the word close should express the word meaning of“A company begin to fail or get weaker or to come to an end.” rather than other meanings.

[0067] The specific method is to find the main word meanings of each keyword in the keyword set through the Oxford dictionary, then take the text sentence labeled by the first use of the text entailment recognition model as the premise, take each word meaning of the keyword contained in the sentence as the hypothesis, and take the required word meaning as the first input hypothesis. If the text entailment recognition model outputs the maximum entailment probability of the first hypothesis of the corresponding keyword of the text sentence, it is finally considered that the text sentence expresses the event type corresponding to the keyword;

[0068] For example, as Figure 2As shown, after the data set is constructed, the overall zero-shot event detection method is designed by fine-tuning the text entailment recognition model based on the data set, and related experiments are performed. Specifically, the event detection experiment on the ACE data set proves that the method can greatly improve the performance of the existing zero-shot event extraction method in the event detection task (the highest accuracy value of the existing supervised method (which depends on high-quality labeled data) on the ACE data set is 72.2%, the highest accuracy value of the existing zero-shot event detection method on the ACE data set is 53.7%, and the method can achieve an accuracy value of 68.8%), which shows that the data set of the application can help improve the event detection capability of the text entailment recognition model, and the overall method can be applied to event detection tasks in different knowledge fields.

[0069] Table 2 is the event type involved in the ACE data set, which is related to the field of legal judgment, etc. The data set constructed and used to fine-tune the text entailment recognition model of the application is related to the 33 event types.

[0070] Table 2 ACE data set event type and corresponding seed keyword

[0071]

[0072]

[0073]

[0074] The zero-shot event detection method based on the text entailment recognition model of the application mainly includes:

[0075] (1) The text entailment recognition model is used as a weak supervision source for the first time, which can obtain relatively high-quality labeled data from a large amount of unlabeled corpus, and as long as a set of hypothesis templates is designed for the event type in the corresponding field, the method can be applied to any research background in theory.

[0076] (2) The use mode of the text entailment recognition model is different from that of the previous method, and the method of fine-tuning the text entailment recognition model is used to maximize the text understanding capability of the model, and the experiment proves that the method can greatly improve the performance of the zero-shot event detection method on the ACE data set and form a certain competition with the supervised method.

[0077] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in some detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, and some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A zero-shot event detection method based on a text implication recognition model, characterized in that: The following steps are involved: S1. Obtain initial text data from The New York Times Annotated Corpus and preprocess it to obtain individual sentence texts. S2, using WordNet to expand seed keywords; S3. Use the expanded keyword set to filter text data. When a text sentence contains keywords corresponding to a certain event type, the sentence has a high probability of expressing a certain event type. S4. Use the text implication recognition model to annotate the data filtered in step S3. The input of the text implication recognition model includes a premise and a hypothesis. The text implication recognition model is used to annotate text sentences. The text sentence is used as the premise, and the hypothesis template corresponding to the event type of all keywords contained in the text sentence is used as the hypothesis. The input of the text implication recognition model is a text sentence and a hypothesis template corresponding to a certain event type. When the implication probability output by the text implication recognition model exceeds the set threshold when the text sentence or any one of the hypotheses in the hypothesis template is used as input, the sentence is marked as the corresponding event type. S5. Further use the text implication recognition model to filter the data obtained in step S4; In step S5, a text sentence containing a keyword is used to input different meanings of the keyword into the text implication recognition model. The meaning that is most similar to the event type definition of the keyword is used as the first input. If the text sentence has the highest implication probability with the first input meaning of the keyword, the labeled data will be retained; otherwise, it will be filtered out. S6. Use the data obtained in step S5 to train a text implication recognition model, and use the trained model to perform event detection on the Automatic Content Extraction dataset.

2. The zero-shot event detection method based on the text implication recognition model according to claim 1, characterized in that: The zero-shot event detection dataset is a text dataset obtained from The New York Times Annotated Corpus.

3. The zero-sample event detection method based on the text implication recognition model according to claim 2 is characterized in that: The preprocessing step involves reading data from XML format files and dividing the text into individual sentences.

4. The zero-sample event detection method based on the text implication recognition model according to claim 1 is characterized in that: The preprocessing is performed using the Spacy text processing library.

5. The zero-shot event detection method based on text implication recognition model according to claim 1, characterized in that: Assume that the template is manually designed with reference to the 33 event type definitions of Automatic Content Extraction.

6. The zero-shot event detection method based on text implication recognition model according to claim 1, characterized in that: WordNet expands keywords in three ways: The first type is words that are in the same synonym set as the seed keyword. The same synonym set refers to synonyms with similar semantics to the keyword; The second type is the hyponym set of the synonym set to which the seed keyword belongs; The third type is the instance hyponym set of the synonym set to which the seed keyword belongs.

7. The zero-shot event detection method based on a text implication recognition model according to any one of claims 1 to 6, characterized in that: The text entailment recognition model is obtained by training the DeBERTa V2 xlarge model using the Multi-Genre Natural Language Inference Corpus dataset.

Images