An event template induction method and system based on large-scale language models

Abstract

The application discloses an event template induction method and system based on a large-scale language model. The method mainly comprises three modules: context-based text conceptualization, confidence-based event template structuring and graph-based event template integration. Specifically, the context-based text conceptualization fully utilizes the generation ability and analogy ability of a large-scale generative pre-training language model through example learning, and converts diversified event natural language expressions into unified conceptualized event template language; the confidence-based event template structuring filters the conceptualized event categories and event argument roles through saliency, reliability and consistency, and thus structures the event template language; and the graph-based event template integration integrates the scattered event templates of the same event through a graph partition clustering algorithm. The application can effectively discover high-quality and high-coverage event templates in an open scene.

Description

Technical Field

[0001] This invention relates to an event template induction method, and more particularly to an event template induction method and system based on a large-scale language model, belonging to the field of natural language processing technology. Background Technology

[0002] Events are a fundamental unit for human understanding and experience of the world. An event represents a specific occurrence involving multiple participants at a particular time and place, such as a wedding. To represent events and model world event knowledge, event templates provide a conceptual, structured, and standardized language to describe event categories and their corresponding event argument roles. Specifically, an event template is a specific framework, such as: "Category: Explosion, Argument Roles: Perpetrator, Victim, Target, Tool". Event templates are crucial for downstream tasks such as event extraction, understanding event relationships, and building event knowledge bases. Due to the importance of event templates, how to automatically discover and construct large-scale, high-quality, and comprehensive event templates urgently needs further research.

[0003] However, due to the openness of event ontology knowledge, the diversity of natural language expressions for events, and the sparsity of event knowledge, event template generalization is not an easy task. First, in real-world applications, the event type system is vast, with new event categories constantly emerging. To address the challenge of open event ontology knowledge, event templates should be able to be automatically generalized and have high coverage across a wide range of domains. Second, such as... Figure 1 As shown, the natural language representation of events is diverse. The same event may be described using different words and structures in different natural languages. To address the challenge of this diversity in event natural language representation, event templates should conceptualize and structure these diverse descriptions into a unified event template language. Finally, due to the principle of economy in language, event expressions are often incomplete. In a specific event description, some event argument roles are frequently omitted. To address the challenge of the scarcity of event knowledge, event templates should aggregate this scattered event template knowledge from different event descriptions.

[0004] Currently, almost all event templates are still manually annotated by human experts, a laborious process, such as in MUC, ACE, and KBP. On the other hand, traditional automated event template induction methods still cannot adequately address the aforementioned challenges of openness, diversity, and sparsity. Specifically, top-down concept linking methods first discover event categories and event argument roles through dependency parsing of natural language expressions, and then link them to existing external concept knowledge bases, such as FrameNet. This method is largely constrained by the quality and coverage of external concept knowledge bases. Bottom-up clustering methods first limit the scope of event templates, such as predefining the common 5W1H (Why, When, Where, Who, What, How), subject-verb-object, or predefining the number of event categories and event argument roles, before performing clustering. This method is heavily limited by the predefined scope of event templates. In summary, how to automatically discover event templates in open scenarios, conceptualize diverse event expressions into a unified event template language, and integrate scattered event template knowledge remain pressing problems to be solved. Summary of the Invention

[0005] To address the challenges of open event ontology knowledge, the diversity of natural language representations of events, and the sparsity of event knowledge, this invention provides an event template induction method and system based on a large-scale language model.

[0006] The technical solution adopted in this invention is as follows: An event template induction method based on a large-scale language model includes the following steps: Context-based text conceptualization: By learning from examples, we can fully leverage the generative and analogical capabilities of large-scale generative pre-trained language models to transform diverse natural language expressions of events into a unified conceptual event template language; Confidence-based event template structuring: Using a conceptual event template language, conceptual event categories and event argument roles are filtered through saliency, reliability, and consistency to obtain structured event templates; Graph-based event template integration: Using structured event templates, scattered event templates of the same event are integrated through graph segmentation and clustering algorithms to obtain the final event template.

[0007] Furthermore, by leveraging the powerful context generation and example learning analogy capabilities of large-scale language models in open scenarios, event template knowledge is extracted from large-scale language models, thereby addressing the challenges of open event knowledge and diverse natural language representation of events in automatic event template induction.

[0008] Furthermore, events are often omitted in natural language expressions. Therefore, event template integration can merge different scattered event templates that describe the same event, thereby obtaining high-quality, complete, and standardized event templates.

[0009] An event template induction system based on a large-scale language model, comprising: The context-based text conceptualization module is used to fully leverage the generative and analogical capabilities of large-scale generative pre-trained language models through example learning, transforming diverse natural language expressions of events into a unified conceptual event template language; The confidence-based event template structuring module is used to filter conceptual event categories and event argument roles through saliency, reliability, and consistency using a conceptual event template language, thereby obtaining structured event templates. The graph-based event template integration module is used to integrate scattered event templates of the same event using structured event templates and graph segmentation clustering algorithms.

[0010] Compared with the prior art, the beneficial effects of the present invention are: 1) It can automatically summarize high-quality, high-coverage event templates.

[0011] 2) Graph-based event template integration significantly improves the clustering performance for discovering events with the same description.

[0012] 3) It has strong generalization ability and can find event templates well in multiple fields and multiple language settings. Attached Figure Description

[0013] Figure 1 This is a summary overview diagram of the process from text to event templates.

[0014] Figure 2 This is a schematic diagram of an event template induction method based on a large-scale language model. Detailed Implementation

[0015] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0016] This invention discloses a method and system for event template induction based on a large-scale language model. The method mainly comprises three modules: context-generated text conceptualization, confidence-based event template structuring, and graph-based event template integration. Specifically, context-generated text conceptualization fully leverages the generative and analogical capabilities of the large-scale generative pre-trained language model through example learning, transforming diverse natural language expressions of events into a unified conceptual event template language. Confidence-based event template structuring filters conceptualized event categories and event argument roles based on saliency, reliability, and consistency, thereby structuring the event template language. Graph-based event template integration integrates scattered event templates for the same event using a graph segmentation and clustering algorithm. This invention can effectively discover high-quality, high-coverage event templates in open scenarios.

[0017] The present invention provides an event template induction method based on a large-scale language model, the key steps of which include: 1) Context-based text conceptualization: By learning from examples, we can fully leverage the generative and analogical capabilities of large-scale generative pre-trained language models to transform diverse natural language expressions of events into a unified conceptual event template language; 2) Confidence-based event template structuring: Conceptualized event categories and event argument roles are filtered through saliency, reliability, and consistency, thereby structuring the event template language; 3) Graph-based event template integration: Integrate scattered event templates of the same event through graph segmentation and clustering algorithms.

[0018] This invention relates to the following key elements: 1. Context-based text conceptualization Given a target text `Text` and a large-scale pre-trained language model, context-based text conceptualization models the transformation process as a text-to-event template generation process: in, These are a series of examples to guide large-scale language models on how to transform natural language descriptions into a unified conceptual event template language. This represents the conceptual event template language. Each example is a <text, event template> pair. The event template language consists of the following structure: "That is, use" Indicates category, It represents multiple argument roles.

[0019] The above text-to-event template generation process is unsupervised. Therefore, it effectively addresses the challenge of open event ontology knowledge and effectively handles the ever-emerging new event categories. By selecting appropriate examples, context-based text conceptualization can also be quickly extended to different domains and languages.

[0020] There are many methods to select suitable examples. To ensure the quality and diversity of the examples, this invention chooses to directly sample the corresponding <text, event template> pairs from existing manually annotated event datasets as examples. Specifically, this invention first groups the sampled <text, event template> pairs from existing manually annotated event datasets according to event type, ensuring, as far as possible, that the event categories within a group of examples are different.

[0021] Finally, in order to retrieve more event ontology knowledge, this invention generates n candidate event template languages ​​for each target text (Text). .

[0022] 2. Event template structuring based on confidence level measurement Given the conceptualized result O of the text, where the j-th sample is , For the generated n candidate event template language, The original natural language text representation of the event.

[0023] First, the event template structure based on confidence measurement is used. This represents all argument roles in n candidate event template languages. To obtain high-quality structured event templates, this invention filters generated event categories and event category roles by measuring saliency, reliability, and consistency. Specifically: Saliency: If an argument role is significant in If an argument role appears frequently in the j-th sample, but its frequency is relatively low across all samples O, it indicates that the argument role better represents the current event and is therefore more significant. Based on the principles of TF-IDF, this invention defines the saliency of the argument role s of the j-th sample as: in, Representative argument role s in Frequency of occurrence in Represents the total number of samples. k This represents the k-th example.

[0024] Reliability: If an argument role appears in n candidate events template language The frequent co-occurrence of argument roles with other argument roles indicates that the large-scale language model consistently selects to predict and generate that argument role during different candidate transformations, thus demonstrating greater reliability. Based on the PageRank concept, this invention defines the reliability of the argument role s of the j-th sample as: in, For hyperparameters, for The total number of characters in the middle of the discussion. , Representative argument roles s and Appearing in the same candidate event template language k Let represent the argument role of the k-th event. This invention sets the initial reliability score for all argument roles to . The algorithm stops and obtains the final reliability estimate when the PageRank algorithm updates for T rounds, or when the change in the reliability scores of all argument roles is less than a threshold.

[0025] Consistency: Since large-scale pre-trained language models may generate event template language that is irrelevant to the input target text during the generation process, this invention estimates the semantic similarity between the generated event template language and the original input target text as the consistency score of the argument role s of the j-th sample.

[0026] in, This is a semantic similarity calculation model based on WordNet, HowNet, and BERT. Representative argument characters s and event categories They belong to the same candidate event template language, C.

[0027] Then, the confidence score of the argument role s of the j-th sample is obtained by combining significance, reliability, and consistency: in, , These are two hyperparameters.

[0028] Finally, for the j-th sample, n candidate event templates are retained. The event category with the highest consistency score is selected as the category for the final structured event template and retained. Argument roles with a medium confidence score exceeding a certain threshold are used as argument roles in the final structured event template.

[0029] 3. Graph-based event template integration Graph-based event template integration first clusters scattered event templates describing the same event, and then integrates the event categories and event argument roles corresponding to the event templates within the same cluster. This invention is based on the observation that if event templates have similar target natural language text expressions, predict the same or synonymous event categories, and have highly overlapping predicted event argument roles, then these event templates are very likely to describe the same event.

[0030] Based on the above observations, given a structured event template O', where the j-th sample is... Graph-based event template integration first constructs a graph to model the similarity between event templates of different samples. Each node in the graph represents an event template corresponding to a sample, and each edge represents the textual expression similarity, event category similarity, and argument role set similarity between two sample event templates. in, Representation diagram The weight of the edge between the i-th node and the j-th node. , , There are three hyperparameters. This is a semantic similarity calculation model based on WordNet, HowNet, and BERT. , , These represent textual expression similarity, event category similarity, and argument set similarity, respectively.

[0031] Given a pre-constructed event template graph, this invention uses the Louvain graph segmentation algorithm to cluster the event templates: in, The event template representing the j-th sample was assigned to the j-th sample. There are 3 clusters. Each cluster represents a specific category of events.

[0032] Finally, given a cluster (Types, Slots), where This represents all aggregated event categories. This represents the class of all aggregated event argument roles. , The graph-based event template integration selects the event category with the highest saliency from the Types as the final event category for this cluster. For argument roles, this invention first uses the Louvain algorithm to discover synonyms in the Slots, and then selects the argument role with the highest saliency for each set of synonyms. Finally, the selected event category and the set of event arguments are output as the final event template.

[0033] The following uses tracking natural language text "Jack and Jone are getting married this weekend." , Jack and Jones The wedding will be held this weekend. , "The company is electing a new project manager this week." , Jack and Jone rented a place for the weekend. The lawn is used for weddings. , "The company's new project needs a new project manager, and the election will be completed this week." The present invention will be further explained using an example.

[0034] Implementation steps: (I) Based on the input natural language text, an event template induction model based on a large-scale language model is constructed. This method includes: conceptualization using context-generated text, structuring event templates using confidence-based metrics, and integrating event templates using graphs.

[0035] (II) Context-based text conceptualization transforms natural language text into an event template language. For example, for a natural language text, we can obtain: "Type: marriage, Slots: groom; bride; time, attendees", "Type: marriage" Marriage, Slots: groom; bride, "Type: wedding, Slots: groom; bride; time" Three candidate event template languages.

[0036] (III) Using confidence-based event template structuring to structure the conceptual event template language, such as structuring the selection of the above three candidate event template languages ​​as follows: "Type: Wedding, Slots: Groom; Bride; Time" .

[0037] (iv) Utilize graph-based event template integration to unify and integrate event templates corresponding to different natural languages. For example, integrate the event templates corresponding to the first, second, and fourth natural languages ​​into a single template. Type: Marriage; Slots: Groom; Bride; Location; Time .

[0038] Another embodiment of the present invention provides an event template induction system based on a large-scale language model that employs the above method, comprising: The context-based text conceptualization module is used to fully leverage the generative and analogical capabilities of large-scale generative pre-trained language models through example learning, transforming diverse natural language expressions of events into a unified conceptual event template language; The confidence-based event template structuring module is used to filter conceptual event categories and event argument roles through saliency, reliability, and consistency using a conceptual event template language, thereby obtaining structured event templates. The graph-based event template integration module is used to integrate scattered event templates of the same event using structured event templates and graph segmentation clustering algorithms.

[0039] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0040] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) that stores a computer program, which, when executed by a computer, implements the various steps of the method of the present invention.

[0041] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. An event template induction method based on a large-scale language model, characterized in that, Includes the following steps: Input natural language text, and fully leverage the generative and analogical capabilities of large-scale generative pre-trained language models through example learning to transform diverse natural language expressions of events into a unified conceptual event template language; By using a conceptual event template language, conceptual event categories and event argument roles are filtered based on saliency, reliability, and consistency to obtain structured event templates; By using structured event templates, the scattered event templates of the same event are integrated through graph segmentation and clustering algorithms to obtain the final event template; The final event template is used for event extraction, event relationship understanding, or event knowledge base construction. The process of converting diverse natural language expressions of events into a unified conceptual event template language includes: Given a target text (Text) and a large-scale pre-trained language model, model the text conceptualization process as a text-to-event template generation process: in, These are a series of examples used to guide large-scale language models in transforming natural language descriptions into a unified conceptual event template language. This represents a conceptual event template language; each example is a <text, event template> pair; the structure of the event template language is: "category On meta-role The process of generating event templates from text is unsupervised. Generate n candidate event template languages ​​for each target text (Text). ; The definitions of significance, reliability, and consistency are as follows: Significance: If an argument role appears frequently among all argument roles in the n candidate event template language, but has a relatively low frequency of occurrence in all samples, it indicates that the argument role can better represent the current event, and is therefore more significant. Reliability: If an argument role frequently co-occurs with other argument roles in n candidate event template languages, it indicates that the large-scale language model consistently selects to predict and generate that argument role during different candidate transformation processes, and therefore has greater reliability. Consistency: The semantic similarity between the generated event template language and the original input target text is estimated as the consistency score of argument roles; The confidence score of the argument role of the sample is obtained by combining the significance, reliability and consistency.

2. The method according to claim 1, characterized in that, The example is obtained by taking the following steps: sampling the corresponding <text, event template> pairs from the existing manually labeled event dataset, grouping them according to event type, and ensuring that the event categories in a group of examples are different during the sampling process.

3. The method according to claim 1, characterized in that, Given the conceptualized result O of the text, where the j-th sample is , For the generated n candidate event template language, use Let n be the candidate events. All argument roles in the template language are represented by the following: The definitions of saliency, reliability, and consistency are as follows: The saliency of the argument role s of the j-th sample is defined as: in, Representative argument role s in Frequency of occurrence in Represents the total number of samples. k This represents the k-th example; The reliability of the argument role s of the j-th sample is defined as: in, For superparameters, for The total number of characters in the meta-analysis. , Representative argument roles s and Appearing in the same candidate event template language k Indicates the first k Each event argument has a specified number of arguments; the initial reliability score for all arguments is set to [value]. ; The consistency score of the argument role s of the j-th sample is: in, This is a semantic similarity calculation model based on WordNet, HowNet, and BERT. Representative argument characters s and event categories They belong to the same candidate event template language, C.

4. The method according to claim 3, characterized in that, The confidence score of argument role s for the j-th sample is obtained by combining the significance, reliability, and consistency: in, , There are two hyperparameters; For the j-th sample, retain n candidate event templates. The event category with the highest consistency score is selected as the category for the final structured event template and retained. Argument roles with a medium confidence score exceeding a certain threshold are used as argument roles in the final structured event template.

5. The method according to claim 1, characterized in that, The method of integrating scattered event templates of the same event using a graph segmentation clustering algorithm first clusters the scattered event templates describing the same event, and then integrates the event categories and event argument roles corresponding to the event templates in the same cluster. The steps include: Given a structured event template O', where the j-th sample is A graph is constructed to model the similarity between event templates of different samples. Each node in the graph represents an event template corresponding to a sample, and each edge represents the textual expression similarity, event category similarity, and argument role set similarity between two sample event templates. in, Representation diagram The weight of the edge between the i-th node and the j-th node. , , There are three hyperparameters. , , These represent textual expression similarity, event category similarity, and argument role set similarity, respectively. Given a pre-constructed event template graph, a graph partitioning algorithm is used to cluster the event templates, with each cluster representing a specific event category. Given a cluster (Types, Slots), where Types represents all aggregated event categories and Slots represents all aggregated event argument roles, select the event category with the highest saliency in Types as the final event category of the cluster. For argument roles, first use a graph partitioning algorithm to find synonyms in Slots, and for each set of synonyms, select the argument role with the highest saliency as the representative. The selected event category and the set of event arguments are output as the final event template.

6. The method according to claim 5, characterized in that, The graph segmentation algorithm is the Louvain algorithm.

7. An event template induction system based on a large-scale language model, employing the method described in any one of claims 1 to 6, characterized in that, include: The context-based text conceptualization module is used to fully leverage the generative and analogical capabilities of large-scale generative pre-trained language models through example learning, transforming diverse natural language expressions of events into a unified conceptual event template language; The event template structuring module based on confidence measurement is used to filter conceptual event categories and event argument roles through saliency, reliability and consistency using a conceptual event template language to obtain structured event templates. The graph-based event template integration module is used to integrate scattered event templates of the same event using a graph segmentation and clustering algorithm, thereby obtaining the final event template.

8. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method described in any one of claims 1 to 6.

Images