Large language model data enhancement method and device for event extraction

By using data enhancement methods based on event mode and frequency sampling in event extraction, high-quality and diversified data are generated, which solves the problems of insufficient samples and insufficient quality in the prior art, and improves the generalization ability and data of the model quality.

CN118551194BActive Publication Date: 2024-11-19AEROSPACE INFORMATION RES INST CAS
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411025466.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-19
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

The existing data augmentation methods based on large language models generate insufficient samples, resulting in event extraction models being too dependent on certain specific patterns, and the generated samples are not as good as real samples, containing unnatural or wrong language structures.

Method used

By obtaining event mode from the initial dataset, the sampling probability is determined based on the frequency of event categories, the sampling data is input into the large language model as a context prompt, the data is generated, and the high-quality target dataset is filtered through preset conditions to prevent the model from being overfitted. Improve generalization ability.

Benefits of technology

The generated data is more diverse and quality, avoiding the model's dependence on specific patterns, improving the generalization ability of event extraction models, and preventing overfitting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118551194B_ABST
    Figure CN118551194B_ABST
Patent Text Reader

Abstract

The present invention provides a large language model data enhancement method and device for event extraction. The method includes: obtaining N event patterns from an initial data set, the event pattern including an event category and an argument role corresponding to the event category; determining the sampling probability of each event category in the N event patterns based on the frequency of occurrence of each event category in the initial data set, the frequency being inversely proportional to the sampling probability; sampling from the initial data set based on the sampling probability of each event category in the N event patterns to obtain sampled data; inputting the sampled data as a first contextual prompt and a prompt template into a large language model, outputting generated data, the prompt template being used to instruct the large language model to generate generated data associated with the prompt template according to the first contextual prompt; and filtering data that meets preset conditions from the generated data, and storing it in the initial data set to obtain a target data set.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a large language model data enhancement method and device for event extraction. Background Art

[0002] Information extraction is an important task in the field of natural language processing. Information extraction refers to the task of extracting structured information from raw text. Event extraction is a complex task in the field of information extraction, and its goal is to identify and extract specific events and related information from text. However, due to the complexity, annotation cost, and context dependency of event extraction tasks, the available data sets are often relatively limited, and insufficient sample size has become a major challenge that limits model performance. Data augmentation is an effective method that can help solve this problem and improve the performance of event extraction models.

[0003] In the process of implementing the concept of the present invention, the inventors found that the samples generated by the current data enhancement method based on large language models are not diverse enough, resulting in the event extraction model being overly dependent on certain specific patterns. In addition, the quality of the generated samples is not as good as the real samples. For example, the generated samples contain some unnatural or erroneous language structures. Summary of the invention

[0004] In view of this, the present invention provides a large language model data enhancement method and device for event extraction.

[0005] One aspect of the present invention provides a large language model data enhancement method for event extraction, comprising: obtaining N event patterns from an initial data set, wherein the event pattern includes an event category and an argument role corresponding to the event category, and N is an integer greater than or equal to 1; determining the sampling probability of each event category in the N event patterns based on the frequency of occurrence of each event category in the initial data set, wherein the frequency is inversely proportional to the sampling probability; sampling from the initial data set based on the sampling probability of each event category in the N event patterns to obtain sampled data; inputting the sampled data as a first contextual prompt and a prompt template into a large language model, and outputting generated data, wherein the prompt template is used to instruct the large language model to generate generated data associated with the prompt template according to the first contextual prompt; and filtering data that meets preset conditions from the generated data and storing it in the initial data set to obtain a target data set.

[0006] According to an embodiment of the present invention, a prompt template includes an event triple, and the event triple includes an event category, an event trigger word and an event argument; the large language model data enhancement method for event extraction also includes: based on the event category in the sampled data, determining the argument role corresponding to the event category from N event patterns; determining the event argument according to the argument role corresponding to the event category; constructing a trigger word dictionary for the event category, wherein the trigger word dictionary includes M trigger words, each trigger word is associated with the event category; M is an integer greater than or equal to 2; and determining the event trigger word from the M trigger words.

[0007] According to an embodiment of the present invention, an event triple includes K, where K is an integer greater than or equal to 2; determining an event trigger word from M trigger words includes: determining the probability of each trigger word in the trigger word dictionary being selected in the corresponding event category; and determining K event trigger words from the M trigger words based on the probability of the trigger word being selected in the event category.

[0008] According to an embodiment of the present invention, the preset conditions include a first preset condition and a second preset condition, the generated data include P text data, P is an integer greater than or equal to 2, each of the text data contains an event triple associated with an event category, and the event triple includes an event category, an event trigger word and an event argument; data that meets the preset conditions is filtered out from the generated data and stored in an initial data set to obtain a target data set, including: based on preset evaluation indicators and evaluation rules corresponding to the preset evaluation indicators, data that meets the first preset condition is filtered out from the P text data to obtain intermediate text data; based on the event triples associated with the event category contained in the intermediate text data, data that meets the second preset condition is filtered out from the intermediate text data to obtain the target text data; and the target text data is stored in the initial data set to obtain the target data set.

[0009] According to an embodiment of the present invention, based on preset evaluation indicators and evaluation rules corresponding to the preset evaluation indicators, data that meets a first preset condition is screened from P text data to obtain intermediate text data, including: inputting each text data in the P text data, the preset evaluation indicators and the evaluation rules corresponding to the preset evaluation indicators into a large language model, and outputting evaluation results corresponding to each text data, wherein the evaluation results are used to characterize the data quality of the text data; and screening the intermediate text data whose evaluation results meet the first preset condition from the P text data.

[0010] According to an embodiment of the present invention, based on event triples associated with event categories contained in intermediate text data, data that meets a second preset condition is screened from the intermediate text data to obtain target text data, including: inputting the intermediate text data and an inference chain prompt template into a large language model, and outputting an inference chain corresponding to the intermediate text data, wherein the inference chain prompt template is used to instruct the large language model to generate an inference chain with a structure similar to the inference chain prompt template based on the intermediate text data; inputting the inference chain as a second context prompt and the intermediate text data into the large language model, and outputting a generated triplet corresponding to the intermediate text data, wherein the second context prompt is used to instruct the large language model to perform a process-based event extraction with an inference process on the intermediate text data; and based on the generated triplet and the event triplet associated with the event category contained in the intermediate text data, data that meets the second preset condition is screened from the intermediate text data to obtain the target text data.

[0011] According to an embodiment of the present invention, generating triples includes generating categories, generating trigger words and generating arguments; inputting the inference chain as the second context prompt and the intermediate text data into the large language model, and outputting the generated triples corresponding to the intermediate text data, including: inputting the inference chain as the second context prompt and the intermediate text data into the large language model, so as to output the generated triples corresponding to the intermediate text data after performing the following operations: based on the second context prompt, identifying event trigger words in the intermediate text data to obtain generated trigger words; classifying event categories for the generated trigger words to obtain generated categories; and identifying event arguments from the intermediate text data based on the generated trigger words and the generated categories to obtain generated arguments.

[0012] According to an embodiment of the present invention, based on the event triples associated with event categories contained in the generated triples and the intermediate text data, data that meets the second preset condition is screened from the intermediate text data to obtain target text data, including: comparing the generated categories, generated trigger words and generated arguments in the generated triples with the event categories, event trigger words and event arguments in the event triples one by one to obtain comparison results; and when it is determined that the comparison results are consistent, the intermediate text data corresponding to the generated triples is determined as data that meets the second preset condition.

[0013] According to an embodiment of the present invention, based on the frequency of occurrence of each event category in the N event patterns in an initial data set, the sampling probability of each event category in the N event patterns is determined, including: determining the inverse of the frequency of occurrence of each event category in the N event patterns in the initial data set; accumulating and summing the inverses of the frequency of occurrence of each event category in the N event patterns in the initial data set to obtain the inverse sum; and determining the product of the inverse of the frequency of occurrence of each event category in the N event patterns in the initial data set and the inverse of the inverse sum as the sampling probability of each event category in the N event patterns.

[0014] Another aspect of the present invention provides a large language model data enhancement device for event extraction, including: an acquisition module, used to acquire N event patterns from an initial data set, wherein the event pattern includes an event category and an argument role corresponding to the event category, and N is an integer greater than or equal to 1; a first determination module, used to determine the sampling probability of each event category in the N event patterns based on the frequency of occurrence of each event category in the N event patterns in the initial data set, wherein the frequency is inversely proportional to the sampling probability; a sampling module, used to sample from the initial data set based on the sampling probability of each event category in the N event patterns to obtain sampled data; a data generation module, used to input the sampled data as a first contextual prompt and a prompt template into a large language model, and output generated data, wherein the prompt template is used to instruct the large language model to generate generated data associated with the prompt template according to the first contextual prompt; and a second determination module, used to filter data that meets preset conditions from the generated data, and store it in the initial data set to obtain a target data set.

[0015] According to an embodiment of the present invention, for each event category in the N event patterns in the initial data set, sampling is performed from the initial data set based on the fact that the frequency of occurrence of each event category in the initial data set is inversely proportional to the sampling probability, and sampling data containing multiple event categories can be obtained. The sampled data containing multiple event categories can be used as contextual prompts to generate diversified generated data, and then high-quality generated data can be used as enhanced data through screening to constitute an enhanced target data set together with the initial data. Since there are many types of sample data in the target data set and the same type has diverse features, overfitting of the event extraction model training can be prevented, and the generalization ability of the event extraction model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0017] Figure 1 A flowchart of a large language model data enhancement method for event extraction according to an embodiment of the present invention is shown;

[0018] Figure 2 A diagram showing a data enhancement structure of a large language model for event extraction according to another embodiment of the present invention is shown;

[0019] Figure 3A A schematic diagram of determining generated data according to an embodiment of the present invention is shown;

[0020] Figure 3B A schematic diagram of a zero-sample thinking chain according to an embodiment of the present invention is shown;

[0021] Figure 3C A schematic diagram of evaluation for generated data according to an embodiment of the present invention is shown;

[0022] Figure 4 A block diagram of a large language model data enhancement device for event extraction according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] In related technologies, data augmentation methods can be used to increase the amount of data by adding slightly modified copies of existing data or synthetic data newly created from existing data. These methods alleviate the failure of deep learning technology due to data scarcity and are widely used in the field of computer vision, such as flipping and rotation, and then introduced into Natural Language Processing (NLP).

[0028] In the field of natural language processing, data augmentation is a key technology that aims to improve the performance and generalization ability of the model by expanding the training data set. For example, the Exploratory Data Analysis (EDA) method is a preliminary method for analyzing data, which aims to understand the basic characteristics and structure of the data, discover patterns and heterogeneity in the data, etc. However, synonym replacement often has similar word vectors, which sometimes cannot really play the role of expansion; random insertion and random exchange of words will change the grammatical structure of the original sentence, which will have an impact on tasks with high requirements for grammatical structure and semantic structure; deleting random words and deleting core words may cause the data to deviate from the original label in the feature space.

[0029] In the process of implementing the concept of the present invention, the inventors found that the samples generated by the current data augmentation method based on large language models are not diverse enough, resulting in the event extraction model being overly dependent on certain specific patterns. In addition, the quality of the generated samples is not as good as the real samples. For example, the generated samples contain some unnatural or erroneous language structures.

[0030] Based on this, an embodiment of the present invention provides a large language model data enhancement method and device for event extraction. The method includes: obtaining N event patterns from an initial data set, wherein the event pattern includes an event category and an argument role corresponding to the event category, and N is an integer greater than or equal to 1; based on the frequency of occurrence of each event category in the N event patterns in the initial data set, determining the sampling probability of each event category in the N event patterns, wherein the frequency is inversely proportional to the sampling probability; based on the sampling probability of each event category in the N event patterns, sampling from the initial data set to obtain sampled data; inputting the sampled data as the first contextual prompt and the prompt template into the large language model, and outputting generated data, wherein the prompt template is used to instruct the large language model to generate generated data associated with the prompt template according to the first contextual prompt; and filtering data that meets the preset conditions from the generated data, and storing it in the initial data set to obtain the target data set.

[0031] The following will be passed Figure 1~Figure 3C The large language model data enhancement method for event extraction according to an embodiment of the present invention is described in detail.

[0032] Figure 1 A flowchart of a large language model data enhancement method for event extraction according to an embodiment of the present invention is shown.

[0033] like Figure 1 As shown, the large language model data enhancement method 100 for event extraction includes operations S110 to S150.

[0034] In operation S110 , N event patterns are acquired from an initial data set.

[0035] In operation S120 , based on the frequency of occurrence of each event category in the N event patterns in the initial data set, a sampling probability of each event category in the N event patterns is determined.

[0036] In operation S130 , sampling is performed from the initial data set based on the sampling probabilities of the event categories in the N event patterns to obtain sampled data.

[0037] In operation S140 , the sample data is input into the large language model as the first contextual prompt and the prompt template, and the generated data is output.

[0038] In operation S150, data satisfying a preset condition is screened from the generated data and stored in an initial data set to obtain a target data set.

[0039] According to an embodiment of the present invention, an event pattern may include an event category and an argument role corresponding to the event category, where N is an integer greater than or equal to 1. An event category may refer to a category of an event that occurs, and is used to describe the nature of an event. An argument role may refer to a relationship between an event argument and an event that it participates in. An event argument may refer to a main attribute of an event, including entities, non-entity participants, time, and other elements.

[0040] According to an embodiment of the present invention, the initial data set may include multiple text data. Each of the multiple text data has a corresponding event category and an argument role corresponding to the event category. For example, N event patterns obtained from the initial data set may be: "Event Category: Finance / Transaction-Price Reduction" and "Argument Role: Time, Price Reduction Party, Price Reduction Item, Price Reduction Range".

[0041] According to an embodiment of the present invention, the frequency may be inversely proportional to the sampling probability. For event categories that appear less frequently in the initial data set, the sampling probability is higher than that of event categories that appear more frequently in the initial data set.

[0042] For example, the sampling weight values ​​of the event categories in the N event patterns can be determined according to the frequencies of occurrence of the event categories in the N event patterns in the initial data set. The sampling probability of the event categories in the N event patterns is determined as the product of the sampling weight values ​​of the event categories in the N event patterns and the frequencies of occurrence of the event categories in the initial data set.

[0043] According to an embodiment of the present invention, the sampled data may include text data of each event category in the N event patterns.

[0044] According to an embodiment of the present invention, the initial data set may have a standard output format, such as text data and an event list. The event list may include event categories, event trigger words, argument role sets, and event arguments corresponding to each argument role in the argument role set. The sampled data may be used as a first contextual prompt in the standard output format.

[0045] According to an embodiment of the present invention, the prompt template may be used to instruct the large language model to generate generated data associated with the prompt template according to the first context prompt.

[0046] According to an embodiment of the present invention, the text data of each event category in the N event patterns can be used as the first contextual prompt. For each event category in the N event patterns: the first contextual prompt and the prompt template corresponding to the event category can be input into the large language model, and the generated data of the event category can be output. The generated data can refer to new text data. The new text data is data generated based on the text data and associated with the prompt template. The generated data can include the generated data of each event category in the N event patterns.

[0047] According to an embodiment of the present invention, for each event category in the N event patterns in the initial data set, sampling is performed from the initial data set based on the frequency of occurrence of each event category in the initial data set being inversely proportional to the sampling probability, and sampling data containing multiple event categories can be obtained. The sampling data containing multiple event categories can be used as contextual prompts to generate diversified generated data, and then the high-quality generated data can be used as enhanced data through screening to form an enhanced target data set together with the initial data. The sample data in the target data set is of high quality and variety, and the features of the same category are diverse, which can prevent overfitting of the event extraction model training and improve the generalization ability of the event extraction model.

[0048] In the process of implementing the embodiments of the present invention, it is found that in the data set of the information extraction task of natural language processing, data distribution is often uneven, and data of certain event categories are very scarce and therefore rarely sampled.

[0049] For example, in a Chinese event extraction task dataset, there are only dozens of data on the event category "life-cheating", while there are thousands of data on the event category "competitive behavior-winning or losing". Every time this dataset is sampled, it will be difficult to sample rare event category data due to the uneven distribution of data in the dataset.

[0050] Based on this, according to an embodiment of the present invention, for the above Figure 1 Operation S120, based on the frequency of occurrence of each event category in the N event patterns in the initial data set, determines the sampling probability of each event category in the N event patterns, which may include the operations of: determining the reciprocal of the frequency of occurrence of each event category in the N event patterns in the initial data set; accumulating and summing the reciprocals of the frequency of occurrence of each event category in the N event patterns in the initial data set to obtain the reciprocal sum; and determining the sampling probability of each event category in the N event patterns by multiplying the reciprocal of the frequency of occurrence of each event category in the N event patterns in the initial data set and the reciprocal of the reciprocal sum.

[0051] According to an embodiment of the present invention, for each acquired event category, the distribution frequency in the initial data set is calculated, that is, the frequency of occurrence in the initial data set.

[0052] For example, num represents the number of occurrences of a certain event category data in the initial data set, freq represents the frequency of occurrence of the event category data in the initial data set, and n is the number of event categories in the initial data set. Then, the calculation method of the distribution frequency of each event category in the initial data set can be as shown in the following formula (1):

[0053] (1)

[0054] The sampling probability of an event category is inversely proportional to its distribution frequency, so that event categories with fewer current occurrences can be sampled with a higher probability.

[0055] For example, prob represents the sampling probability of a certain event category data in the current situation. In the current situation, the sampling probability of the event category is as follows (2):

[0056] (2)

[0057] in: is the sum of the inverses of the distribution frequencies of all event categories. The sampling probability calculated by the above formula (2) is inversely proportional to the distribution frequency of each event category in the initial data set.

[0058] According to an embodiment of the present invention, the sampling probability of an event category is inversely proportional to its distribution frequency, so that event categories with fewer current occurrences can be sampled with a higher probability, which can make the event categories in the target data set after data enhancement sufficiently uniform.

[0059] According to an embodiment of the present invention, a prompt template may include an event triple, and the event triple may include an event category, an event trigger word, and an event argument.

[0060] In addition to the large language model data enhancement methods for event extraction, Figure 1 In addition to the operations S110 to S150 shown, the operations may also include: determining the argument role corresponding to the event category from N event patterns based on the event category in the sampled data; determining the event argument according to the argument role corresponding to the event category; constructing a trigger word dictionary for the event category, wherein the trigger word dictionary includes M trigger words, each trigger word is associated with the event category; M is an integer greater than or equal to 2; and determining an event trigger word from the M trigger words based on the probability of the trigger word being selected in the event category.

[0061] According to an embodiment of the present invention, each event category may correspond to multiple argument roles, and based on the argument roles in the multiple argument roles, event arguments matching each argument role may be determined from the sampled data.

[0062] For example, when the event category of the sampled data is "Organizational Relationship-Layoffs", the argument roles corresponding to the event category "Organizational Relationship-Layoffs" can be "Time, Layoff Party, Number of Layoffs". The event argument that matches the argument role "Time" can be a specific time value in the sampled data. The event argument that matches the argument role "Layoff Party" can be a specific layoff party field in the sampled data. The event argument that matches the argument role "Number of Layoffs" can be a specific number of layoffs value in the sampled data.

[0063] According to an embodiment of the present invention, a trigger word dictionary for each event category can be constructed, and trigger words can be selected from the trigger word dictionary according to the event category, so as to construct an event triple. A trigger word can refer to an event trigger word, which can be a noun or verb expressing the occurrence of an event, or a feature word that determines the core of the event category. The trigger word dictionary can contain common trigger words for the event category, as well as their synonyms or near-synonyms. Existing vocabulary resources, synonym dictionaries, or pre-trained word vector models can be used to capture more changes and diversity of trigger words, and triggers with different changes and diversity can be used to construct a trigger word dictionary.

[0064] For example, when the event category is "organizational relations-layoffs", a dictionary of related trigger words can be established, including "layoffs, staff reductions, layoffs, reductions, layoffs, dismissals, and dismissals".

[0065] According to an embodiment of the present invention, for data of a certain event category, a trigger word can be randomly selected or selected according to a certain probability distribution based on the trigger word dictionary of the event category, thereby ensuring that the generated data is associated with the event category and can cover different expressions of the event category.

[0066] According to an embodiment of the present invention, by including a prompt template of an event triple, different event triples can be generated for the same event category, and different prompt templates can be provided for a large language model to output diversified generated data. The same event category has diversified generated data, which can prevent the model from overfitting and improve the generalization ability when used as training samples for the event extraction model.

[0067] According to an embodiment of the present invention, an event triplet may include K, where K is an integer greater than or equal to 2; determining an event trigger word from M trigger words may include the following operations: determining the probability of each trigger word in the trigger word dictionary being selected in the corresponding event category; and determining K event trigger words from the M trigger words based on the probability of the trigger word being selected in the event category.

[0068] According to an embodiment of the present invention, the probability of each trigger word in the trigger word dictionary being selected in the corresponding event category may be determined based on the degree of association between the trigger words in the trigger word dictionary and the event category.

[0069] According to an embodiment of the present invention, the trigger words in the trigger word dictionary can be sorted in descending order according to the probability of each trigger word in the trigger word dictionary being selected in the corresponding event category, and then K event trigger words are determined from the M trigger words in sequence according to the sorting. There is only one event trigger word in each event triple, and K event triples can be generated.

[0070] According to an embodiment of the present invention, event trigger words are determined based on the probability of trigger words being selected, ensuring that in multiple generations, the selection of trigger words conforms to their distribution probability, making the generated data more diverse.

[0071] According to another embodiment of the present invention, determining an event trigger word from M trigger words may include the following operations: determining a weight value of each trigger word in the M trigger words based on the degree of association between the trigger word and the event category; sorting the weight values ​​to obtain sorted weight values; and accumulating and summing the sorted weight values ​​in order according to the sorting; when it is determined that the cumulative sum exceeds a preset value, determining the trigger word corresponding to the last weight value in the cumulative sum as the event trigger word.

[0072] For example, the weight value of each trigger word can represent its probability of being selected under the event category. The sum of the probabilities is equal to 1. When generating data, a random number r, i.e., a preset value, ranging from 0 to 1 can be generated. According to the cumulative weight values ​​of the trigger words, the first trigger word that makes the cumulative probability exceed r is selected. For example, in the trigger word dictionary of the event category "Organizational Relations-Layoffs", "Layoffs" (weight: 0.3), "Staff Reductions" (weight: 0.2), "Abolitions" (weight: 0.2), "Layoffs" (weight: 0.1), "Retirement" (weight: 0.1), and "Dismissal" (weight: 0.1) are set. When the generated random number r is 0.55, since the sum of the weight values ​​corresponding to layoffs, staff reductions, and abolitions is greater than 0.55, the trigger word "Abolitions" can be determined as the event trigger word.

[0073] According to an embodiment of the present invention, the prompt template may include generation requirements in addition to the event triplet. Different prompt templates may be determined according to different event triplet.

[0074] For example, the determined event triple is <organizational relationship-layoffs, abolition, time, layoff party, number of layoffs>, and the prompt template for the large language model can be: "You are a data generator for an event extraction task. Now we need to collect more data for an initial data set for Chinese event extraction. Your task is to generate 3 pieces of data through the following context prompts. The event category in the generated data is required to be "organizational relationship-layoffs", the event trigger word is "abolition", and the event arguments are "time, layoff party, number of layoffs". Please output strictly according to the first context prompt and generation requirements, where text represents text data, event_list represents event list, event_type represents event category, trigger represents event trigger word, arguments is argument set, role represents argument role, and argument represents event argument.

[0075] The first contextual hint is: {"text": "XX laid off 4,000 people","event_list": [{"event_type": "Organizational relations-Layoffs", "trigger": "Layoffs", "arguments": [{ "role": "Layoff party","argument": "XX"}, {"role": "Number of layoffs", "argument": "4,000 people"}]]}".

[0076] The generated data for the large language model can be, for example: {"text": "XX announced internal optimization and plans to lay off some employees, with the number of layoffs exceeding 12,000", "event_list": [{"event_type": "Organizational Relationship-Layoffs", "trigger": "Layoffs", "arguments": [{"role": "Time", "argument": ""}{"role": "Layoff party", "argument": "XX"}, {"role": "Number of layoffs", "argument": "12,000"}]]}.

[0077] The event triplet of the generated data is <organizational relationship-layoffs, abolition, layoff party: "XX", number of layoffs: "more than 12,000 people">, indicating that the generated data event category is "organizational relationship-layoffs", the event trigger word is "layoffs", and the event arguments and roles are "layoff party: XX" and "number of layoffs: more than 12,000 people".

[0078] According to the embodiment of the present invention, using a variety of prompt templates to generate data can effectively improve the diversity of generated data.

[0079] According to an embodiment of the present invention, the generated data may include P text data, where P is an integer greater than or equal to 2. Screening data that meets preset conditions from the generated data and storing them in the initial data set to obtain a target data set may include the following operations: inputting each text data in the P text data, preset evaluation indicators, and evaluation rules corresponding to the preset evaluation indicators into a large language model, and outputting evaluation results corresponding to each text data, wherein the evaluation results are used to characterize the data quality of the text data; screening target text data whose evaluation results meet the preset conditions from the P text data; and storing the target text data in the initial data set to obtain the target data set.

[0080] According to an embodiment of the present invention, the preset evaluation index may include at least one of the following: a negative index, an information integrity index, a factual consistency index, and an evaluation rule corresponding to the negative index. The negative index may be used to evaluate whether the information in the event triple contained in the text data is non-existent or erroneous, the information integrity index may be used to evaluate whether the text data contains all the information of the event triple, and the factual consistency index may be used to evaluate whether the text data conforms to objective facts and whether it is logical.

[0081] According to an embodiment of the present invention, the preset condition may include a condition that can meet the data quality requirement. For example, it may be a preset quality threshold. When the evaluation result indicates that the data quality of the text data meets the preset quality threshold, the corresponding text data may be used as the target text data.

[0082] According to an embodiment of the present invention, by performing data quality evaluation on the generated data, such as evaluation of negative indicators, information integrity indicators, and factual consistency indicators, the quality of the generated data can be improved.

[0083] According to an embodiment of the present invention, generated data may include P text data, where P is an integer greater than or equal to 2; data that meets preset conditions is screened from the generated data, and stored in an initial data set to obtain a target data set, which may include the following operations: inputting each text data in the P text data and an inference chain prompt template into a large language model, and outputting the inference chain corresponding to each text data; inputting the inference chain corresponding to each text data as a second context prompt and each text data in the P text data into the large language model, and outputting the generated triples corresponding to each text data; screening the target text data whose generated triples meet the preset conditions from the P text data; and storing the target text data in the initial data set to obtain the target data set.

[0084] According to an embodiment of the present invention, the inference chain prompt template may be used to instruct the large language model to generate an inference chain having a structure similar to the inference chain prompt template according to each text data in the P text data.

[0085] According to an embodiment of the present invention, the second contextual prompt may be used to instruct the large language model to perform a streamlined event extraction with a reasoning process on each of the P text data.

[0086] According to an embodiment of the present invention, generating triples includes generating categories, generating trigger words, and generating arguments. The preset conditions include conditions that can satisfy triple screening. For example, the preset conditions can be whether the generated triples are consistent with the event triples associated with the event categories contained in the sampled data. When it is determined that the generated triples are consistent with the event triples associated with the event categories contained in the sampled data, the corresponding text data is determined as the target text data.

[0087] According to an embodiment of the present invention, the generated triples corresponding to the generated data can be determined through the reasoning logic of the large model, and high-quality data can be screened through the evaluation of the generated triples.

[0088] However, the evaluation of the generated data itself can only solve the problems of irrelevant or false information in the text data generated by the large language model, incomplete coverage of the given triple information in the prompt template, detachment from objective facts, and lack of logic. The reasoning of the large model after the data is generated can only self-evaluate the data quality. The target text data determined by the two perspectives still have data quality issues.

[0089] Based on this, in another embodiment of the present invention, the preset conditions may include a first preset condition and a second preset condition, and the generated data may include P text data, where P is an integer greater than or equal to 2, and each of the text data contains an event triple associated with an event category, and the event triple may include an event category, an event trigger word, and an event argument.

[0090] Regarding the above Figure 1 Operation S150, filtering data satisfying a preset condition from the generated data, and storing it in an initial data set to obtain a target data set, may include the following operations: based on a preset evaluation indicator and an evaluation rule corresponding to the preset evaluation indicator, filtering data satisfying a first preset condition from P text data to obtain intermediate text data; based on an event triple associated with an event category contained in the intermediate text data, filtering data satisfying a second preset condition from the intermediate text data to obtain target text data; and storing the target text data in the initial data set to obtain a target data set.

[0091] According to an embodiment of the present invention, the preset evaluation index can be determined in advance according to the characteristics of the data itself, such as whether the information is complete, whether the logic is reasonable, etc. The evaluation rules corresponding to the preset evaluation index can be used to evaluate the quality of the generated data.

[0092] According to an embodiment of the present invention, the first preset condition may include a condition that satisfies a preset evaluation index and an evaluation rule corresponding to the preset evaluation index. For example, the first preset condition may be that the text data does not contain information that does not exist or is erroneous in the event triple, the text data contains all information of the event triple, and the text data conforms to objective facts and has logic, etc.

[0093] According to an embodiment of the present invention, the second preset condition may include a condition satisfying an event triple associated with an event category. For example, the second preset condition may be that the generated triple inferred from the text data is consistent with the event triple associated with the event category contained in the text data.

[0094] Figure 2 A diagram showing a data enhancement structure of a large language model for event extraction according to another embodiment of the present invention.

[0095] like Figure 2As shown, the initial data set can be analyzed for data distribution and event pattern in the analyzer, and then sampling can be performed according to the data distribution of data of different event categories to obtain sampled data. Using the generator, the event triples and generation requirements determined by the trigger word dictionary and the sampled data are input into the large language model to obtain generated data. The discriminator is used to judge the quality of the generated data from negative indicators, information consistency indicators and fact consistency indicators to obtain the generated data that has passed the judgment. Then, the evaluator is further used to generate the reasoning chain with the help of the reasoning logic thinking of the large language model, and then the process event extraction is performed, such as trigger word recognition to obtain the generated trigger word, event classification of the generated trigger word to obtain the generated category, and argument recognition and classification based on the generated trigger word and the generated category to obtain the generated argument. The generated trigger word, the generated category and the generated argument form a generated triple, and the generated data is evaluated based on the generated triple and the event triple.

[0096] According to the embodiments of the present invention, through two screenings, it is not only possible to directly evaluate from the generated data side, but also to solve the problems of irrelevant or false information in the text data generated by the large language model, incomplete coverage of the given triple information in the prompt template, detachment from objective facts, and insufficient logic. It is also possible to further verify the quality of the generated data from the reasoning verification side of the generated data through the reasoning logic ability of the large language model itself, so that the enhanced target data set has high-quality data.

[0097] According to an embodiment of the present invention, based on preset evaluation indicators and evaluation rules corresponding to the preset evaluation indicators, data that meets a first preset condition is screened from P text data to obtain intermediate text data, which may include the following operations: inputting each text data in the P text data, the preset evaluation indicators and the evaluation rules corresponding to the preset evaluation indicators into a large language model, and outputting evaluation results corresponding to each text data, wherein the evaluation results are used to characterize the data quality of the text data; and screening the intermediate text data whose evaluation results meet the first preset condition from the P text data.

[0098] According to an embodiment of the present invention, the natural language understanding capability of a large language model can be used to understand the generated data generated by itself, and score the data according to preset evaluation indicators using evaluation rules corresponding to the preset evaluation indicators to obtain evaluation results.

[0099] According to an embodiment of the present invention, the preset evaluation index may include at least one of the following: a negative index, an information integrity index, a factual consistency index, and an evaluation rule corresponding to the negative index. The negative index is an index for evaluating whether the information in the event triple contained in the text data does not exist or is wrong, the information integrity index is an index for evaluating whether the text data contains all the information of the event triple, and the factual consistency index is an index for evaluating whether the text data conforms to objective facts and whether it is logical.

[0100] Figure 3A A schematic diagram of discrimination of generated data according to an embodiment of the present invention is shown.

[0101] According to an embodiment of the present invention, the score of each preset evaluation indicator can be set to 0.0~5.0. Based on the machine learning paradigm (few-shot learning), a large language model is used to score the generated data. The full score is 15.0. The score of a single evaluation indicator is called the indicator level score, and the average score of each evaluation indicator is called the sample level score.

[0102] like Figure 3A As shown, discriminative prompt templates can be designed, such as the role, content, examples, and given text of the large model.

[0103] The generated text data should not contain information that is not present or is wrong in the triples. The large language model should reject text data that contains information that is not present or is wrong in the triples and generate a score.

[0104] For example, continuing the example of generating data above, such as Figure 3A As shown, the large language model should not reject this data, and the negative indicator score is 5 points, which is the full score. Reason: There is no wrong or non-existent information in the sentence, so there is no condition to trigger the negative indicator.

[0105] The generated text data should contain all the information in the given triple. The large language model checks whether the generated text covers all the information in the given triple, ensures that the semantic information in the generated text is complete, without missing or wrong parts, and examines the correlation between different parts in the generated text to ensure that the logical relationship between the information is reasonable and there is no isolated or irrelevant part of the information.

[0106] For example, continuing the example of generating data above, such as Figure 3A As shown, the score of the information completeness index is 3.5 points. Reason: The sentence covers the event category, event trigger words, and related arguments. The information completeness is relatively complete, but the score is not full because the given sentence does not mention the time, and the reasons or methods for layoffs can be described in more detail in "Plan to lay off some employees."

[0107] The generated text data should be consistent with objective facts and be logical. The large language model judges whether the text information of the generated data is consistent with objective facts and ensures that the logical relationship between the text information is consistent, avoiding the situation where the generated text is self-contradictory or inconsistent with basic logic. If the generated text contains information that contradicts the facts, the large language model should lower its quality score.

[0108] For example, continuing the example of generating data above, such as Figure 3A As shown, the score of the factual consistency index is 4. Reason: The information in the sentence does not involve any situation that contradicts the objective facts, but the lack of further context may lead to limitations in the assessment.

[0109] The large language model scored an overall score of 12.5 on the generated data. Overall, this sentence scored relatively high, but there is still room for improvement in terms of information completeness and factual consistency, especially in providing more detailed context and explaining the reasons for layoffs.

[0110] Reliable samples can be selected by the score threshold filtering method. In the embodiment of the present invention, the text data can be filtered and the score threshold can be set: the indicator level score is 3.0 and the sample level score is 11.0. In the present invention, a higher score indicates that the text data has a higher quality. By filtering and screening the two thresholds, high-quality generated text data can be selected.

[0111] For example, level-score represents the level score of a certain indicator of the current text data, and sample-score represents the sample level score of the current text data. Using this score threshold filtering method, the following equations (3) to (4) must be satisfied:

[0112] (3)

[0113] (4)

[0114] For example, continuing with the example of generated data mentioned above, the large language model scores the text data as follows: the negative indicator is 5 points, the information completeness index is 3.5 points, and the factual consistency index is 4 points, all greater than 3; the total score is 12.5 points, which is greater than 11. Therefore, the data meets the first preset condition and is high-quality generated text data.

[0115] According to an embodiment of the present invention, based on event triples associated with event categories contained in intermediate text data, data that meets a second preset condition is screened from the intermediate text data to obtain target text data, which may include the following operations: inputting the intermediate text data and an inference chain prompt template into a large language model, and outputting an inference chain corresponding to the intermediate text data, wherein the inference chain prompt template is used to instruct the large language model to generate an inference chain similar in structure to the inference chain prompt template based on the intermediate text data; inputting the inference chain as a second context prompt and the intermediate text data into the large language model, and outputting a generated triplet corresponding to the intermediate text data, wherein the second context prompt is used to instruct the large language model to perform a process-based event extraction with an inference process on the intermediate text data; and based on the generated triplet and the event triplet associated with the event category contained in the intermediate text data, data that meets the second preset condition is screened from the intermediate text data to obtain the target text data.

[0116] According to an embodiment of the present invention, a large language model may be used to automatically generate an inference chain, and the inference chain may be used as a second context prompt.

[0117] Figure 3B A schematic diagram of a zero-sample thinking chain according to an embodiment of the present invention is shown.

[0118] A reasoning chain prompt template can be designed to define the triggering method of the zero-shot chain of thought (Zero-shot-CoT). Chain of Thought (CoT) can effectively stimulate the reasoning ability of the large language model. Based on Zero-shot-CoT, a simple prompt is used to let the large language model think about the extraction results step by step. In the present invention, the triggering method of Zero-shot-CoT is defined as prompting the large language model to "think step by step" or "reason step by step".

[0119] For example Figure 3B As shown, the initial data set can be divided into clusters based on event categories, such as event category 1, event category 2... event category k, etc., and a piece of data is randomly selected from each cluster of the initial data set as an event category 1 sample, an event category 2 sample... event category k sample, etc., and input into the large language model respectively, and zero-sample thinking chain reasoning or thinking is used to obtain contextual reasoning for each event category sample, and a second context prompt is constructed based on these contextual reasoning.

[0120] For example, in an embodiment of the present invention, the sample data used to construct the second context prompt for the event category of "organizational relationship-layoffs" can be: "XX time May 24, according to the local media "XXXXX": The company invested by XX and XX team leader XX has laid off a large number of employees and is at risk of bankruptcy."

[0121] The inference chain prompt template can be: "On May 24, XX time, according to the local media "XXXXX", the company invested by XX and XX team captain XX has laid off a large number of employees and is at risk of bankruptcy." Please infer step by step why the event category of this sentence is "Organizational Relationship-Layoffs" and the event trigger word is "Layoffs".

[0122] Zero-shot-CoT can be used to generate the reasoning chain for each clustered sampled data to construct the corresponding second contextual cue.

[0123] The sampled data and the inference chain prompt template are input into the large language model, and the output answer, i.e., the inference chain, can be: Based on the given sentence, the following points are observed: 1. The sentence contains "the company invested by XX and XX team leader XX", which indicates that the sentence involves the situation of a company. 2. The sentence mentions "layoffs", which can be inferred that the company is in the process of laying off employees. In summary, based on the information in the sentence, the event category is judged to be "organizational relationship-layoffs" and the event trigger word is "layoffs".

[0124] Figure 3C A schematic diagram of evaluation of generated data according to an embodiment of the present invention is shown.

[0125] According to an embodiment of the present invention, generating triples may include generating categories, generating trigger words, and generating arguments; inputting the reasoning chain as the second context prompt and the intermediate text data into the large language model, and outputting the generated triples corresponding to the intermediate text data, which may include the following operations: inputting the reasoning chain as the second context prompt and the intermediate text data into the large language model, so as to output the generated triples corresponding to the intermediate text data after performing the following operations: based on the second context prompt, identifying event trigger words in the intermediate text data to obtain generated trigger words; classifying the generated trigger words into event categories to obtain generated categories; and identifying event arguments from the intermediate text data based on the generated trigger words and the generated categories to obtain generated arguments.

[0126] According to an embodiment of the present invention, the above-mentioned reasoning chain can be used as a second contextual prompt, and a large language model can be used to understand the nature of events in text data, thereby detecting and identifying event trigger words, and then classifying event categories based on the event trigger words.

[0127] For example, for text data: "It is understood that this Horse Expo will open at 20:00 on the evening of August 18 at the Cultural, Sports and Tourism Center."

[0128] The second context prompt generated by the large language model based on the inference chain prompt template can be: "Based on the given sentence, the following points are observed: 1. The sentence mentions "Horse Expo", which determines that the sentence involves an event or meeting. 2. The sentence mentions "opening", and it is inferred that the event will open at 20:00 on the evening of August 18. In summary, based on the information in the sentence, the event trigger word is judged to be "opening" and the event category is "organizational behavior-opening.".

[0129] According to an embodiment of the present invention, an event category sample can be used as a question, and contextual reasoning for the event category sample can be used as an answer to construct a reasoning chain sample, such as Figure 3C As shown, reasoning chain sample 1, reasoning chain sample 2, and so on. The reasoning chain sample is used as contextual reasoning, that is, the second context prompt. The large language model combines contextual reasoning to identify event trigger words from the generated data to obtain generated trigger words, and classifies events based on the generated trigger words to obtain generated categories. Based on the generated trigger words and generated categories, the arguments are identified and classified to obtain generated arguments and role classification for the generated arguments.

[0130] For example, continuing with the second contextual prompt example above, the result of the large language model for identifying generated arguments and role classification can be: "Based on the information provided, gradually infer the generated arguments: 1. Generate trigger words: "Opening" 2. Generate categories: "Organizational behavior-opening" Analyze each argument: Time: "8 p.m. on August 18th", according to the information provided, the specific time of the opening is 8 p.m. on August 18th. This indicates the official start time of the Horse Expo, which may include a series of celebrations and ceremonies. Location: "Cultural Sports and Sports Center", the opening ceremony will be held at the Cultural Sports and Sports Center. This location may be a large venue, suitable for accommodating a large number of participants and spectators to demonstrate the scale and importance of the Horse Expo. Event name: "Horse Expo", Horse Expo is the name of the event for this opening ceremony. Based on the name, it is speculated that this may be a horse-related expo, which may include equestrian competitions, cultural displays, commercial exhibitions and other activities. This can also explain why it needs to be held in a large venue such as the Cultural Sports and Sports Center."

[0131] According to an embodiment of the present invention, Figure 3C As shown, the inference chain sample can be used as the inference chain, and multiple inference chains can be sampled for the large language model to extract the generated triples from the generated data as context reasoning, and multiple extraction results can be obtained, such as extraction result 1, extraction result 2, and extraction result 3, etc. The extraction results are voted by majority, and the most self-consistent result is selected as the final result. The inference chains can be summarized, and the inference chain corresponding to the final result can be taken as the final inference chain. Preferably, the present invention can sample 5 inference chains, for example.

[0132] For example, the event triplet is: <Organizational behavior - opening, opening, time: "8:00 p.m. on August 18", location: "Cultural Sports and Sports Center", event name: "Horse Expo">, and the text data: "It is understood that this Horse Expo will be opened at 20:00 p.m. on August 18 at the Cultural Sports and Sports Center." After processing, the generated triplet is: <Organizational behavior - opening, opening, time: "8:00 p.m. on August 18", location: "Cultural Sports and Sports Center", event name: "Horse Expo">.

[0133] According to an embodiment of the present invention, based on the event triples associated with event categories contained in the generated triples and the intermediate text data, data that meets the second preset condition is screened from the intermediate text data to obtain the target text data, which may include the following operations: comparing the generated categories, generated trigger words and generated arguments in the generated triples with the event categories, event trigger words and event arguments in the event triples one by one to obtain comparison results; and when it is determined that the comparison results are consistent, determining the intermediate text data corresponding to the generated triples as data that meets the second preset condition.

[0134] The above event triples and generated triples are continued to be used. Since the information in the event triples is consistent with the content of the generated triples, the generated data passes the self-assessment and is high-quality data.

[0135] According to an embodiment of the present invention, based on the automatic construction of the reasoning chain, the extraction of process events is realized, the generated triples are compared with the event triples, and on the basis of the evaluation of the generated data level, high-quality data is further screened out to enhance the initial data set.

[0136] Figure 4 A block diagram of a large language model data enhancement device for event extraction according to an embodiment of the present invention is shown.

[0137] like Figure 4 As shown, the large language model data enhancement device 400 for event extraction includes an acquisition module 410, a first determination module 420, a sampling module 430, a data generation module 440 and a second determination module 450.

[0138] The acquisition module 410 is used to acquire N event patterns from the initial data set, wherein the event pattern includes an event category and an argument role corresponding to the event category, and N is an integer greater than or equal to 1.

[0139] The first determination module 420 is used to determine the sampling probability of each event category in the N event patterns based on the frequency of occurrence of each event category in the N event patterns in the initial data set, wherein the frequency is inversely proportional to the sampling probability.

[0140] The sampling module 430 is used to sample from the initial data set based on the sampling probabilities of the event categories in the N event patterns to obtain sampled data.

[0141] The data generation module 440 is used to input the sampled data as the first contextual prompt and the prompt template into the large language model, and output the generated data, wherein the prompt template is used to instruct the large language model to generate the generated data associated with the prompt template according to the first contextual prompt;

[0142] The second determination module 450 is used to filter data that meets a preset condition from the generated data, and store the data into the initial data set to obtain a target data set.

[0143] According to an embodiment of the present invention, a prompt template may include an event triple, and the event triple may include an event category, an event trigger word, and an event argument; the large language model data enhancement device 400 for event extraction may also include: an argument role determination module, an event argument determination module, a dictionary construction module, and an event trigger word determination module.

[0144] The argument role determination module is used to determine the argument role corresponding to the event category from N event patterns based on the event category in the sampling data; the event argument determination module is used to determine the event argument according to the argument role corresponding to the event category; the dictionary construction module is used to construct a trigger word dictionary of the event category, wherein the trigger word dictionary includes M trigger words, each trigger word is associated with an event category; M is an integer greater than or equal to 2; the event trigger word determination module is used to determine the event trigger word from the M trigger words based on the probability of the trigger word being selected in the event category.

[0145] According to an embodiment of the present disclosure, any multiple modules of the acquisition module 410, the first determination module 420, the sampling module 430, the data generation module 440, and the second determination module 450 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 410, the first determination module 420, the sampling module 430, the data generation module 440, and the second determination module 450 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 410, the first determination module 420, the sampling module 430, the data generation module 440 and the second determination module 450 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0146] It should be noted that the large language model data enhancement device part for event extraction in the embodiment of the present invention corresponds to the large language model data enhancement method part for event extraction in the embodiment of the present invention. The description of the large language model data enhancement device part for event extraction specifically refers to the large language model data enhancement method part for event extraction, which will not be repeated here.

[0147] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or couplings fall within the scope of the present invention.

[0148] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A large language model data enhancement method for event extraction, It is characterized in that The method comprises: Obtain N event patterns from the initial data set, wherein the event pattern includes an event category and an argument role corresponding to the event category, and N is an integer greater than or equal to 1; Determine the sampling probability of each of the event categories in the N event patterns based on the frequency of occurrence of each of the event categories in the N event patterns in the initial data set, wherein the frequency is inversely proportional to the sampling probability; Based on the sampling probabilities of the event categories in the N event patterns, sampling from the initial data set to obtain sampled data; Inputting the sampled data as a first contextual prompt and a prompt template into a large language model, and outputting generated data, wherein the prompt template is used to instruct the large language model to generate the generated data associated with the prompt template according to the first contextual prompt; and Filtering data that meets preset conditions from the generated data and storing them in the initial data set to obtain a target data set; Wherein, the prompt template includes an event triple, and the event triple includes the event category, the event trigger word and the event argument; The method further comprises: Based on the event category in the sampled data, determining the argument role corresponding to the event category from the N event patterns; Determining the event argument according to the argument role corresponding to the event category; Constructing a trigger word dictionary for the event category, wherein the trigger word dictionary includes M trigger words, each of which is associated with the event category; M is an integer greater than or equal to 2; and Determine the event trigger word from the M trigger words; The event triples include K, where K is an integer greater than or equal to 2; The determining the event trigger word from the M trigger words includes: Determining the probability of each trigger word in the trigger word dictionary being selected in the corresponding event category; and Based on the probability of the trigger word being selected in the event category, K event trigger words are determined from the M trigger words.

2. The method according to claim 1, It is characterized in that The preset conditions include a first preset condition and a second preset condition, the generated data include P text data, P is an integer greater than or equal to 2, each of the text data includes an event triple associated with the event category, and the event triple includes the event category, an event trigger word, and an event argument; The step of screening the data satisfying the preset conditions from the generated data and storing the data in the initial data set to obtain the target data set includes: Based on a preset evaluation index and an evaluation rule corresponding to the preset evaluation index, data satisfying the first preset condition is selected from the P text data to obtain intermediate text data; Based on the event triples associated with the event category contained in the intermediate text data, filtering data satisfying the second preset condition from the intermediate text data to obtain target text data; and The target text data is stored in the initial data set to obtain the target data set.

3. The method according to claim 2, It is characterized in that The step of selecting data satisfying the first preset condition from the P pieces of text data based on the preset evaluation index and the evaluation rule corresponding to the preset evaluation index to obtain the intermediate text data comprises: Inputting each of the P text data, a preset evaluation index, and an evaluation rule corresponding to the preset evaluation index into the large language model, and outputting an evaluation result corresponding to each of the text data, wherein the evaluation result is used to characterize the data quality of the text data; and The intermediate text data whose evaluation results satisfy the first preset condition is screened from the P text data.

4. The method according to claim 2, It is characterized in that The step of filtering data satisfying the second preset condition from the intermediate text data based on the event triples associated with the event category contained in the intermediate text data to obtain the target text data includes: Inputting the intermediate text data and the inference chain prompt template into the large language model, and outputting the inference chain corresponding to the intermediate text data, wherein the inference chain prompt template is used to instruct the large language model to generate the inference chain similar in structure to the inference chain prompt template according to the intermediate text data; Inputting the inference chain as a second contextual prompt and the intermediate text data into the large language model, and outputting a generated triple corresponding to the intermediate text data, wherein the second contextual prompt is used to instruct the large language model to perform a process-based event extraction with an inference process on the intermediate text data; and Based on the generated triples and the event triples associated with the event category contained in the intermediate text data, data satisfying the second preset condition is screened from the intermediate text data to obtain the target text data.

5. The method according to claim 4, It is characterized in that The generation triples include a generation category, a generation trigger word and a generation argument; The step of inputting the inference chain as a second contextual prompt and the intermediate text data into the large language model, and outputting a generated triple corresponding to the intermediate text data, comprises: The inference chain is input into the large language model as the second contextual hint and the intermediate text data, so as to output the generated triples corresponding to the intermediate text data after performing the following operations: Based on the second context prompt, identifying the event trigger word on the intermediate text data to obtain the generation trigger word; Classifying the generation trigger word into the event category to obtain the generation category; and Based on the generation trigger word and the generation category, the event argument is identified from the intermediate text data to obtain the generation argument.

6. The method according to claim 5, It is characterized in that The step of filtering data satisfying the second preset condition from the intermediate text data based on the generated triples and the event triples associated with the event category contained in the intermediate text data to obtain the target text data includes: Compare the generation category, the generation trigger word and the generation argument in the generation triple with the event category, the event trigger word and the event argument in the event triple respectively one by one to obtain a comparison result; and When it is determined that the comparison results are all consistent, the intermediate text data corresponding to the generated triples is determined as data that meets the second preset condition.

7. The method according to claim 1, It is characterized in that The determining, based on the frequency of occurrence of each of the event categories in the N event patterns in the initial data set, the sampling probability of each of the event categories in the N event patterns comprises: Determine the inverse of the frequency of occurrence of each of the event categories in the N event patterns in the initial data set; Cumulatively summing the inverses of the frequencies of occurrence of each of the event categories in the N event patterns in the initial data set to obtain an inverse sum; and The sampling probability of each of the event categories in the N event patterns is determined by multiplying the product of the reciprocal of the frequency of occurrence of each of the event categories in the initial data set and the reciprocal of the sum of the reciprocals.

8. A large language model data enhancement device for event extraction, It is characterized in that The device comprises: An acquisition module, used for acquiring N event patterns from an initial data set, wherein the event pattern includes an event category and an argument role corresponding to the event category, and N is an integer greater than or equal to 1; A first determination module is used to determine the sampling probability of each of the event categories in the N event patterns based on the frequency of each of the event categories in the N event patterns appearing in the initial data set, wherein the frequency is inversely proportional to the sampling probability; A sampling module, configured to sample from the initial data set based on the sampling probabilities of the event categories in the N event patterns to obtain sampled data; a data generation module, configured to input the sampled data as a first contextual prompt and a prompt template into a large language model, and output generated data, wherein the prompt template is used to instruct the large language model to generate the generated data associated with the prompt template according to the first contextual prompt; and A second determination module is used to filter data that meets a preset condition from the generated data and store it in the initial data set to obtain a target data set; Wherein, the prompt template includes an event triple, and the event triple includes the event category, the event trigger word and the event argument; The device also includes: An argument role determination module, configured to determine, based on the event category in the sampled data, the argument role corresponding to the event category from the N event patterns; An event argument determination module, configured to determine the event argument according to the argument role corresponding to the event category; A dictionary construction module, used to construct a trigger word dictionary of the event category, wherein the trigger word dictionary includes M trigger words, each of which is associated with the event category; M is an integer greater than or equal to 2; and An event trigger word determination module, used to determine the event trigger word from the M trigger words; The event triples include K, where K is an integer greater than or equal to 2; The determining the event trigger word from the M trigger words includes: Determining the probability of each trigger word in the trigger word dictionary being selected in the corresponding event category; and Based on the probability of the trigger word being selected in the event category, K event trigger words are determined from the M trigger words.

Citation Information

Cited By

  • Low-resource event extraction method based on large language model enhancement

    CN120234380A