An Automatic Generation and Iterative Optimization Method for Annotation Guidelines Oriented to Event Extraction

Abstract

This invention relates to the field of natural language processing and information extraction technology, and discloses a method for automatic generation and iterative optimization of annotation guidelines for event extraction. The method includes: generating initial annotation guidelines based on annotation samples using a large language model and compressing them into lightweight guidelines for predefined event types; employing a two-stage extraction strategy, first identifying event types using the lightweight guidelines, and then recalling complete guidelines based on the types for end-to-end extraction to obtain prediction results; identifying and clustering prediction errors based on a predefined set of event-layer and argument-layer error types to form error clusters; and generating update operations for the annotation guidelines using a large model for each error cluster, and completing iterative optimization after verification of effectiveness. This invention achieves automated generation and continuous evolution of annotation guidelines, and significantly improves the maintainability, generalization ability, and event extraction performance of the guidelines through structured representation and error clustering mechanisms.

Description

Technical Field

[0001] This invention relates to the field of natural language processing and information extraction technology, specifically to an event extraction method based on a large language model, and further to an automatic generation and iterative optimization method for annotation guidelines for event extraction tasks. Background Technology

[0002] Event extraction is a core task in the field of information extraction. Its goal is to identify and extract structured representations of real-world events from unstructured text, including event types, trigger words, and their participating arguments (such as participants, time, and location). Due to the complexity and diversity of event representations, the fine-grained semantic boundaries, and the involvement of predefined event type systems and argument role specifications, event extraction places extremely high demands on the semantic understanding and discrimination capabilities of models, making it a significant challenge.

[0003] In traditional methods, human annotators typically rely on detailed annotation guidelines to complete event extraction tasks. These guidelines contain extremely fine-grained definitions and numerous exceptions to ensure consistency in annotation. However, traditional deep learning models cannot directly understand the content of these guidelines; instead, they learn them indirectly through large amounts of manually annotated data. When the model is tested in a new domain or under a new annotation system, its performance deteriorates significantly due to insufficient training, and its transferability is limited. This often necessitates rebuilding high-quality training data, which is costly.

[0004] In recent years, the contextual learning capabilities of large language models have brought more possibilities to event extraction. Large models can achieve good results on new tasks with only a small number of examples, without additional fine-tuning. However, even so, their performance still lags behind traditional models because the fine-grained rules required for event extraction are difficult to effectively convey with only a small number of samples. To alleviate this problem, existing research has attempted to replace the traditional small number of examples by providing annotation guidelines to large models. Current work has explored using manually defined annotation guidelines to guide large models in information extraction, showing that annotation guidelines help models distinguish fine-grained event types, improve performance on low-frequency events, and further enhance cross-modal generalization ability. However, existing explorations of annotation guidelines still have significant limitations, mainly in two aspects: First, they rely on high-quality manually defined annotation guidelines. Second, manually defined annotation guidelines often contain a large number of implicit rule definitions, and the expected interpretation of the guidelines often deviates from the actual understanding of the large model, lacking explicit understanding through actual samples. Third, although some work has proposed error-driven guide generation methods, they cannot effectively retrieve and maintain existing guidelines, and lack mechanisms for generating more generalized guidelines.

[0005] Regarding event extraction based on large models, existing technologies employ various approaches. For example, Chinese invention patent CN120234380A discloses a low-resource event extraction method based on large language model enhancement. This method constructs a candidate sample space by performing zero-sample event pre-extraction on unlabeled text, generates an argument candidate set from existing labeled samples, searches for similar event samples using multi-level similarity that integrates text semantics, event structure, and dependencies, generates new samples that maintain trigger word-argument dependencies using a large language model, and finally filters high-quality generated samples based on quality scores and automatically labels them. Through these steps, this type of technology aims to expand training data and improve event extraction performance under few-sample conditions. Essentially, this approach still belongs to the few-sample event extraction paradigm: training or enhancing downstream models with a small number of labeled samples and automatically generated samples, rather than optimizing model behavior at the task rule level. Its limitations are mainly reflected in three points: First, the effect is highly dependent on the quality and distribution of the initial small number of labeled samples. If the original samples do not adequately cover boundary cases or rare argument types, the generated samples will amplify this bias, making it difficult to alleviate the problem of missing fine-grained rules. Second, the generated samples are more of a "superficial diversity" and do not explicitly characterize the event definition, argument boundaries and exclusion conditions. The model still completes the extraction by "imitating the data" rather than "understanding the rules".

[0006] Furthermore, Chinese invention patent CN120579545A, "A Method and System for Named Entity Recognition Based on Representative Sample Selection and Context Enhancement," primarily targets named entity recognition tasks. It optimizes named entity recognition under few-sample conditions through representative sample selection, entity definition construction, dynamic context retrieval, large-model recognition, and error-based prompt updates. Its core process involves: selecting representative samples from unlabeled data as few-sample examples; constructing entity definitions from labeled data; retrieving the most similar examples from the text to be recognized and combining them with prompts; and finally, summarizing annotation guidelines based on prediction errors to generate the next round of prompt text to improve the output. However, this method is limited to named entity recognition and cannot handle the complex relationships involving trigger words, event types, and multi-argument structures in event extraction, resulting in insufficient technical scope. Secondly, it lacks a method for generating generalized annotation guidelines; summarizing guidelines from a single sample may lead to error propagation due to the sample's specificity, thus reducing extraction performance. Thirdly, this method does not propose any retrieval, location, or maintenance mechanisms for guidelines, cannot perform rule-level additions, deletions, or modifications, and cannot support stable multi-round iterations.

[0007] In summary, current event extraction methods based on large language models face the following technical bottlenecks: First, event extraction tasks rely heavily on fine-grained annotation specifications, while contextual learning based on a limited number of examples struggles to fully convey these complex rules. Second, while current research attempts to improve extraction performance by providing manually written annotation guidelines to the model, these guidelines are generally not publicly available or difficult to obtain, and their content often contains numerous implicit rules, leading to a discrepancy between the large language model's understanding of the guidelines and the actual annotation specifications. Third, existing methods based on error-driven or natural language rule-based inductive annotation guidelines still have significant shortcomings. On the one hand, they lack a mechanism to abstract generalizable rules from errors; on the other hand, they lack fine-grained management capabilities for annotation guidelines, making it impossible to add, delete, or modify the guidelines at the rule-level. Therefore, there is an urgent need for an event extraction annotation guide method that can be automatically generated, continuously optimized, and easily maintained to improve the performance and adaptability of large language models in complex event extraction tasks. Summary of the Invention

[0008] To address the issues of lack of high-quality manually defined annotation guidelines, poor maintainability of annotation guidelines, and poor generalization of annotation guidelines generated by large models in existing event extraction tasks based on annotation guidelines, this invention proposes an iterative optimization method for annotation guidelines based on error clustering. This method enables large models to continuously learn from errors and automatically evolve annotation specifications without the need for manual writing of complete guidelines, thereby improving event extraction performance.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] In a first aspect, the present invention provides a method for automatic generation and iterative optimization of annotation guidelines for event extraction, comprising the following steps:

[0011] Step S1: Annotation Guide Initialization and Compression: For each event type in the predefined set of event types, based on its corresponding annotation samples, an initial annotation guide containing event definition and argument definition is generated using a large language model; and each initial annotation guide is compressed to generate a lightweight guide for event type identification.

[0012] Step S2: Two-stage event extraction: For each type of labeled sample, firstly, the lightweight guide set of the current iteration version is used to guide the large language model to identify one or more event types contained in the sample; secondly, based on the identified event type, the corresponding annotation guide is retrieved from the annotation guide set of the current iteration version, and the retrieved annotation guide is used to guide the large language model to perform end-to-end event extraction to obtain the prediction result.

[0013] Step S3: Error Clustering: Compare the prediction results with the true annotations, identify errors based on a predefined set of event-level error types that includes event-level error types and argument-level error types, and aggregate errors belonging to the same error type into error clusters;

[0014] Step S4: Guideline Iterative Update: For each error cluster, analyze the error instances in the error cluster using a large language model to generate an update operation for the current annotation guideline; apply the update operation to obtain the updated annotation guideline and verify its performance. If the event extraction performance is improved, the current iteration is completed.

[0015] Furthermore, the initial annotation guide is a structured representation that contains rule units identified by unique identifiers; the update operation in step S4 is an addition, modification, or deletion operation performed on the target rule unit based on the unique identifier.

[0016] Furthermore, in step S1, the annotation guide initialization step includes:

[0017] A structured annotation guideline framework is constructed, which defines an event-level rule set and an argument-level rule set for each event type, and each rule has a unique identifier;

[0018] For each event type, sample a number of samples from its corresponding labeled samples;

[0019] Based on sampled annotation data and a structured annotation guide framework, prompt words for guide generation are constructed. Input a large language model to generate initial annotation guidelines for this event type; the prompt words It includes task description, event type name, text and annotations in the initial corpus, a structured annotation guideline framework definition, and constraints on the guideline content; the initial corpus is composed of annotation samples corresponding to the event types.

[0020] Further, in step S1, the step of generating lightweight guides includes: constructing prompts for compressing them. Input a large language model to generate compressed text that retains only the key points of the event definition and typical trigger words; the prompt words This includes task descriptions and initial annotation guidelines.

[0021] Furthermore, in step S2, the two-stage event extraction steps include:

[0022] First, event categorization: Constructing prompts for event categorization. Input a large language model to obtain one or more event types contained in the input text; the prompt words It includes an event classification task description, a lightweight guide set for the current iteration, an output format description, and input text; the input text is a labeled sample corresponding to the event type.

[0023] Secondly, end-to-end event extraction: Constructing cue words for end-to-end event extraction. Input the large language model and output the prediction result; the prompt words It includes an end-to-end event extraction task description, a set of annotation guidelines corresponding to the event types predicted in the first stage, an output format description, and input text.

[0024] Furthermore, the predefined set of event extraction error types includes: missing events, multiple extracted events, event type confusion, trigger word boundary error, missing arguments, multiple extracted arguments, argument type confusion, and argument boundary error.

[0025] Furthermore, the verification in step S4 specifically involves: re-extracting events from the samples constituting the error clusters using the updated annotation guidelines, and calculating the difference in argument classification F1 scores before and after the update to determine whether performance has improved.

[0026] Secondly, the present invention provides an apparatus for automatic generation and iterative optimization of annotation guidelines for event extraction, used to implement the method, comprising:

[0027] The guide management module is used to store and provide versioned complete collections of annotation guides and lightweight collections of guides;

[0028] The two-stage extraction module is used to call the large language model and sequentially perform event type identification based on the lightweight guide set and end-to-end event extraction based on the complete annotation guide set.

[0029] The error clustering module is used to identify and classify the differences between the extracted results and the true labels based on a predefined set of error types, forming error clusters;

[0030] The iterative update module is used to generate guide update instructions based on the error cluster-driven large language model, and to perform performance verification and version iteration on the updated guide.

[0031] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described thereon.

[0032] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0033] The present invention has the following beneficial effects:

[0034] (1) The invention achieves automated generation and continuous evolution of event extraction annotation guidelines: By designing a complete initialization and iterative optimization framework, the invention can automatically generate initial guidelines from a small number of annotation samples using a large language model without relying on manual pre-writing of complete and high-quality annotation guidelines. The guidelines can then continuously learn from subsequent extraction errors and optimize their content. This greatly reduces the reliance on professional annotation knowledge and manual costs, and improves the automation level and availability of annotation standards.

[0035] (2) An addressable and maintainable structured annotation guide system was constructed: This invention creatively constructs the annotation guide as a set of rules (including event-level and argument-level) indexed by unique identifiers, realizing fine-grained (rule-level) management of the guide content. Based on this structured representation, the system can accurately perform add, delete, and modify operations on specific rules and achieve version management. This significantly improves the editability, traceability, and maintainability of the annotation guide, laying the foundation for long-term, stable iterative optimization.

[0036] (3) Improved generalization ability and efficiency of rule optimization through error clustering mechanism: This invention differs from methods that repair rules based on single-sample errors. By predefining a standardized error type system and using a large language model to classify specific errors into corresponding error types, a homogeneous error cluster is formed. Guideline updates are driven by error clusters (rather than single samples), enabling the large language model to abstract common problems and general patterns from multiple similar error instances, thereby generating rules that are more generalizable and can better cover a class of boundary cases. This effectively avoids rule fragmentation and overfitting problems, improving the quality and efficiency of guideline optimization.

[0037] (4) The two-stage extraction strategy effectively solves the contradiction between the context constraints of large models and the loading of multiple types of guides: This invention designs a two-stage event extraction strategy of "lightweight guide classification + complete guide extraction". The first stage uses compressed lightweight guides to quickly and accurately identify the event types involved in the text; the second stage loads only the complete guides of the relevant event types for accurate extraction based on the identification results. This design cleverly balances the limitation of the model input context length and the requirement for the integrity of complex guide information of multiple event types, ensuring the feasibility and efficiency of the method under a large-scale event type system. Attached Figure Description

[0038] Figure 1 This is an example of a structured annotation guideline in an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the initialization process of the annotation guide in an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the complete iterative process of the present invention. Detailed Implementation

[0041] To make the present invention easier to understand, the present invention will be further described below with reference to specific embodiments and accompanying drawings. These embodiments and drawings are not intended to limit the present invention in any way. They are only used to illustrate the present invention and are not intended to limit the scope of the present invention. Any modifications or changes to the present invention that are easily implemented by those skilled in the art without departing from the technical solutions of the present invention will fall within the scope of the claims of the present invention.

[0042] This invention provides an automatic generation and iterative optimization method for annotation guidelines oriented towards event extraction. This method automatically constructs and optimizes annotation guidelines around a predefined set of event types to guide large language models in performing event extraction. Through a closed-loop iteration of "extraction-correction-update-verification," the annotation guidelines gradually evolve from an initial version, improving the performance of large language models in event extraction tasks.

[0043] An automatic generation and iterative optimization method for annotation guidelines oriented towards event extraction includes the following steps:

[0044] Step S1: Initialize the annotation guide;

[0045] This invention establishes a structured annotation guide framework for storing and maintaining extraction specifications for each type of event. Furthermore, to accelerate the optimization efficiency of the annotation guide, this invention samples a small number of data points and initializes the annotation guide content using a large model.

[0046] (1) Event type traversal and sample sampling:

[0047] Obtain the labeled event extraction dataset and divide it into a training set and a validation set (obtain a batch of raw text data, manually label at least a portion of the raw text data to obtain labeled data, then select a portion of the labeled data as the training set for training the event extraction model, and another portion as the test set for verifying the final performance of the method).

[0048] The set of all predefined event types in the event extraction task Perform a traversal, for each event type Perform the following operations:

[0049] 1. Filter out events containing the specified event types from the training set. Annotated samples;

[0050] 2. Randomly sample K samples from the labeled samples to form the event types. initial corpus Each sample This includes the original text and the corresponding standard event annotations;

[0051]

[0052] (2) Initial annotation guide generation

[0053] For each event type, an initial annotation guide for that event type is generated using a large language model, specifically including:

[0054] 1. First, a structured annotation guideline framework is established to store and maintain extraction specifications for each event type. This framework uses "rules" as the smallest unit, encoding each rule as a JSON item with a unique identifier (gid). For each event type, an event-level rule set (describing event definitions, trigger words, and type-distinguishing conditions) and an argument-level rule set (describing the extraction boundaries and exceptions for each argument role) are constructed. Based on the unique identifiers of the rules, fine-grained update operations such as adding, modifying, and deleting rules can be performed on the guideline in subsequent iterations, thereby supporting addressable management and continuous optimization of the annotation guideline. A schematic representation of this structured annotation guideline is shown in Figure 1.

[0055] 2. Construct prompts for guide generation The prompts include a task description (a guide for the large model to generate structured annotations for specified events based on given samples), event type names, and the initial corpus. The text and annotations, the structured annotation guidelines framework definition, and the constraints on the content of the guidelines (definition of demand events, definition of arguments, typical trigger words and argument examples, etc.) are all included.

[0056] 3. Use the prompt words and initial corpus Input a large language model to obtain event types Initial annotation guide text :

[0057]

[0058] 4. Collect the initial annotation guide text for all event types to obtain the initial annotation guide set.

[0059] (4) Annotation Guidelines Compression

[0060] To support the subsequent event classification phase, for each event type Initial annotation guide text To compress the data, follow these steps:

[0061] 1. Construct prompts for the compression guide. The prompts include a task description (instructing the large model to retain only key information for event detection and differentiation) and initial annotation guidelines text. ;

[0062] 2. Prompt words With the initial annotation guide text Input a large language model and generate a lightweight guide: The lightweight annotation guide is not merely an abbreviation of the complete guide, but rather serves the crucial functions of event type filtering and information delivery. By retaining only the core semantic features required for event detection, such as key event definitions, typical trigger words, and main distinguishing clues, it significantly reduces text length while maintaining discriminative power.

[0063] 3. Collect lightweight annotation guidelines for all event types to obtain a lightweight guide set. .

[0064] Step S2: Two-stage event extraction based on annotation guidelines

[0065] Due to the limitations of the large model context, it is impossible to input all guidelines completely when there are diverse event types. This invention divides the extraction process into two stages: the first stage, the event classification stage, uses only compressed lightweight annotation guidelines to classify events, identify possible event types in the text, and effectively retrieve and recall annotation guidelines corresponding to event types; the second stage, the end-to-end extraction stage, relies on the recalled annotation guidelines to extract complete events.

[0066] Each iteration samples a batch of samples containing the same event type, in the 1st iteration... During the next iteration, for the event type Randomly sample a set of samples containing this event type from the training set. and for each of the texts The two-stage event extraction process includes the following steps:

[0067] (1) Event Classification

[0068] 1. Construct cue words for event categorization The prompt includes a description of the event classification task (indicating that the large model relies on lightweight annotation guidelines to classify events in the input text) and the set of lightweight guidelines for the current iteration. Output format description, input text ;

[0069] 2. Prompt words With input text Input the large language model together to obtain the input text. The possible set of event types:

[0070]

[0071] 3. According to Recall the large language model from the input text from the current iteration's annotation guide set. The set of annotation guidelines corresponding to each predicted event type:

[0072]

[0073] (2) End-to-end event extraction

[0074] 1. Construct cue words for end-to-end event extraction. The prompt includes a description of the end-to-end event extraction task (instructing the large model to perform end-to-end event extraction using the provided annotation guidelines) and the set of annotation guidelines corresponding to the event types predicted in the first stage. Output format description and input text ;

[0075] 2. Prompt words Input a large language model and output the prediction results:

[0076]

[0077] Step S3: Extracting and clustering error identification

[0078] Instead of employing the existing "single-sample-based error interpretation" approach in the error analysis stage, this invention predefines error types covering both the event and argument layers, structuring the prediction bias of the large model into standardized error labels, and generating corresponding error descriptions from the large model. Based on this, the invention further aggregates errors of the same type into error clusters, enabling unified classification of similar extraction biases under different expressions and contexts.

[0079] In one implementation, in the first In the next iteration, all prediction results are obtained through step S2, and the following operations are performed on each prediction result:

[0080] (1) Constructing error samples

[0081] 1. Standardized and accurate labeling Compared with the prediction results The event text excludes differences in structure, order, and other formatting.

[0082] 2. For the sample Read the actual annotations , and the prediction results If the two samples are completely identical, they are recorded as the correct sample; otherwise, an incorrect sample is constructed.

[0083] (2) Error identification

[0084] 1. To accurately identify the causes of errors, define a finite, mutually exclusive set of error types. This set can fully cover the differences between any prediction result and the true annotation, including:

[0085] a) Event-level error types: Missing event (missing_event( ), multi-draw event (spurious_event( ), event type confusion (event_type_confusion) , ()), trigger_boundary_error( ));

[0086] b) Event argument level error type: Missing argument (missing_argument) , ), spurious_argument( , Argument type confusion (argument_type_confusion) , , Argument boundary error ()) , )).

[0087] 2. Construct error analysis prompt words The prompts include a description of the error identification task (instructing the large model to identify error types and providing error descriptions), and error samples. Event Extraction Error Type Collection .

[0088] 3. Use the prompt words Input large language model and obtain error samples List of structured error identification results:

[0089]

[0090] in, = {errpr_type, error_description}, represents each error identification result, where errpr_type represents the error type and error_description represents the specific error description. The error type is used to perform specific clustering operations in subsequent steps, and the specific error description is used to let the large model understand the localization error in subsequent steps.

[0091] (3) Error clustering

[0092] In this invention, error clustering is not performed directly on a sample-by-sample basis, but rather the error identification result is used as the smallest unit of analysis.

[0093] 1. To facilitate subsequent unified processing, this invention will process each erroneous sample... Each error identification result Combined with its corresponding context information, a structured error entry is constructed, which is defined as a quadruple: For incorrect samples The corresponding set of structured error entries can be represented as:

[0094]

[0095] Each item is a quadruple containing the input text. Authentic labeling Prediction results and error identification results .

[0096] 2. In a batch or iteration, summarize the structured error entries corresponding to all error samples to form a complete set of error entries:

[0097]

[0098] 3. For all error entries in the entire set, cluster them according to error type, and for each error type... Construct the corresponding error cluster:

[0099]

[0100] Where C represents the set of predicted error types in the current batch of samples. Each error cluster contains multiple instances of the same type of error that occur under different input texts.

[0101] Step S4: Annotation Guideline Update and Verification

[0102] This invention employs a batch rule update mechanism based on error clusters during the annotation guide update phase, rather than generating rules for each individual error. The "single error → single rule" update method often generates rules that overly rely on the linguistic expression of a specific sample, easily leading to rule fragmentation and overfitting. This invention, by aggregating multiple instances from a single error cluster, allows large models to summarize common patterns from multiple errors of the same type, statistically filtering out occasional / noise samples and making the generated guidelines more stable.

[0103] (1) Generate guide update operation

[0104] 1. For each error type Corresponding error cluster A set of annotation guidelines for the event types involved in the recall error cluster. .

[0105] 2. Constructing prompt words The prompt includes a task description for generating guideline update operations (instructing the large model to optimize the annotation guideline using similar error samples, generating structured optimization operations), and the error type. Annotation Guide Collection Operation type description and output format constraints.

[0106] 3. Enter prompt words Generate a set of guide update operations corresponding to the error type from the large language model:

[0107]

[0108] Each update operation It consists of operation type and operation parameters. The operation type includes modification, addition and deletion. The parameters are used to specify the guide item number and provide specific guide content.

[0109] (2) Application update operation

[0110] 1. For each update operation Address the target event type based on the target number in the guide bar. Corresponding annotation guidelines After updating the application guidelines, the updated annotation guidelines are obtained. .

[0111] 2. After the annotation guidelines are updated, the annotation guidelines will be recompressed. Get the lightweight guide .

[0112] (3) Verify extraction performance

[0113] 1. Random sampling includes event types sample set Above, use the updated guidelines , Repeat the steps in S2 to obtain new prediction results.

[0114] 2. Use the F1 score of argument classification as the evaluation, calculate the difference ΔF1 between the current F1 score and the original F1 score. If ΔF1>0, it means that the performance index on the error set is better than before the update, and the update of this round of guidelines is accepted; otherwise, the update is discarded.

[0115] 3. Once all update operations have been applied, this iteration is complete, and the update guide collection version is released. .

[0116] Example

[0117] In this embodiment, the method of the present invention is applied to the DuEE Chinese event extraction dataset. Taking the "Competition Behavior - Retirement" event type in the DuEE dataset as an example, the implementation process of this method is illustrated below:

[0118] S1: Annotation Guidelines Initialization and Compression

[0119] S1.1: A collection of all predefined event types in the DuEE event dataset. For the event type "Competition Behavior - Retirement", perform a traversal, select labeled samples containing this time type from the training set, and randomly sample 2 samples from them:

[0120] Sample 1: {"text":"Nasir talks about his relaxed retirement life and suggests Ahmed give his partner more support","event_list":[{"event_type":"Competitive Behavior - Retirement","trigger":"Retirement","arguments":[{"role":"Retiree","argument":"Nasir"}]}]}

[0121] Sample 2: {"text":"After the end of Season 8, Zz1tai announced his retirement and became a League of Legends streamer!","event_list":[{"event_type":"Competitive Behavior - Retirement","trigger":"Retirement" ,"arguments":[{"role":"Retired Person","argument":"Zz1tai"}]}]}

[0122] S1.2: Construct prompts for guide generation The prompts include a task description, event type name, text and annotations from the sampled corpus, a structured annotation guideline format definition, and annotation guideline content constraints. The prompts will be displayed... Inputting a large language model, for the event type "competitive behavior - retirement", and taking the samples sampled in S1.1 as input, the initial annotation guide text output is as follows, denoted as: ：

[0123] {"event_type":"Competitive Behavior - Retirement","event_guideline":[{"gid":"2500-01","content":"Event Definition: The act of an athlete / player or related competitive personnel announcing or ending their professional competition / competition career (including formal announcements of retirement, descriptions of retired status, and expressions indicating the end of a professional career)."},{"gid":"2500-02","content":"Typical trigger words include 'retirement', etc."}],"arguments":[{"argument_type":"Time","argument_guideline":[{"gid":"2501-01","content":"Definition: The time expression of the occurrence or announcement of the retirement event, including the specific date / moment and relative time."}]},{"argument_type":"Retiree","argument_guideline":[{"gid":"2502-01","content":"Definition: The subject of retirement, which can be an individual (athlete, coach, esports player, etc.."}}]}]}

[0124] S1.3: After obtaining the results of S1.2, construct the prompt words for compressing the guide. The prompts include a task description and annotation guidelines. After inputting into the large language model, the compressed lightweight guide for the event type "competitive behavior - retirement" is as follows, denoted as: :

[0125] {"event_type":"Competitive Behavior - Retirement","compressed_guideline":"Retirement refers to an athlete's or competitor's announcement or termination of their professional competitive career. The common trigger word is 'retirement'"}

[0126] S2: Two-stage event extraction based on annotation guidelines

[0127] In the first iteration, a two-stage sampling is performed on a batch of samples for the event type "Competitive Behavior - Retirement". For the event type "Competitive Behavior - Retirement", the two randomly sampled samples are:

[0128] Sample 1: {"text": "On June 13th, Beijing time, three-time Olympic badminton silver medalist and Malaysian badminton legend Lee Chong Wei held a special press conference, announcing his official end to his 19-year professional career." , "trigger": "end", "arguments": [{"role": "time", "argument": "June 13th, Beijing time"}, { "role": "retired", "argument": "three-time Olympic badminton silver medalist and Malaysian badminton legend Lee Chong Wei"}]}

[0129] Sample 2: {"text": "Son of legend Ji Minshang! Guangdong team's 20-year-old player Ji Xiang announces retirement", "trigger": "retirement", "arguments": [{"role": "retired", "argument": "Guangdong team's 20-year-old player Ji Xiang"}]}

[0130] The specific steps for execution are as follows:

[0131] S2.1: Construct prompts for event categorization , prompt words The content includes a description of the event classification task, a lightweight guide set for the current iteration, a description of the output format, and input samples. (Hint words) After inputting the data into the large language model, the possible event types for these two samples are obtained, and the classification result for both is: ["Competitive Behavior - Retirement"]. ， At the same time, there are guidelines for labeling recall events.

[0132] S2.2: After obtaining the classification results from S2.1, construct prompt words for end-to-end event extraction. The prompts include a description of the end-to-end event extraction task, a complete set of guidelines for the first-stage recall, a description of the output format, and the input text. The prompts are then input into a large language model to generate prediction results.

[0133] Sample 1 prediction result: [{"event_type":"Competitive Behavior - Retirement","trigger":"End","arguments":[["Retiree","Lee Chong Wei"],["Time","June 13"]]}]

[0134] Sample 2 prediction result: [{"event_type":"Competitive Behavior - Retirement","trigger":"End","arguments":[["Retiree","Ji Xiang"]]}]

[0135] S3: Extracting Error Detection and Clustering

[0136] S3.1: The prediction results of the two samples in step S2.2, after being normalized using the script code, are inconsistent with the actual annotations and are recorded as erroneous samples. , .

[0137] S3.2: Construct error analysis prompt words The prompt words include a description of the error identification task, error samples, and error type definitions. Input a large model, and the model will identify extraction errors in events based on predefined error types. The output will be:

[0138] Sample 1 Error Analysis Results: [{"origin_event":1,"error_type":"argument_boundary_error(Competition Behavior - Retirement, Retired Person)","error_description":"The model only extracted "Lee Chong Wei," omitting the preceding titles "Three-time Olympic Badminton Silver Medalist, Malaysian Star," resulting in an excessively narrow argument boundary for retired persons."} , {"origin_event":1,"error_type":"argument_boundary_error(Competition Behavior - Retirement, Time)","error_description":"Missing the "Beijing Time" time partition prefix"}]

[0139] Sample 2 error analysis results: [{"origin_event":1,"error_type":"argument_boundary_error(Competition Behavior - Retirement, Retired Player)","error_description":"Only "Ji Xiang" was extracted, and "Guangdong Team 20-year-old player" was not included"}]

[0140] S3.3: Based on the results in S3.2, the erroneous samples... and Clustering was performed based on the error type labels "argument_boundary_error(competition behavior - retirement, retired person)" and "argument_boundary_error(competition behavior - retirement, time)" to obtain cluster p1 and cluster p2.

[0141] S4: Annotation Guidelines Update and Verification

[0142] S4.1: Obtain the sample set clustered by error type in S3.3, and construct the prompt words used to generate the guide update operation. The prompt message includes the current error type, the set of error samples corresponding to that error type, and the annotation guidelines for the current version of the event type. Description of executable operation types (add, modify, delete). Input the prompt words into the large language model, and the model will automatically generate a list of update operations. For clusters p1 and p2, output the guide operations:

[0143] Annotation guidelines for cluster p1: [{"operation_type":"add","arguments":{"gid":"2502-02","content":"Retired individuals usually appear in the form of 'name' or 'with title + name'. When the structure 'title / achievement + player's name' appears (e.g., 'Three-time Olympic badminton silver medalist, Malaysian star Lee Chong Wei'), the entire retired individual should be extracted as an argument, and only the last part of the name should not be extracted."}}]

[0144] Annotation guidelines for cluster p2: [{"operation_type":"add","arguments":{"gid":"2501-02","content":"When the original time expression contains time zone / time system cues, the annotated time span must include them; you cannot simply retain the date or time itself."}}]

[0145] S4.2: The program parses and applies the annotation guidelines from S3.3 to obtain new annotation guidelines:

[0146] {"event_type":"Competitive Behavior - Retirement","event_guideline":[{"gid":"2500-01","content":"Event Definition: The act of an athlete / competitor or related competitive personnel announcing or ending their professional competition / competition career (including formal announcements of retirement, descriptions of retired status, statements of ending a professional career, etc.)."},{"gid":"2500-02","content":"Typical trigger words include 'retirement', etc."}],"arguments":[{"argument_type":"Time","argument_guideline":[{"gid":"2501-01","content":"Definition: The time expression of the retirement event, including the specific date / moment and relative time."},{"gid":"2501-02","content":"When the original time expression contains time zone / time system prompts, the time span of the annotation is..."} The argument must include the date or time itself; it is not acceptable to retain only the date or time itself. {"argument_type":"Retired Person","argument_guideline":[{"gid":"2502-01","content":"Definition: The subject of retirement, which can be an individual (athlete, coach, esports player, etc.). "},{"gid":"2502-02","content":"Retired persons usually appear in the form of "name" or "with title + name". When the structure "title / achievement + player's name" appears (e.g., "three-time Olympic badminton silver medalist, Malaysian star Lee Chong Wei"), it should be extracted as a whole as the retired person argument, and not just the last part of the name. "}]}]}

[0147] S4.3: To determine whether the update is effective, the updated annotation guideline is used to re-execute the two-stage event extraction step S2 on the samples that generated the current erroneous cluster. The argument classification F1 scores on this batch of samples are calculated using both the pre-update and post-update guidelines, resulting in the difference ΔF1. In this embodiment, ΔF1 > 0 indicates that the update is effective, therefore this round of updates is accepted, and this iteration is completed. ΔF1 > 0, therefore this round of iterations is complete.

[0148] This invention is compared with existing event extraction methods for large models. Table 1 shows the evaluation results, which report the extraction performance of the event extraction on the DuIE and ACE05 datasets.

[0149] Dataset

[0150] The ACE2005 dataset is a classic event extraction dataset released by the National Institute of Standards and Technology (NIST), primarily targeting English text scenarios. This dataset contains text of various genres and defines a structured event type system and argument role specifications. Event trigger words and their argument information are manually annotated, and it is commonly used for training and performance evaluation of event extraction models.

[0151] The DuEE dataset is a large-scale event extraction dataset built for Chinese scenarios, containing real news texts from multiple domains. This dataset uses predefined event types and argument structures to annotate event trigger words and their related arguments, making it suitable for training, validating, and comparing Chinese event extraction models.

[0152] Existing methods

[0153] ChatIE is a multi-turn question-answering framework for zero-shot information extraction, in which the first stage collects all possible event types, and the second stage performs information extraction for each event type separately.

[0154] GuidelineEE is an event extraction method based on annotation guidelines. It guides large language models to more accurately understand the semantics and structural constraints of events by explicitly introducing the textual definition of event types and their argument roles into the instruction fine-tuning process.

[0155] DecomposeEE is an event extraction method for large language models, designed to alleviate the illusion problem in information extraction tasks of generative models. This method decomposes the event extraction task into two sub-tasks: event detection and event argument extraction, and guides each sub-task with refined cue words.

[0156] AEC (Agent-Event-Coder) is a multi-agent framework for zero-shot event extraction, modeling event extraction as a structured code generation process. This method involves multiple specialized agents working collaboratively, including retrieval, planning, encoding, and verification, and represents event patterns as executable code, thereby achieving strict constraints and iterative correction of the event structure.

[0157] Based on previous research, this invention employs four standard event extraction evaluation metrics: (1) Trigger Word Recognition (TI): measures the model's accurate matching of trigger word ranges; (2) Trigger Word Classification (TC): further requires correct prediction of the event type for each trigger word based on TI; (3) Argument Recognition (AI): evaluates whether the model can accurately extract the range of arguments associated with the predicted trigger word; and (4) Argument Classification (AC): the most comprehensive metric, requiring correct prediction of the role type for each argument in addition to argument recognition. This invention reports the average microF1 score for all metrics on the constructed test set partition, where Ours represents the performance of this method.

[0158]

[0159] Experimental results:

[0160] Experimental data show that this method outperforms previous methods based on large models on both the DuEE and ACE2005 datasets.

[0161] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. However, the above description is merely a specific embodiment of the present invention, and the technical features of the present invention are not limited thereto. Any other embodiments derived by those skilled in the art without departing from the technical solution of the present invention should be covered within the patent scope of the present invention.

Claims

1. A method for automatic generation and iterative optimization of annotation guidelines for event extraction, characterized in that, Includes the following steps: Step S1: Annotation Guide Initialization and Compression: For each event type in the predefined set of event types, based on its corresponding annotation samples, an initial annotation guide containing event definition and argument definition is generated using a large language model; and each initial annotation guide is compressed to generate a lightweight guide for event type identification. Step S2: Two-stage event extraction: For each type of labeled sample, firstly, the lightweight guide set of the current iteration version is used to guide the large language model to identify one or more event types contained in the sample; secondly, based on the identified event type, the corresponding annotation guide is retrieved from the annotation guide set of the current iteration version, and the retrieved annotation guide is used to guide the large language model to perform end-to-end event extraction to obtain the prediction result. Step S3: Error Clustering: Compare the prediction results with the true annotations, identify errors based on a predefined set of event-level error types that includes event-level error types and argument-level error types, and aggregate errors belonging to the same error type into error clusters; Step S4: Guideline Iterative Update: For each error cluster, analyze the error instances in the error cluster using a large language model to generate an update operation for the current annotation guideline; apply the update operation to obtain the updated annotation guideline and verify its performance. If the event extraction performance is improved, the current iteration is completed.

2. The method for automatic generation and iterative optimization of annotation guidelines for event extraction according to claim 1, characterized in that, The initial annotation guide is a structured representation that contains rule units identified by unique identifiers; the update operation in step S4 is an addition, modification, or deletion operation performed on the target rule unit based on the unique identifier.

3. The method for automatic generation and iterative optimization of annotation guidelines for event extraction according to claim 1, characterized in that, In step S1, the annotation guide initialization steps include: A structured annotation guideline framework is constructed, which defines an event-level rule set and an argument-level rule set for each event type, and each rule has a unique identifier; For each event type, sample a number of samples from its corresponding labeled samples; Based on sampled annotation data and a structured annotation guide framework, prompt words for guide generation are constructed. Input a large language model to generate initial annotation guidelines for this event type; the prompt words It includes task description, event type name, text and annotations in the initial corpus, a structured annotation guide framework definition, and constraints on the guide content; the initial corpus is composed of annotation samples corresponding to the event types.

4. The method for automatic generation and iterative optimization of annotation guidelines for event extraction according to claim 1, characterized in that, In step S1, the step of generating a lightweight guide includes: constructing prompts for compressing them. Input a large language model to generate compressed text that retains only the key points of the event definition and typical trigger words; the prompt words This includes task descriptions and initial annotation guidelines.

5. The method for automatic generation and iterative optimization of annotation guidelines for event extraction according to claim 1, characterized in that, In step S2, the two-stage event extraction steps include: First, event categorization: Constructing prompts for event categorization. Input a large language model to obtain one or more event types contained in the input text; the prompt words It includes an event classification task description, a lightweight guide set for the current iteration, an output format description, and input text; the input text is a labeled sample corresponding to the event type. Secondly, end-to-end event extraction: Constructing cue words for end-to-end event extraction. Input the large language model and output the prediction result; the prompt words It includes an end-to-end event extraction task description, a set of annotation guidelines corresponding to the event types predicted in the first stage, an output format description, and input text.

6. The method for automatic generation and iterative optimization of annotation guidelines for event extraction according to claim 1, characterized in that, The predefined set of event extraction error types includes: missing events, multiple events extracted, event type confusion, trigger word boundary error, missing arguments, multiple arguments extracted, argument type confusion, and argument boundary error.

7. The method for automatic generation and iterative optimization of annotation guidelines for event extraction according to claim 1, characterized in that, The verification in step S4 specifically involves: re-extracting events from the samples that constitute the erroneous clusters using the updated annotation guidelines, and calculating the difference in the argument classification F1 score before and after the update to determine whether the performance has improved.

8. An apparatus for automatically generating and iteratively optimizing annotation guidelines for event extraction, used to implement the method described in any one of claims 1-7, characterized in that, include: The guide management module is used to store and provide versioned complete collections of annotation guides and lightweight collections of guides; The two-stage extraction module is used to call the large language model and sequentially perform event type identification based on the lightweight guide set and end-to-end event extraction based on the complete annotation guide set. The error clustering module is used to identify and classify the differences between the extracted results and the true labels based on a predefined set of error types, forming error clusters; The iterative update module is used to generate guide update instructions based on the error cluster-driven large language model, and to perform performance verification and version iteration on the updated guide.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images