A few-shot passage-level event extraction method based on a large language model thought chain

Abstract

The present application relates to the technical field of natural language processing, and particularly relates to a few-shot passage-level event extraction method based on a large language model thinking chain, comprising: in order to adapt to a few-shot task, processing to-be-extracted data into two parts of a support set and a query set; writing a task description and requirements of passage-level event extraction; constructing an event argument table and an event definition table, and indexing event information to expand the content of the thinking chain; constructing a positive example according to the support set of the to-be-extracted data, and constructing to-be-extracted text according to the query set of the to-be-extracted data; using the event type, argument type, event definition, positive example, to-be-extracted text, task description of passage-level event extraction and other supplementary texts of a single to-be-extracted data, designing a thinking chain for multiple rounds of questioning of the large language model; inputting the complete thinking chain into the large language model for multiple rounds of question answering, obtaining the generated content of the large language model, and processing the generated content to obtain the final extraction result.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a method for extracting few-sample, chapter-level events based on the thought chain of a large language model. Background Technology

[0002] With the development of the internet, a large amount of event information is widely disseminated through various channels. Document-level event extraction, as a subtask in the field of natural language processing, involves extracting structured event information from document-level text. This helps users quickly and accurately capture event information of interest from a large amount of data.

[0003] Current research largely focuses on sentence-level event extraction, which extracts event information from a single sentence. However, when dealing with document-level texts, this approach suffers from common problems encountered in document-level event extraction (argument dispersion, where arguments for an event are scattered across multiple sentences, and role overlap, where an argument may play different roles in different events within a document). This results in incomplete and inadequate extraction of event information (number of events, type, and elements). To address these common problems in document-level event extraction, such as argument dispersion and role overlap, existing models are highly complex and sophisticated, making training difficult, and they also exhibit poor performance across different domains and datasets.

[0004] Furthermore, the high cost of manual annotation of event information leads to a scarcity of high-quality datasets, and in practical applications, there are generally not enough labeled samples for model training. To handle emerging event information with limited annotations and adapt it to real-world document-level scenarios, few-shot document-level event extraction has become a research hotspot, offering significant research value and a high degree of challenge. However, due to long documents and limited examples, this novel task is extremely difficult, and traditional models generally perform poorly, resulting in low extraction scores. Therefore, large language models that can generate results with just a few hints have the potential to solve the few-shot event extraction task.

[0005] Chinese patent document CN202211669257.7 discloses a method, apparatus, device, and storage medium for text-level event extraction based on machine reading comprehension. This method provides an input text and a corresponding fill-in-the-blank template. The model generates fill-in-the-blanks for the corresponding argument slots in the template based on the text content, thus extracting event arguments. This invention addresses the problems of text-level event extraction and few-shot event extraction. It utilizes machine reading comprehension mechanisms to implement traditional event extraction tasks, providing a pattern information fusion method for event extraction at the fill-in-the-blank template level and a machine reading comprehension method based on a pre-trained language model. This invention further discloses an end-to-end encoder-decoder model. By leveraging the natural language understanding capabilities of the pre-trained language model and mining its latent knowledge, it can improve the model's generalization performance in few-shot scenarios and enhance its adaptability in extraction tasks across different domains.

[0006] Machine reading comprehension methods typically require designing numerous templates, which may be limited by specific language structures and event types. In contrast, the large language model-based thought chain approach only requires designing a general thought chain structure, eliminating the need for numerous predefined templates and allowing for more flexible handling of different event types and structures.

[0007] Compared to machine reading comprehension methods, this invention does not require training with a training set, greatly reducing training time and resource consumption.

[0008] Compared to simple template methods, the thought chains of this invention can better utilize the contextual information of the text, which means they can better understand the correlation between events, and the invention performs better when dealing with event extraction from few-sample texts. Summary of the Invention

[0009] The purpose of this invention is to provide a few-sample document-level event extraction method based on the thought chain of a large language model, in order to solve the problems of low performance, very complex model structure, great training difficulty, and poor performance across domains and datasets when existing traditional models are dealing with few-sample document-level event extraction tasks due to the difficulty of having only a very limited number of example samples.

[0010] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:

[0011] A few-sample, text-level event extraction method based on the thought chain of a large language model includes the following steps:

[0012] S1: To adapt to tasks with few samples, the data to be extracted is processed into two parts: a support set and a query set.

[0013] S2: Write the task description and requirements for chapter-level event extraction, as the content of the thought chain;

[0014] S3: Construct an event argument table and an event definition table based on the event types and argument types of all the data to be extracted. Use the event argument table to index the event type of a single data item, and then use the event definition table to index the definition of the event. Use the event type, argument type, and the definition of the event as the content of the thought chain.

[0015] S4: Construct positive examples based on the support set of the data to be extracted, and construct the text to be extracted based on the query set of the data to be extracted. Use the positive examples and the text to be extracted as the content of the thought chain.

[0016] S5: Using the event type, argument type, event definition, positive example, text to be extracted, task description of chapter-level event extraction, and other supplementary text of a single piece of data to be extracted, design a thought chain for asking multiple rounds of questions to a large language model.

[0017] S6: Input the complete thought chain into the large language model for multiple rounds of question and answer, obtain the generated content of the large language model, process the generated content to obtain the final extraction result;

[0018] Furthermore, step S1 specifically includes: given a support set m s and query set m q ;

[0019] Define the data set to be extracted as M = {m s ,m q}, Among them (b) i ,t i ) indicates the support set m s and query set m q Chinese document (doc) i The boundary of the type of the i-th event argument. It is a doc i The set of all event arguments in the text;

[0020] Furthermore, step S3 specifically includes: based on the event argument type a of each data point in the dataset M to be extracted. i The event argument table T is constructed by classifying the events and identifying the corresponding argument types for each event type. A ;

[0021] Then, based on the event argument table T A The event type and its corresponding argument type are input into the large language model, and the model is prompted to complete the definition of the event, thus obtaining the definition and description of each event type, and finally the event definition table T. D ;

[0022] Define the event argument table as T A (E,A), define the event definition table as T. D (E,D), where E represents the set of event types {e1,e2,…,e...} contained in the data to be extracted. n}, where A represents the set of event argument types {a1, a2, ..., a} contained in the data to be extracted. n}, where D represents the set of definitions {d1, d2, ..., d} for each event type contained in the data to be extracted. n}, where n is the number of event types;

[0023] Furthermore, the data set to be extracted, given a single piece of data to be extracted m i The support set containing this data The query set for this data and the event argument type a of this data. i Defined as And support set m s Includes support for set text sequences w s Supports set label sequences l s Query set m q It contains the query set text sequence w q Query set label sequence l q , defined as m s ={w s ,l s},m q ={w q ,l q The data set M to be extracted is the data set {m1, m2, ..., m}. n}, where n is the number of data points to be extracted;

[0024] Furthermore, step S4 specifically includes: obtaining the data set M to be extracted, and extracting the support set m from it. s and query set m q Given a text sequence w and a label sequence l; convert the text sequence w into natural text w. p Transform the label sequence l into the correct answer l p The natural text can be used to populate the prompt template to complete the construction of the thought chain:

[0025] w p =σ1(w)

[0026] l p =σ2(l)

[0027] Where σ1 represents the text sequence processing function; σ2 represents the label sequence processing function;

[0028] Define the correct example in the thought chain: P = {w sp ,l sp};

[0029] Where w sp To support well-processed natural text sets, l sp The correct answer is obtained from the processed support set;

[0030] Given a single data point, the text to be extracted is It is natural text obtained by processing a text sequence of a single query set;

[0031] Furthermore, step S5 specifically includes: locating the event type of the data based on the event argument table, and then indexing the definition and description of the event based on the event definition table; given a single piece of data m to be extracted... i According to the type a of the data event argument it contains. i In the event argument table T A The event type e that indexes this data in (E,A) is... i Then, based on the indexed event type e i Go to event definition table T D (E,D) indexes the definition and description of this event type. i .

[0032] The complete thought process for designing a multi-round questioning approach for a large language model includes:

[0033] Defining the thought process of a single data point

[0034] Where e i The event type of this data; a i The argument type for this data event; d i This defines and describes the event type; P i This is a correct example of the data set after processing. 'r' is the text to be extracted from this data; 'r' is the fixed natural text in the thought chain that provides prompts to the large language model, which includes: the task requirement template, i.e., the task description and requirements for providing chapter-level event extraction, the necessary supplementary text before providing data to the large language model, and the supplementary requirements put forward to the large language model, etc.

[0035] Furthermore, step S6 specifically includes: inputting the previously constructed thought chain into the large language model for multi-round question-and-answer processing.

[0036]

[0037] Where hi This refers to the final content generated by the large language model for this data; LLM stands for Large Language Model; T i The thought process behind this data;

[0038] For the final generated content h i Extract the dictionary table using regular expressions. That is, predict the answer. Then, based on the dictionary table... The key and value, i.e., the argument type And the original text of the prediction Generate a predicted label sequence similar to the label sequence of the query set. In order to perform performance verification:

[0039]

[0040]

[0041] Where σ3 is the regular expression for processing the generated content; σ4 is the tag generation function; This is a dictionary table extracted from content generated from a large language model. The argument type of the dictionary table; The predicted original text content for the dictionary table; It is a predicted label sequence;

[0042] On the other hand, this invention proposes a few-sample, text-level event extraction system based on the thought chain of a large language model, comprising:

[0043] The data preprocessing module divides the data to be extracted into two parts: a support set and a query set, to adapt to tasks with few samples.

[0044] The extension module is used to expand the content of the thought process, including writing the task description and requirements for chapter-level event extraction; indexing event types, argument types and the definition of the event; and constructing positive examples and texts to be extracted.

[0045] The thought chain construction module uses the event type, argument type, event definition, positive example, text to be extracted, task description of chapter-level event extraction, and other supplementary text of a single piece of data to be extracted to design a thought chain for multiple rounds of questioning of a large language model.

[0046] The extraction module is used to input the complete thought chain into the large language model for multi-round question answering, and obtain the output text of the test text in the query set by extracting a small number of chapter events. The output text is then processed to obtain the final extraction result.

[0047] On the other hand, the present invention proposes an electronic device, comprising: a processor and a memory;

[0048] The memory is used to store one or more program instructions;

[0049] The processor is used to run one or more program instructions to execute the above-mentioned method and system for extracting few-sample, chapter-level events based on the thought chain of a large language model.

[0050] The beneficial effects of this invention are:

[0051] Because large language models learn from massive amounts of data during training, they possess strong generalization capabilities. Even with limited samples, they can leverage pre-trained knowledge to infer unseen situations and provide reasonable solutions. This invention utilizes the generalization capability of large language models, enabling them to extract events even with limited samples, directly reducing reliance on large amounts of manually labeled data.

[0052] Furthermore, by leveraging the deep semantic understanding capabilities of large language models, this invention can accurately identify and extract event information from text. This is reflected in the model's understanding of the complex semantic relationships (such as causality, contrast, and parallelism) implicit in textual discourse, enabling it to grasp the meaning of the text at a deep level and thus achieve precise location and extraction of events.

[0053] Compared to traditional constructing prompt templates, the thought chain constructed in this invention adds reasoning steps for processing tasks. By simulating the human thinking process, it effectively improves the reasoning ability of large language models for complex tasks, enhances the understanding and utilization of information in a small number of samples, and enables large language models to generate answers according to a specified format.

[0054] The method of this invention does not rely on large amounts of domain-specific data. Through the construction of thought chains and multi-turn question-and-answer using a large language model, the model can adapt to different domains and datasets. Technically, this means that the model can learn more general language features and event patterns during training, thus maintaining high extraction accuracy and robustness when faced with text from different domains.

[0055] This invention simplifies the model structure by introducing a thought chain and eliminates the need for training a training set, thus reducing the difficulty and resource consumption of model training. Furthermore, the few-shot performance of this invention surpasses that of traditional models with complex structures and high training difficulty.

[0056] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 A flowchart of a few-sample, chapter-level event extraction method based on a large language model thinking chain provided by the present invention;

[0059] Figure 2 A schematic diagram illustrating the principle of a few-sample, chapter-level event extraction method based on a large language model thinking chain provided by this invention.

[0060] Figure 3 This is a schematic diagram of the structure of a few-sample, chapter-level event extraction system based on a large language model thinking chain, provided by the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Example 1

[0063] This embodiment describes a few-sample, text-level event extraction method based on a large language model's thought chain, comprising the following steps:

[0064] S1: To adapt to tasks with few samples, the data to be extracted is processed into two parts: a support set and a query set.

[0065] S2: Write the task description and requirements for chapter-level event extraction, as the content of the thought chain;

[0066] S3: Construct an event argument table and an event definition table based on the event types and argument types of all the data to be extracted. Use the event argument table to index the event type of a single data item, and then use the event definition table to index the definition of the event. Use the event type, argument type, and the definition of the event as the content of the thought chain.

[0067] S4: Construct positive examples based on the support set of the data to be extracted, and construct the text to be extracted based on the query set of the data to be extracted. Use the positive examples and the text to be extracted as the content of the thought chain.

[0068] S5: Using the event type, argument type, event definition, positive example, text to be extracted, task description of chapter-level event extraction, and other supplementary text of a single piece of data to be extracted, design a thought chain for asking multiple rounds of questions to a large language model.

[0069] S6: Input the complete thought chain into the large language model for multiple rounds of question and answer, obtain the generated content of the large language model, process the generated content to obtain the final extraction result;

[0070] In this embodiment, step S1 specifically includes: given a support set m s and query set m q ;

[0071] Define the data set to be extracted as M = {m s ,m q}, Among them (b) i ,t i ) indicates the support set m s and query set m q Chinese document (doc) i The boundary of the type of the i-th event argument. It is a doc i The set of all event arguments in the text;

[0072] In this embodiment, step S3 specifically includes: according to the event argument type a of each data point in the dataset M to be extracted. i The event argument table T is constructed by classifying the events and identifying the corresponding argument types for each event type. A ;

[0073] Then, based on the event argument table T A The event type and its corresponding argument type are input into the large language model, and the model is prompted to complete the definition of the event, thus obtaining the definition and description of each event type, and finally the event definition table T. D ;

[0074] Define the event argument table as T A (E,A), define the event definition table as T. D (E,D), where E represents the set of event types {e1,e2,…,e...} contained in the data to be extracted. n}, where A represents the set of event argument types {a1, a2, ..., a} contained in the data to be extracted. n}, where D represents the set of definitions {d1, d2, ..., d} for each event type contained in the data to be extracted. n}, where n is the number of event types;

[0075] In this embodiment, the data set to be extracted is given as a single data entry m. i The support set containing this data The query set for this data and the event argument type a of this data. i Defined as And support set m s Includes support for set text sequences w s Supports set label sequences l s Query set m q It contains the query set text sequence w q Query set label sequence l q , defined as m s ={w s ,l s},m q ={w q ,l q The data set M to be extracted is the data set {m1, m2, ..., m}. n}, where n is the number of data points to be extracted;

[0076] In this embodiment, step S4 specifically includes: obtaining the data set M to be extracted, and extracting the support set m from it. s and query set m q Given a text sequence w and a label sequence l; convert the text sequence w into natural text w. p Transform the label sequence l into the correct answer l p The natural text can be used to populate the prompt template to complete the construction of the thought chain:

[0077] w p =σ1(w)

[0078] l p =σ2(l)

[0079] Where σ1 represents the text sequence processing function; σ2 represents the label sequence processing function;

[0080] Define the correct example in the thought chain: P = {w sp ,l sp};

[0081] Where w sp To support well-processed natural text sets, l sp The correct answer is obtained from the processed support set;

[0082] Given a single data point, the text to be extracted is It is natural text obtained by processing a text sequence of a single query set;

[0083] In this embodiment, step S5 specifically includes: locating the event type of the data based on the event argument table, and then indexing the definition and description of the event based on the event definition table; given a single piece of data m to be extracted... i According to the type a of the data event argument it contains. i In the event argument table T A The event type e that indexes this data in (E,A) is... i Then, based on the indexed event type e i Go to event definition table T D (E,D) indexes the definition and description of this event type. i .

[0084] The complete thought process for designing a multi-round questioning approach for a large language model includes:

[0085] Defining the thought process of a single data point

[0086] Where e i The event type of this data; a i The argument type for this data event; d i This defines and describes the event type; P i This is a correct example of the data set after processing. 'r' is the text to be extracted from this data; 'r' is the fixed natural text in the thought chain that provides prompts to the large language model, which includes: the task requirement template, i.e., the task description and requirements for providing chapter-level event extraction, the necessary supplementary text before providing data to the large language model, and the supplementary requirements put forward to the large language model, etc.

[0087] In this embodiment, step S6 specifically includes: inputting the previously constructed thought chain into the large language model for multi-round question-and-answer processing.

[0088]

[0089] Where h i This refers to the final content generated by the large language model for this data; LLM stands for Large Language Model; T i The thought process behind this data;

[0090] For the final generated content h i Extract the dictionary table using regular expressions. That is, predict the answer. Then, based on the dictionary table... The key and value, i.e., the argument type And the original text of the prediction Generate a predicted label sequence similar to the label sequence of the query set. In order to perform performance verification:

[0091]

[0092]

[0093] Where σ3 is the regular expression for processing the generated content; σ4 is the tag generation function; This is a dictionary table extracted from content generated from a large language model. The argument type of the dictionary table; The predicted original text content for the dictionary table; It is a predicted label sequence;

[0094] On the other hand, this invention proposes a few-sample, text-level event extraction system based on the thought chain of a large language model, comprising:

[0095] The data preprocessing module divides the data to be extracted into two parts: a support set and a query set, to adapt to tasks with few samples.

[0096] The extension module is used to expand the content of the thought process, including writing the task description and requirements for chapter-level event extraction; indexing event types, argument types and the definition of the event; and constructing positive examples and texts to be extracted.

[0097] The thought chain construction module uses the event type, argument type, event definition, positive example, text to be extracted, task description of chapter-level event extraction, and other supplementary text of a single piece of data to be extracted to design a thought chain for multiple rounds of questioning of a large language model.

[0098] The extraction module is used to input the complete thought chain into the large language model for multi-round question answering, and obtain the output text of the test text in the query set by extracting a small number of chapter events. The output text is then processed to obtain the final extraction result.

[0099] On the other hand, the present invention proposes an electronic device, comprising: a processor and a memory;

[0100] The memory is used to store one or more program instructions;

[0101] The processor is used to run one or more program instructions to execute the above-mentioned method and system for extracting few-sample, chapter-level events based on the thought chain of a large language model.

[0102] Example 2

[0103] This embodiment describes a few-sample, text-level event extraction method based on a large language model's thought chain. The following section will combine... Figure 1 and Figure 2 Please provide an explanation.

[0104] like Figure 1 As shown, in step S1, to adapt to the few-sample task, the data to be extracted is processed into two parts: a support set and a query set.

[0105] The dataset used in this embodiment is the FewDocAE dataset. FewDocAE is a few-sample document-level event argument extraction dataset based on DocEE, the largest document-level event extraction dataset. Therefore, the structure of this dataset already includes the support set and the query set.

[0106] like Figure 1 As shown, in step S2, the task description and requirements for chapter-level event extraction are written as the content of the thought chain.

[0107] This embodiment describes a task for document-level event extraction applicable to the FewDocAE dataset:

[0108] ['…,whose goal is to extract the argument information of structuredevents from text when the event type is known…']

[0109] like Figure 1 As shown, in step S2, an event argument table and an event definition table are constructed based on the event types and argument types of all the data to be extracted. The event argument table is used to index the event type of a single data item, and the event definition table is used to index the definition of the event. The event type, argument type, and definition of the event are used as the content of the thought chain.

[0110] In this embodiment, the FewDocAE dataset, i.e., the data set M to be extracted, is obtained, and each data point in this dataset is given as m. i m i The support set containing this data The query set for this data and the event argument type a of this data. i Defined as And support set m s Includes support for set text sequences w s Supports set label sequences l s Query set m q It contains the query set text sequence w q Query set label sequence l q , defined as m s ={w s ,l s},m q ={wq ,l q};

[0111] Where M is the FewDocAE dataset {m1, m2, ..., m} n}, where n is the number of data points in the FewDocAE dataset;

[0112] For each data sample m in the FewDocAE dataset i Based on event argument type a i The event argument table T, containing 59 event types, was obtained by classifying the event types and identifying the corresponding argument types for each event type. A (E,A). Then, based on the event argument table T... A The 59 event types and their corresponding argument types are input into the large language model, and the model is prompted to supplement the definition of the event, resulting in the definition and description of each event type, thus obtaining the event definition table T. D (E,D), where E represents the set of event types {e1,e2,…,e...} contained in the data to be extracted. n}, where A represents the set of event argument types {a1, a2, ..., a} contained in the data to be extracted. n}, where D represents the set of definitions {d1, d2, ..., d} for each event type contained in the data to be extracted. n}, where n is the number of event types.

[0113] After constructing the event argument table and event definition table, randomly sample one data point from the FewDocAE dataset as the data to be extracted. i Using the event argument type a of the data i Remove event argument table T A The index retrieves the event type e corresponding to the data. i Then use the event type e i Go to event definition table T D The index retrieves the event definition and description corresponding to the data. i .

[0114] The large language model in this embodiment uses a commercial large language model API service, and the model called has more than 20 billion parameters.

[0115] like Figure 1 As shown, in step S4, positive examples are constructed based on the support set of the data to be extracted, and text to be extracted is constructed based on the query set of the data to be extracted. The positive examples and text to be extracted are used as the content of the thought chain.

[0116] In this embodiment, the support set m is first extracted from the FewDocAE dataset, i.e., the dataset M to be extracted. sand query set m q The FewDocAE dataset contains a text sequence w and a label sequence l. Because the FewDocAE dataset is in JSON format, with words separated by commas and quotation marks, it is not suitable for asking questions or providing examples for a large language model. Therefore, the text sequence needs to be processed into natural text for constructing prompts, and the label sequence needs to be processed into a formatted example.

[0117] Pre-processing support set m s and query set m q Here are some examples:

[0118] ['AGL','issued','$','1','million','penalty','by','Protection'……]

[0119] ['Agency','O','O','Amount','Amount','O','O','Regulatory'......]

[0120] Convert the text sequence w into natural text w p This allows the text to be populated into the prompt template to complete the construction of the thought chain:

[0121] w p =σ1(w)

[0122] Where σ1 represents the text sequence processing function;

[0123] Convert the label sequence l into the correct answer l p The natural text can be used to populate the prompt template to complete the construction of the thought chain:

[0124] l p =σ2(l)

[0125] Where σ2 represents the tag sequence processing function;

[0126] After processing, support set m s and query set m q Here are some examples:

[0127] [AGL issued $1million penalty by Environment……]

[0128] ['Amount':'1million','Regulatory':'Protection Authority','Agency':'AGL']

[0129] Then query set mq The processed text sequence is used as the text to be extracted. Define the correct example in the thought chain: P = {w sp ,l sp}, where w sp To support well-processed natural text sets, l sp The correct answer is obtained from the processed support set;

[0130] like Figure 1 As shown, in step S5, the event type, argument type, event definition, positive example, text to be extracted, task description of chapter-level event extraction, and other supplementary text of a single piece of data to be extracted are used to design a thought chain for multiple rounds of questioning of the large language model.

[0131] Defining the thought chain T for a single data point i ={r,e i ,a i ,d i ,P i ,w qpi};

[0132] In this embodiment, e i The event type of this data; a i The argument type for this data event; d i This defines and describes the event type; P i This is a correct example obtained after processing the support set for this data; w qpi 'r' is the text to be extracted from this data; 'r' is the fixed natural text in the thought chain that provides prompts to the large language model, which includes: the task requirement template, i.e., the task description and requirements for providing chapter-level event extraction, the necessary supplementary text before providing data to the large language model, and the supplementary requirements put forward to the large language model, etc.

[0133] By constructing a thought chain adapted to the FewDocAE dataset using the above content, and in order to meet the input requirements of the commercial large language model API service called in this example, the thought chain T needs to be modified. i The format needs to be modified further:

[0134] X i =σ5(T i )

[0135] Where X i σ5 represents the processed thought chain; σ5 is the function for processing the thought chain format.

[0136] Modified Mind Chain X i Example as follows:

[0137] ['role':'user','content':'Help me solve a document-level eventargument extractiontask…',

[0138] 'role':'user','content':'Provides event types,their definitions for the texttobe extractednext:1.eventtype:{e i}-{d i}2.argumenttype:{a i}...',

[0139] 'role':'user','content':'Provides a positive sample of the text to beextracted next,showing the format of the input and expected output:{P i}…',

[0140] 'role':'user','content':'Based on the above prompts and positive examples,extract this input and provide me the expected outputdirectly.Input: …']

[0141] Although we've already used the mind chain approach to help the large language model understand the task's requirements and output, we still need to further refine the mind chain X in order to ensure that the large oracle model's response closely matches our expected output. i Add the following notes to the `r` function: First, the large language model is required not to summarize, ensuring that the expected output is extracted from the input provided later, preventing extraction failure due to discrepancies between the output and the expectation. Second, the final output should be a dictionary containing argument types and text. Third, if the argument type is not mentioned in the text, mark it as "not mentioned".

[0142] like Figure 1As shown, in step S6, the complete thought chain is input into the large language model for multiple rounds of question answering to obtain the output text of the text to be tested in the query set by extracting few-sample chapter events. The output text is then processed to obtain the final extraction result and label sequence for performance verification.

[0143] In this embodiment, the text to be extracted is randomly sampled from the FewDocAE dataset. Following the steps described above, a complete thought chain is obtained. Then, the commercial large language model API service is called, and this thought chain is input to obtain the final answer, i.e., the output text.

[0144] h i =LLM(X i )

[0145] Where h i This refers to the final content generated by the large language model for this data; LLM stands for Large Language Model; X i The thought process behind this data;

[0146] Because the output of a large language model is uncontrollable, it is necessary to control the final generated content h. i Extract the dictionary table using regular expressions. That is, predict the answer. Then, check the dictionary table. A series of processes are performed: The dictionary entries containing "not mentioned" are cleared, meaning there is no corresponding content for that argument type in the text; based on the dictionary table... The key and value, i.e., the argument type And the original text of the prediction A label sequence is generated; this label sequence is then processed into the same format as the dataset, i.e., using commas and quotation marks as delimiters, to obtain the final extracted label sequence.

[0147]

[0148]

[0149] Where σ3 is the regular expression for processing the generated content; σ4 is the tag generation function; This is a dictionary table extracted from content generated from a large language model. The argument type of the dictionary table; The predicted original text content for the dictionary table; It is a predicted label sequence;

[0150] Example 3

[0151] As described in this embodiment, a few-sample, chapter-level event extraction system based on a large language model's thought chain is used. Figure 3As shown, it specifically includes:

[0152] The data preprocessing module divides the data to be extracted into two parts: a support set and a query set, to adapt to tasks with few samples.

[0153] The extension module is used to expand the content of the thought process, including writing the task description and requirements for chapter-level event extraction; indexing event types, argument types and the definition of the event; and constructing positive examples and texts to be extracted.

[0154] The thought chain construction module uses the event type, argument type, event definition, positive example, text to be extracted, task description of chapter-level event extraction, and other supplementary text of a single piece of data to be extracted to design a thought chain for multiple rounds of questioning of a large language model.

[0155] The extraction module is used to input the complete thought chain into the large language model for multi-round question answering, and obtain the output text of the test text in the query set by extracting a small number of chapter events. The output text is then processed to obtain the final extraction result.

[0156] Example 4

[0157] To illustrate the effectiveness of this invention, it is compared with existing methods by conducting experiments on the large-scale public few-shot document-level event extraction dataset FewDocAE. This dataset is similar in setup to most few-shot learning datasets, consisting of 3-Way-1-Doc (i.e., one document containing three argument types), 3-Way-2-Doc (i.e., two documents containing three argument types), and 6-Way-2-Doc (i.e., two documents containing six argument types). Furthermore, since existing methods use traditional models, for ease of comparison, this embodiment converts the previously processed label sequences into matrix vectors consistent with traditional models for consistent performance evaluation experiments.

[0158] This embodiment uses three basic metrics commonly used in event extraction: precision (P), recall (R), and F1 score. Precision is the proportion of correct results extracted out of the total number of results extracted, measuring the accuracy of the extraction. Recall is the proportion of correct results extracted out of the total number of correct results, measuring the comprehensiveness of the extraction. The F1 score is a weighted average of precision and recall, used for overall system performance evaluation; a higher F1 score indicates better model performance.

[0159] The experimental results are shown in Table 1:

[0160] Table 1 Event Extraction Results from the FewDocAE Dataset

[0161]

[0162] The experimental results show that, in the 3-Way-1-Doc task (i.e., a document containing three argument types), the experimental group's model achieved an average accuracy improvement of 212% and an average F1 score improvement of 60.9% compared to the best baseline model, ProtoNet-LongFormer; in the 3-Way-2-Doc task (i.e., two documents containing three argument types), the experimental group's model achieved an average accuracy improvement of 231.8% and an average F1 score improvement of 27.9% compared to the best baseline model, ProtoNet-LongFormer; and in the 6-Way-2-Doc task (i.e., two documents containing six argument types), the experimental group's model achieved an average accuracy improvement of 248.7% and an average F1 score improvement of 49.5% compared to the best baseline model, ProtoNet-LongFormer.

[0163] As can be seen, compared with traditional extraction models, this invention uses a large language model to generate extraction results and employs a thought chain method to enable the large language model to better understand the task during extraction, reducing the problem of unstable output format and thus improving the performance of few-sample event extraction. Furthermore, the model structure of this invention has low complexity, requires no training or special fine-tuning, and only needs a small number of samples to complete the extraction task.

[0164] In summary, this invention proposes a method and system for few-sample, text-level event extraction based on a large language model's thought chain. The method includes: processing the data to be extracted into a support set and a query set to adapt to few-sample tasks; writing a task description and requirements for text-level event extraction; constructing an event argument table and an event definition table, indexing event information to expand the thought chain content; constructing positive examples based on the support set of the data to be extracted, and constructing the text to be extracted based on the query set of the data to be extracted; designing a thought chain for multi-round questioning of the large language model using the event type, argument type, event definition, positive example, text to be extracted, the task description for text-level event extraction, and other supplementary text for a single piece of data to be extracted; inputting the complete thought chain into the large language model for multi-round question-and-answering to obtain the generated content of the large language model, and processing the generated content to obtain the final extraction result.

[0165] Compared to traditional few-shot document-level event extraction models, the method constructed in this invention effectively optimizes its problems such as low performance with few samples, complex model structure, high training difficulty, and poor performance across domains and datasets. This invention requires only a very small number of examples to complete the extraction task, and has a simple structure, is suitable for various datasets, and can be used for various few-shot document-level event extraction tasks.

[0166] Compared to traditional constructive prompt templates, the thought chain constructed in this invention not only includes input and output prompts, but also intermediate steps of reasoning, which improves the reasoning ability of large language models for complex tasks and effectively improves the accuracy and F1 score of discourse-level argument extraction under conditions of few samples.

[0167] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for extracting few-sample, text-level events based on the thought chain of a large language model, characterized in that, Includes the following steps: S1: To adapt to tasks with few samples, the data to be extracted is processed into two parts: a support set and a query set. S2: Write the task description and requirements for chapter-level event extraction, as the content of the thought chain; S3: Construct an event argument table and an event definition table based on the event types and argument types of all the data to be extracted. Use the event argument table to index the event type of a single data item, and then use the event definition table to index the definition of the event. Use the event type, argument type, and the definition of the event as the content of the thought chain. S4: Construct positive examples based on the support set of the data to be extracted, and construct the text to be extracted based on the query set of the data to be extracted. Use the positive examples and the text to be extracted as the content of the thought chain. S5: Using the event type, argument type, event definition, positive example, text to be extracted, task description of chapter-level event extraction, and other supplementary text of a single piece of data to be extracted, design a thought chain for asking multiple rounds of questions to a large language model. S6: Input the complete thought chain into the large language model for multiple rounds of question and answer, obtain the generated content of the large language model, process the generated content to obtain the final extraction result; Step S1 specifically includes: given a support set and query set ; Define the data set to be extracted , , , ,in Indicates support set Chinese document The The boundary of an event argument type, where Represents query set Chinese document The The boundary of an event argument type; yes The set of all event arguments in the set. yes The set of all event arguments in the text; Step S3 specifically includes: according to The event argument type of each data item By classifying the events, we can determine which argument type corresponds to each event type, and thus establish an event argument table. ; Then, based on the event argument table The event type and its corresponding argument type are input into the large language model, and the model is prompted to complete the definition of the event. This process yields the definition and description of each event type, resulting in the event definition table. ; Define the event argument table as Define the event definition table as follows ,in This represents the set of event types contained in the data to be extracted. , This represents the set of event argument types contained in the data to be extracted. , The set of definitions representing each event type contained in the data to be extracted. , It is the number of event types included; The M, given a single data point to be extracted. The support set containing this data The query set of this data and the event argument type of this data Defined as ; and support set Includes support for set text sequences Supports set tag sequences Query set Contains a query set text sequence Query set label sequence .

2. The method for extracting few-sample, chapter-level events based on the thought chain of a large language model as described in claim 1, characterized in that: Step S4 specifically includes: obtaining the data set to be extracted. Extract the support set from it and query set text sequence Tag sequence ; Text sequence Convert to natural text , label sequence Convert to the correct answer The natural text is then used to populate the prompt template, thus completing the construction of the thought chain: in, This represents a function for processing text sequences; This represents the tag sequence processing function; Define the correct example P in the thought chain ; in To provide a good support set of natural text, The correct answer is obtained from the processed support set; Given a single data point, the text to be extracted is , It is natural text obtained by processing a text sequence of a single query set.

3. The method for extracting few-sample, chapter-level events based on the thought chain of a large language model as described in claim 1, characterized in that: Step S5 specifically includes: determining the event type of the data based on the event argument table, and then indexing the definition and description of the event based on the event definition table; given a single piece of data to be extracted... Based on the type of the data event argument it contains. In the event argument table The event type of this data is indexed in the middle. Then, based on the indexed event type... Go to the event definition table Index out the definition and description of this event type. ; The complete thought process for designing a multi-round questioning approach for a large language model includes: Defining the thought process of a single data point ; in The event type of this data entry; The argument type for this data event; This defines and describes the event type. This is a correct example of the data set after processing. The text to be extracted from this data; It is the fixed natural text in the thought chain that provides prompts to the large language model, which includes: task requirement template, which provides task description and requirements for chapter-level event extraction; necessary supplementary text before providing data to the large language model; and supplementary requirements for the large language model. Step S6 specifically includes: inputting the previously constructed thought chain into the large language model for multi-round question and answer: in This is the final content generated by the large language model for this data. For large language models; The thought process behind this data; For the final generated content Extract the dictionary table using regular expressions. That is, predict the answer; then, according to the dictionary table The key and value, i.e., the argument type And the original text of the prediction This generates a predicted label sequence similar to the label sequence of the query set. In order to perform performance verification: in Regular expressions for processing generated content; A function to generate tags; This is a dictionary table extracted from content generated from a large language model. The argument type of the dictionary table; The predicted original text content for the dictionary table; It is a predicted label sequence.

4. A few-sample, text-level event extraction system based on a large language model's thought chain, characterized in that... For performing the method as described in any one of claims 1-3, comprising: The data preprocessing module divides the data to be extracted into two parts: a support set and a query set, to adapt to tasks with few samples. The extension module is used to expand the content of the thought process, including writing the task description and requirements for chapter-level event extraction; indexing event types, argument types and the definition of the event; and constructing positive examples and texts to be extracted. The thought chain construction module uses the event type, argument type, event definition, positive example, text to be extracted, task description of chapter-level event extraction, and other supplementary text of a single piece of data to be extracted to design a thought chain for multiple rounds of questioning of a large language model. The extraction module is used to input the complete thought chain into the large language model for multi-round question answering, and obtain the output text of the test text in the query set by extracting a small number of chapter events. The output text is then processed to obtain the final extraction result.

5. An electronic device, characterized in that: include: Processor and memory; The memory is used to store one or more program instructions; The processor is configured to run one or more program instructions to perform the steps of the few-sample document-level event extraction method based on the large language model thought chain as described in any one of claims 1-3.

Images