A method for multi-document script extraction enhanced by a large language model

Abstract

This invention discloses a multi-document script extraction method enhanced by a large language model, comprising the following steps: using a large language model, extracting individual scripts from each document through an iterative prompting strategy based on context learning; establishing a three-layer structure to reveal the interrelationships between scripts, transitioning from shallow single scripts to deep aggregated scripts, and introducing a two-stage hierarchical recognition method to stratify scripts or steps; and enhancing suboptimal steps by integrating additional information from other steps and the large language model through a two-stage iterative prompting strategy. This application is the first to identify key challenges and propose solutions to the multi-document script extraction problem; it combines a large language model, adopts an extraction-assembly approach, and provides a computationally more efficient alternative. The quality of the extracted scripts is higher than that of most large language models, and its performance is comparable to GPT-4.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, and in particular relates to a method for extracting multi-document scripts using a large language model. Background Technology

[0002] Script extraction and structured representation are crucial for human knowledge acquisition. People are accustomed to acquiring knowledge from documents that possess script-like characteristics. A script consists of a series of steps designed to achieve a specific goal; it is defined as "a standardized sequence of events describing typical human activities, such as video production." Therefore, the automatic extraction of scripts from unstructured documents has attracted widespread interest from researchers, for applications such as executable robotic systems and problem-oriented reasoning systems.

[0003] Existing techniques employ unsupervised broad coverage methods and Transformers for script extraction. Recently, large language models have also demonstrated high efficiency in identifying procedural steps with zero / few samples. However, these methods primarily focus on extracting scripts from single documents, known as single-document script extraction, such as... Figure 1 As shown in (a). This task setting may lead to information loss and thus fail to meet actual needs, specifically in two aspects: (1) A single document usually only details one method, which compresses the decision-making space. For example, Figure 1 The script described in (a) omits some other available strategies, such as "making a video" which could also be achieved through "making a short film". (2) The steps in the script may be vague or lack detail, thus reducing the feasibility of the entire process. For example... Figure 1 (a) Step 2 lacks sufficient details regarding "how to set up the device".

[0004] Previous work on scripts can be divided into two categories: (1) narrative scripts: descriptions of consecutive events (e.g., “traffic accidents involving collisions”); and (2) goal-oriented scripts: appropriate sequences of steps to achieve a goal, such as “going to see a doctor”, including steps such as “making an appointment” and “going to the hospital”. Our work focuses on learning the steps to achieve a given goal by analyzing multiple documents, and therefore belongs to goal-oriented script extraction.

[0005] Traditional script extraction methods can be broadly categorized into two types: unsupervised distribution methods and language model-based methods. In the former, the pioneering work of Chambers and Jurafsky provided an unsupervised ranking-based method for extracting broadly covered scripts from large text corpora. In the latter, Lyu et al. utilized Transformers models to extract target-oriented scripts, thereby improving script extraction performance. Recently, Yuan et al. demonstrated the effectiveness of extracting script knowledge from large language models, showcasing the potential of large language models in script extraction. Furthermore, recent research has proposed several target-oriented script datasets to improve the script knowledge extraction capabilities of language models. However, these methods and datasets primarily focus on single-document script extraction, which may lead to significant information loss. Summary of the Invention

[0006] To address these limitations, this application proposes a multi-document script extraction method. Specifically, given a definite target and multiple related documents, multi-document script extraction requires extracting all scripts relevant to the target and then enhancing and structuring them. Consider... Figure 1 (b) Given a specific goal, such as “making a video”, the task requires the model to analyze the complex correlations between multiple documents to generate potentially satisfactory scripts, such as “creating a documentary video” and “making a short film”, which are associated with complete steps.

[0007] Multi-document script extraction can effectively address the first problem mentioned earlier by generating multiple scripts. Furthermore, the steps within the scripts can potentially enhance or correct each other, thus addressing the second problem mentioned earlier. For example, through… Figure 1 Step (b) "Record and keep the image stable" can refine step 1.3 "Record video" into "Record a video with stable image". By using the extracted script, users can effectively understand the overall overview of the objective, thereby simplifying the learning process and enhancing knowledge acquisition efficiency. To explore the essence of the problem, this application conducts a comprehensive analysis of multi-document script extraction. The challenges of multi-document script extraction mainly include three points:

[0008] There may be some missing steps in the script, which may break the continuity of the script. How can we fully explore all the steps covered in the documentation?

[0009] The scripts can enhance and complement each other. How can we establish such complex relationships based on multiple documents?

[0010] How can we use the correct factual knowledge in some scripts to correct suboptimal steps in other scripts that contain inaccurate information?

[0011] To extract scripts from multiple documents and capture richer information and finer details more comprehensively, this application proposes a new framework based on the "extraction-assembly" concept and envisions a multi-document script extractor enhanced by a large language model, namely LEMSE. It first extracts scripts from each document, then determines their hierarchical structure and assembles them into aggregated scripts based on the hierarchy. LEMSE is tailored to this task with three carefully designed modules: (1) an iterative extraction module that uses an iterative hint strategy to extract scripts from each document, improving the quality of script extraction through multi-stage reasoning, which helps solve RQ1. (2) a hierarchy identification module that uses a two-stage similarity calculation method to construct a three-layer hierarchy, identifying how scripts enhance and complement each other, which helps solve RQ2. (3) a step enhancement module that enhances suboptimal steps in the scripts by integrating other steps and additional information from within the large language model, which helps solve RQ3. Furthermore, we develop another solution for limited computational resources, LEMSE-FT, by fine-tuning a smaller, dedicated model.

[0012] To enhance the completeness of multi-document script extraction tasks, this application developed a custom benchmark dataset, MDScript. It first merges multiple reference documents for the same objective to facilitate the identification of multiple scripts. MDScript contains 2,000 objectives and 15,493 documents containing high-quality scripts. Furthermore, due to the complex correlations between scripts, a three-level hierarchical structure was implemented to more clearly present these correlations. The hierarchy describes all scripts under an objective, their steps, and the potential relationships of supplementary details. Thus, MDScript is expected to serve as a test platform for evaluating multi-document script extraction models.

[0013] To achieve the above objectives, this application discloses a method for extracting multi-document scripts using a large-scale language model enhancement, comprising the following steps:

[0014] Receive multiple documents;

[0015] Using a large language model, a single script is extracted from each document through an iterative prompting strategy learned from context.

[0016] A three-layer structure is established to reveal the interrelationships between scripts, enabling the transition from shallow single scripts to deep aggregated scripts. A two-stage hierarchical identification method is introduced to classify scripts or steps into layers.

[0017] The suboptimal step is enhanced by integrating additional information from other steps and the large language model through a two-stage iterative suggestion strategy.

[0018] Output the aggregate script.

[0019] Preferably, the iterative prompting strategy for context learning includes four stages, in which the large language model is guided by specific prompts to validate and enhance its initial output, taking a single document as input and outputting a corresponding single script consisting of a series of consecutive steps;

[0020] The first stage is the initial extraction stage, in which task-specific prompts guide the Large Language Model (LLM) to extract scripts from a given document;

[0021] The goal of the second phase is to identify the missing steps and generate a comprehensive script;

[0022] The third stage aims to remove steps that do not appear in the document;

[0023] The fourth stage focuses on removing unnecessary details from the process.

[0024] Preferably, the three-layer structure includes:

[0025] Backbone layer: For a given goal, the backbone layer contains a series of scripts that explicitly describe how to achieve the goal, connecting the goal to these scripts;

[0026] Detail layer: The detail layer scripts provide detailed guidance for the steps in the backbone layer, connecting each script in this layer with the corresponding step;

[0027] Supplementary Layer: The supplementary layer consists of a series of steps that provide additional factual knowledge to validate and refine the suboptimal steps in the backbone and detail layers.

[0028] Preferably, the two-stage hierarchical recognition method achieves the hierarchical division of scripts or steps through a two-stage similarity score sim2;

[0029] First, the script for the backbone layer is identified using the following formula, which includes calculating a given target. and each script SIM2 scores between the two, with scores above a predefined threshold α, are classified as backbone layer scripts. express:

[0030]

[0031] Where i∈{1,2,…,|D|}, and D is a multi-document;

[0032] Subsequently, the following formula is used to identify the scripts in the detail layer: For step s0 in the backbone layer script, calculate s0 and each script The sim2 scores between them are then used to designate the script with the highest score as the detailed guide for s0, denoted as H2(s0):

[0033]

[0034] Where j∈{1,2,…,|D|};

[0035] Use the following formula to identify steps in the supplementary layer: For step s0 in the trunk layer or detail layer, calculate s0 and s0 for each step in the other script. k The sim2 scores between the steps are used as a complement to s0, denoted as H3(s0):

[0036] H3(s0) = argmax(sim2(s0, s k ))

[0037] in

[0038] Scripts are grouped according to their hierarchy. Scripts belonging to the backbone layer are linked to the target, scripts belonging to the detail layer are linked to the corresponding extension steps, and steps in the supplementary layer are used to enhance and refine existing steps.

[0039] Preferably, the two-stage similarity scoring includes:

[0040] Given a pair of texts a and b, encode them independently using an unsupervised BERT-base model, and calculate the first-stage similarity score sim1(a,b) based on their BERTScore to evaluate their semantic relevance.

[0041] Use a supervised BERT-based model Texts a and b are encoded and combined with a trainable weight matrix W to enable the model to recognize fine-grained features at different levels. The formula for calculating the similarity score in the second stage is as follows:

[0042]

[0043] Where oper(·) is an l-dimensional vector, along with the weight matrix. Multiplying them transforms them into a scalar, where λ is the weight coefficient of sim1, and W and λ are optimized through backpropagation;

[0044] Fine-tuning using labeled data Minimize the negative log-likelihood of the correct answer b in the candidate list c(a) ([c(a)=[b1,b2,…,b2)) n ]), where the log-likelihood is calculated as:

[0045]

[0046] The standard answer is when no element in c(a) matches a, where sim1(a,unmatch) is assigned the value min(sim1(a,b)). i )), [unmatch] is a marker.

[0047] Preferably, the suboptimal step is an extraction of a script containing ambiguous or incorrect steps. The suboptimal step is enhanced by integrating additional information from other steps and the large language model through a two-stage iterative prompting strategy, including: a first stage including locating the useful parts of the supplementary steps, and a second stage using the useful parts and common-sense knowledge from the large language model to refine and correct the suboptimal step.

[0048] Preferably, single-stage reasoning replaces the multi-stage reasoning of the large language model. During the tuning process, the training dataset is used as the output label. Then, the results of the first model and the second model are aligned with the label. The first model and the second model replace the iterative extraction and step enhancement modules, respectively. The first model takes a single document as input to generate a single script containing a series of steps. The second model takes suboptimal steps and supplementary steps as input and outputs the enhanced steps.

[0049] The beneficial effects of this application are as follows:

[0050] This application is the first to identify key challenges and propose solutions to the multi-document script extraction problem;

[0051] This application combines a large language model, proposes a new framework, adopts the idea of ​​"extraction-assembly", and provides a more computationally efficient alternative.

[0052] The quality of the extracted scripts in this application is higher than that of most large language models, and the performance is comparable to GPT-4. Attached Figure Description

[0053] Figure 1 Example scripts for extracting the target "video production". (a) shows the result of single-document script extraction, and (b) shows the result of multi-document script extraction. Underlined text represents parts enhanced by leveraging knowledge from other steps;

[0054] Figure 2 The overall structure of one embodiment of this application includes extracting a single script from each document (module 1), hierarchical identification (module 2), and step enhancement (module 3);

[0055] Figure 3 Iterative extraction process;

[0056] Figure 4 The overall architecture of one embodiment of this application. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0058] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0059] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0060] The technical solutions provided in this application involve technologies such as machine learning and natural language processing in artificial intelligence, which are specifically described and illustrated through the following embodiments.

[0061] Following the immense success of large language models, contextual learning has demonstrated significant potential in solving tasks using textual task instructions and a few examples. To improve the inference quality of large language models, this application devises an iterative prompting strategy tailored to them. This strategy aims to guide the large language model to extract more reliable scripts and refine them by applying the model's common-sense knowledge. Furthermore, when used for dataset construction, data samples generated by large language models sometimes outperform crowdsourced human-generated data in terms of realism and fluency. This provides a promising alternative to expensive, large-scale crowdsourcing for constructing large language model datasets. Inspired by this discovery, this application leverages contextual learning from large language models to better extract scripts and utilizes the large language model as a reliable crowdsourcing worker. This approach helps to scale script data into reusable datasets that can be used to train smaller models.

[0062] The definitions used in this application are as follows:

[0063] Single document: A single document d consists of a title t and content c, denoted as d = {t, c}.

[0064] Multiple Documents: A multiple document collection is a set of multiple single documents, denoted by D = {d1, d2, ..., dn}. |D|}express.

[0065] Objective: Defined as the expected result or goal that the user wants to achieve, such as "making a video".

[0066] Single script: A single script is defined as a list of steps extracted from a single document, denoted as... like Figure 1 As shown in (a).

[0067] Aggregate scripts: Aggregate scripts are defined as aggregates of single scripts, organized according to a specific hierarchical structure, and represented as follows: Where E represents their hierarchical structure, such as Figure 1 As shown in (b).

[0068] Task Definition: Given a target and its associated multiple documents, multi-document script extraction requires the model to generate an aggregate script about the target, represented as follows: Where M is the multi-document script extraction model.

[0069] The LEMSE framework proposed in this application adopts the concept of "extraction-assembly", such as Figure 2 As shown. It consists of three modules: (1) an iterative extraction module, which extracts individual scripts from each document using the iterative prompting strategy of the large language model; (2) a hierarchy recognition module, which establishes a three-layer structure, reveals the complex interrelationships between scripts, and introduces a two-stage hierarchy recognition method; and (3) a step enhancement module, which enhances suboptimal steps by integrating additional information from other steps and the large language model.

[0070] The limitations of large language models on input length and long text understanding prevent them from processing all documents simultaneously. To address this issue, this application first extracts a single script from each document through context learning tailored for large language models. Empirical research in this application reveals several shortcomings in the scripts extracted by InstructGPT, such as missing steps, unfaithful steps, or redundant details. To resolve these issues, this application develops an iterative hinting strategy that progressively evaluates and improves the completeness, reliability, and conciseness of the scripts through multi-stage reasoning.

[0071] Figure 3This demonstrates an example of a prompting strategy using contextual learning for iterative prompting. At each stage, the large language model is guided by specific prompts to validate and enhance its initial output. This process takes a single document as input and outputs a corresponding single script consisting of a series of sequential steps.

[0072] Table 1(I) outlines the initial extraction phase, where task-specific prompts guide the LLM to extract scripts from a given document. Ideally, this would involve two steps: "Buy a camera" and "Record video." However, only "Record video" is extracted in the initial extraction phase.

[0073] The goal of the second phase is to identify the missing steps and generate a more comprehensive script. As shown in Table 1(II), LLM identified steps such as "make a budget" and "purchase a camera, such as a DSLR...". We repeated this phase until the number of steps in the script no longer increased.

[0074] The third stage, as shown in Table 1(III), aims to remove steps that do not appear in the document. For example, LLM identifies a step that does not mention "making a budget" in the document, considers it unfaithful, and therefore removes it from the script.

[0075] Finally, the fourth stage, as shown in Table 1(IV), focuses on removing unnecessary details from the removal steps, including removing redundant examples such as "SLR, digital camera, etc.", thereby extracting the core elements of the script. Table 1 shows examples of iterative prompting strategies for single-script extraction using context learning. The text generated for each stage is in bold.

[0076]

[0077]

[0078] Hierarchical recognition module

[0079] The complex relationships between scripts present a significant challenge in determining how they reinforce and complement each other. To address this issue, this application establishes a three-tiered hierarchical structure for structuring scripts and steps. This hierarchy enables shallow single-script ( Figure 1 (a) Transitioning to deeper aggregation scripts ( Figure 1 (b) Furthermore, a two-stage similarity calculation method is introduced to identify script levels.

[0080] First, the three-layer hierarchical structure is defined:

[0081] Backbone Layer: For a given goal, the backbone layer contains a series of scripts that explicitly describe how to achieve the goal, linking the goal to these scripts. arrive Connect them together. The backbone layer scripts provide multiple methods to achieve the goal.

[0082] Detail Layer: The detail layer scripts provide in-depth, detailed guidance for the steps in the backbone layer. We associate each script in this layer with its corresponding step (s to...). Connect them together. These scripts provide detailed instructions for some of the broader steps.

[0083] Supplementary Layer: The supplementary layer consists of a series of steps that provide additional factual knowledge to validate and refine the suboptimal steps in the backbone and detail layers.

[0084] Hierarchical identification process: The hierarchical identification of scripts or steps is achieved through a two-stage similarity scoring sim2. First, the scripts of the backbone layer are identified using formula (1). This includes calculating a given target... and each script The Sim2 score is used to determine the relative scores. Scripts scoring above a predefined threshold α are classified as backbone scripts. express.

[0085]

[0086] Where i∈{1,2,…,|D|}.

[0087] Subsequently, formula (2) is used to identify the scripts in the detail layer. For step s0 in the backbone layer script, s0 and each script are calculated. The Sim2 scores are then calculated. The script with the highest score is then designated as the detail guide for s0, denoted as H2(s0).

[0088]

[0089] Where j∈{1,2,…,|D|}.

[0090] Furthermore, formula (3) is used to identify the steps in the supplementary layer. For step s0 in the backbone layer or detail layer, s0 is calculated along with each step s in the other script. k The sim2 score between them. The step with the highest score is taken as the complement of s0, denoted as H3(s0).

[0091] H3(s0) = argmax(sim2(s0, s k ))

[0092] in

[0093] Combine scripts according to their hierarchy. For example... Figure 1As shown in (b), scripts belonging to the backbone layer are linked to the target, while scripts belonging to the detail layer are linked to the corresponding extension steps. Steps in the supplementary layer are used to enhance and refine existing steps.

[0094] Cosine similarity cannot adequately capture the hierarchical classification of scripts or steps. Therefore, this application develops a two-stage similarity calculation method to navigate this complex search domain. Initially, given a pair of texts a and b, we encode them independently using an unsupervised BERT-based model. The first-stage similarity score, denoted as sim1(a,b), is determined by their BERTScore and evaluates their semantic relevance.

[0095] Subsequently, the second stage uses a supervised BERT-based model. Texts a and b are encoded and combined with a trainable weight matrix W. This enables the model to recognize fine-grained features at different levels. The second-stage similarity score calculation formula is as follows:

[0096]

[0097] Where oper(·) is an l-dimensional vector, along with the weight matrix. Multiplying transforms it into a scalar, where λ is the weighting coefficient of sim1. Both W and λ are optimized via backpropagation.

[0098] Fine-tuning using labeled data Minimize the negative log-likelihood of the correct answer b in the candidate list c(a) ([c(a)=[b1,b2,…,b2)) n ]), where the log-likelihood is calculated as:

[0099]

[0100] This application introduces a special marker [unmatch], which is considered the standard answer when no element in c(a) matches a, where sim1(a,unmatch) is assigned the value min(sim1(a,b)). i )).

[0101] It is worth noting that the extracted script may contain ambiguous or incorrect steps, referred to as suboptimal steps. To address these issues, suboptimal steps are enhanced by incorporating additional information from supplementary layer steps and the large language model. However, it is difficult to determine which parts of the supplementary steps are beneficial. To address this, this application designs a two-stage iterative prompting strategy. The first stage involves identifying the beneficial parts of the supplementary steps. The second stage uses the beneficial parts and common-sense knowledge from the large language model to refine and correct the suboptimal steps. Table 2 provides a detailed example of step enhancement. It first identifies "keeping continuous shots stable" as a beneficial part of the supplementary steps. Then, the suboptimal step "recording video" is enhanced to "recording video with stable footage," thereby improving the clarity and correctness of the step.

[0102] Table 2 shows examples of iterative prompting strategies for step enhancement through context learning. The bold text represents the generated text.

[0103]

[0104]

[0105] LEMSE incurs significant computational costs due to its iterative hinting strategy, which incrementally improves the output stage by stage. To optimize the efficiency of LEMSE without sacrificing its effectiveness, this application proposes LEMSE-FT, which utilizes a smaller, specialized model fine-tuned on MDScript to achieve superior output through single-stage inference.

[0106] Figure 4 This paper demonstrates how LEMSE-FT replaces the multi-stage inference of large language models with a more streamlined single-stage inference. During tuning, the MDScript training dataset is used as the output labels. The results of the specialized model are then aligned with the labels, achieving performance comparable to multi-stage inference. We developed two special models to replace the iterative extraction and step augmentation modules, respectively. The first model takes a single document as input to generate a single script containing a series of steps, thus replacing the iterative extraction module. The second model takes suboptimal and supplementary steps as input and outputs augmented steps, thus replacing the step augmentation module. This design significantly reduces the frequency of model calls and the computational resources required per call.

[0107] Existing datasets lack support for MDSE, and manual annotation of these datasets is extremely expensive and labor-intensive. Therefore, this application combines automatic extraction from LEMSE with manual annotation to develop the MDScript dataset, specifically tailored for multi-document script extraction tasks.

[0108] A new multi-document script dataset, MDScript, was constructed by combining LEMSE with manual annotation. Initially, 14,961 article titles were collected from the WikiHow dataset as seed targets, and relevant documents were retrieved for each target. For targets that produced more than 20 results, only the top 20 documents were selected as relevant documents. Each target and its relevant documents were then processed using LEMSE to generate an aggregation script. In total, this process generated aggregation scripts for 2,000 targets from 14,821 relevant documents.

[0109] 100 targets are randomly selected as the validation set. 20 targets as test set The remaining 1880 targets and their set scripts were allocated as the training set.

[0110] Analyze the dataset in terms of diversity and category composition.

[0111] Table 3 Statistics of the MDScript dataset and other script datasets

[0112] Dataset size Average number of documents Average number of characters Average number of steps Hierarchical structure proScript 6,414 1 532.33 5.45 × wikiHow 112,111 1 668.82 5.93 × CoScript 55,000 1 - 5.96 × MDScript 2,000 7.43 8,697.12 22.73 √

[0113] MDScript is compared with other script datasets based on crowdsourcing or large language models. Table 3 shows that: (1) Although MDScript is smaller than other datasets, this difference is due to the fact that other datasets use only one source document per objective, while MDScript contains an average of 7.43 source documents per objective. (2) The source documents in the MDScript dataset have a higher word count than other datasets, facilitating script extraction from multi-document sources and long texts. (3) MDScript exhibits a hierarchical structure, with each script containing more steps, enhancing the depth of narrative extraction.

[0114] To comprehensively evaluate the effectiveness of this application, it was compared with competitors in the MDSE task. This application uses InstructGPT and GPT-4 as baseline models for LEMSE, and T5 (encoder-decoder structured language model) and Llama2-7B (decoder-only language model) as fine-tuning models for LEMSE-FT. To reduce computational costs, Qlora was used for efficient fine-tuning of Llama2-7B.

[0115] Given the novelty of the MDSE task, there is currently no direct solution. Therefore, this application adapts several relevant studies into our task scenario, including ASER, CEE-IEA, and LM-GNN. Furthermore, this application proposes a retrieval-based approach as a lower bound for this task. In the ablation study, each module is removed or replaced with a relevant method, including OGF for iterative extraction, SBERT for hierarchical recognition, and EMINER for step enhancement.

[0116] This application uses ERouge as the evaluation metric. Specifically, for the reference text sequence E = [e1, e2, ... e... n The text sequence Z = [z1, z2, ... z] generated by the model. m (where e and z represent individual texts), the ERouge-1 (ER-1) score is calculated as follows:

[0117]

[0118] The overlap function measures the overlap between two texts. This application employs two overlap criteria: "strict" and "synonym matching." The former requires the generated text string to match the reference text exactly, while the latter allows for the use of synonyms in the text to relax the comparison criteria.

[0119] ER-1 evaluates the percentage of single text sequences generated by the model that are covered by reference text sequences, while ERougeL (ER-L) focuses on evaluating the matching between their longest common subsequences. ER-L can be represented as:

[0120]

[0121] In the formula, E′ and Z′ are subsequences of E and Z, respectively, and v is the length of the subsequence.

[0122] Script quality is evaluated at three granular levels. First, the ER-1 of the backbone script headings is calculated, denoted as ER-1(Objective), to assess the proportion of backbone scripts that are correctly classified. Next, the ER-L of steps within an individual script is calculated, denoted as ER-L(Script), to assess the accuracy and order of the steps. Finally, the ER-L of the detail scripts is calculated, denoted as ER-L(Step), which assesses the percentage of steps that are correctly linked to their respective detail scripts.

[0123] Table 4 Overall evaluation results (%).

[0124]

[0125]

[0126] Table 4 shows the overall evaluation results. The analysis shows that: (1) Both LEMSE-FT and LEMSE significantly outperform their competitors, demonstrating the effectiveness of this application in solving the MDSE task. (2) The performance of LEMSE-FT (T5-2.8B) is significantly lower than that of LEMSE-FT (Llama2-7B) because the former has a smaller parameter size and its maximum length (512 tokens) is insufficient to handle longer documents. (3) The quality of the scripts extracted by LEMSE-FT (Llama2-7B) is higher than that of most large language models, and its performance is comparable to GPT-4. This shows that smaller specialized models can outperform large language models when trained on appropriate datasets. (4) The ERouge score of synonym matching significantly exceeds that of strict matching, indicating that although the multi-document scripts extracted by LEMSE are semantically similar to manually annotated scripts, there are significant differences in the use of vocabulary.

[0127] The beneficial effects of this application are as follows:

[0128] This application aims to summarize the skeleton of multiple documents through scripts, proposing a novel large language model-enhanced extraction and assembly framework, LEMSE. Furthermore, a benchmark dataset called MDScript is constructed specifically for multi-document script extraction tasks. To address the limitation of computational resources, this application also proposes an alternative, LEMSE-FT. Extensive experiments validate the effectiveness of the proposed framework, and smaller models trained on MDScript even outperform large language models on certain metrics.

[0129] As used herein, the term "preferred" is meant as an example, illustration, or illustration. Any aspect or design described herein as "preferred" need not be construed as being more advantageous than other aspects or designs. Rather, the use of the term "preferred" is intended to present the concept in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusionary "or." That is, unless otherwise specified or clear from the context, "X uses A or B" naturally includes either of the permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0130] Furthermore, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components (e.g., elements, etc.), the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure performing the functions in the exemplary implementations of this disclosure shown herein. Moreover, although specific features of this disclosure have been disclosed with respect to only one of several implementations, such features may be combined with one or more features of other implementations that may be desirable and advantageous for a given or particular application. Furthermore, with regard to the use of the terms “comprising,” “having,” “containing,” or variations thereof in the Detailed Description or claims, such terms are intended to be included in a manner similar to the term “including.”

[0131] The functional units in this invention embodiment can be integrated into a processing module, or each unit can exist physically separately, or multiple units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. The aforementioned devices or systems can execute the storage methods in the corresponding method embodiments.

[0132] In summary, the above embodiments are one implementation of the present invention, but the implementation of the present invention is not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made that deviate from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A method for extracting multi-document scripts using a large-scale language model enhancement, characterized in that, Includes the following steps: Receive multiple documents; Using a large language model, a single script is extracted from each document through an iterative prompting strategy learned from context. A three-layer structure is established to reveal the interrelationships between scripts, enabling the transition from shallow single scripts to deep aggregated scripts. A two-stage hierarchical identification method is introduced to classify scripts or steps into layers. The suboptimal step is enhanced by integrating additional information from other steps and the large language model through a two-stage iterative suggestion strategy. Output the aggregation script; The iterative prompting strategy for context learning includes four stages. In each stage, the large language model is guided by specific prompts to validate and enhance its initial output, taking a single document as input and outputting a corresponding single script consisting of a series of consecutive steps. The first stage is the initial extraction stage, in which task-specific prompts guide the Large Language Model (LLM) to extract scripts from a given document; The goal of the second phase is to identify the missing steps and generate a comprehensive script; The third stage aims to remove steps that do not appear in the document; The fourth stage focuses on removing unnecessary details from the process; The three-layer structure includes: Backbone layer: For a given goal, the backbone layer contains a series of scripts that explicitly describe how to achieve the goal, connecting the goal to these scripts; Detail layer: The detail layer scripts provide detailed guidance for the steps in the backbone layer, connecting each script in this layer with the corresponding step; Supplementary Layer: The supplementary layer consists of a series of steps that provide additional factual knowledge to validate and refine the suboptimal steps in the backbone and detail layers.

2. The method for extracting multi-document scripts with large-scale language model enhancement according to claim 1, characterized in that, The two-stage hierarchical recognition method uses a two-stage similarity score. To achieve layering of scripts or steps; First, the script for the backbone layer is identified using the following formula, which includes calculating a given target. and each script Between Score, score higher than a predefined threshold It is classified as a backbone script, used express: in D stands for multiple documents; Subsequently, the following formula is used to identify the scripts in the detail layer: for the steps in the backbone layer script ,calculate and each script Between Score, and then designate the script with the highest score as Detailed guidelines, recorded as : in ; Use the following formula to identify steps in the supplementary layer: for steps in the trunk layer or detail layer. ,calculate Each step in other scripts Between The highest-scoring step is designated as the score. The supplement is denoted as : in , ; Scripts are grouped according to their hierarchy. Scripts belonging to the backbone layer are linked to the target, scripts belonging to the detail layer are linked to the corresponding extension steps, and steps in the supplementary layer are used to enhance and refine existing steps.

3. The method for extracting multi-document scripts with large-scale language model enhancement according to claim 2, characterized in that, The two-stage similarity scoring includes: Given a pair of texts and They were independently encoded using an unsupervised BERT-based model, and their first-stage similarity scores were calculated based on their BERT Score. , used to assess their semantic relevance; Use a supervised BERT-based model For text and Encode and combine with a trainable weight matrix. The second-stage similarity score calculation formula enables the model to recognize fine-grained features at different levels: in With one A dimensional vector, and a weight matrix Multiplying them transforms them into a scalar. for The weighting coefficients, and Optimization is achieved through backpropagation; Fine-tuning using labeled data To minimize the candidate list The correct answer negative log-likelihood The log-likelihood is calculated as follows: when No element matches At that time, it was the standard answer, in which The value assigned is [unmatch] is a marker.

4. The method for extracting multi-document scripts with large-scale language model enhancement according to claim 3, characterized in that, The suboptimal step is an extracted script containing ambiguous or incorrect steps. The suboptimal step is enhanced by integrating additional information from other steps and the large language model through a two-stage iterative prompting strategy, including: the first stage includes locating the useful parts of the supplementary steps, and the second stage uses the useful parts and common-sense knowledge from the large language model to refine and correct the suboptimal step.

5. The method for extracting multi-document scripts with large-scale language model enhancement according to claim 4, characterized in that, Replace the multi-stage reasoning of large language models with single-stage reasoning, and use the training dataset as the output labels during the tuning process; The results of the first and second models are then aligned with the labels; the first and second models replace the iterative extraction and step enhancement modules, respectively, with the first model taking a single document as input to generate a single script containing a series of steps; The second model takes the suboptimal step and the supplementary step as input and outputs the enhanced step.

Images