Computer implementation methods, computer programs, and computer systems (adaptive prompt correction based on LLM reflection in multi-stage workflows)

Abstract

Currently, prompt refinement using LLM introspection is limited in both the method and the scope of refinement. [Solution] While processing a prompt through the model, errors in the model's output are identified. The errors are converted into a score. The model is triggered to convert the prompt into an argument structure. Using the score and arguments, the model is made to identify the portion of the argument that should be replaced. By replacing that portion with a replacement portion, the model generates a replacement portion corresponding to that portion, so that the revised prompt results in a lower revised score. The model is used to replace the portion within the argument with the replacement portion to form a revised argument, and the model and the revised argument are used to generate a revised prompt.

Description

Technical Field

[0001] The present invention generally relates to the field of artificial intelligence using generative models such as large language models, automated machine learning, prompt engineering, and data science. More particularly, the present invention relates to a method, system, and computer program for adaptive prompt correction based on LLM introspection in a multi-stage workflow.

Background Art

[0002] Artificial intelligence (AI) technology has advanced significantly in the past few years. Recent AI systems are achieving human-level performance in cognitive tasks such as speech-to-text conversion, object and image recognition, and translation between different languages. This evolution is expected to lead to new and improved applications in many industries.

[0003] A large language model (LLM, multiple LLMs, model, multiple models) is a type of software designed to understand and generate text like a human. LLMs are trained on vast amounts of data from books, articles, websites, and other document sources. Fundamentally, LLMs use neural networks in a transformer architecture with layers of interconnected nodes that process and interpret text data. An artificial neural network (ANN) is a computing system composed of a large number of simple, highly interconnected processing elements that process information through their dynamic state responses to external inputs. An ANN is a processing device (algorithm and / or hardware) that roughly models the neuronal structure of the mammalian cerebral cortex on a smaller scale. A large-scale ANN implementation of an LLM can have tens of millions of interconnected nodes. In comparison, the mammalian brain has billions of neurons, and correspondingly, the magnitude of their overall interactions and emergent behavior increases.

[0004] A multi-stage workflow refers to a process in which a complex task is broken down into multiple stages or phases, with each stage focusing on a different aspect of the problem. Each stage may involve different processing techniques, models, or prompts to achieve more accurate or precise results than a single-stage approach.

[0005] Multi-stage workflows are particularly useful for tasks that require multiple layers of reasoning, understanding, or transformation. In LLM, such workflows help improve the quality of output, making it more reliable for tasks such as question answering, document summarization, code generation, and many other tasks.

[0006] A multi-stage workflow involves executing several stages sequentially, where the output of one stage becomes the input for the next stage, with or without a separate prompt. A multi-stage workflow may include intermediate inference or extraction of useful information before generating the final result.

[0007] LLM reflection refers to the process by which large-scale language models engage in self-evaluation or self-correction. This reflective capability allows the model to analyze its own generated responses, infer potential errors or inconsistencies, and systematically improve its output.

[0008] In LLM, reflection can be seen as a higher-order reasoning process that allows the model to improve the quality of its responses, reduce errors, and increase their alignment with user intent. Essentially, the reflection process allows the model to "think" more deeply about what it is producing, much like how humans reflect on their own thoughts or actions. Using reflection, the LLM can generate a response to a prompt and then reanalyze or re-evaluate it by examining that response for clarity, accuracy, or consistency. Based on this reflection, the model may revise its initial output. For example, if the LLM generates an inaccurate answer to a fact-based question, the model can reflect on its reasoning, compare the response to known facts, and provide a corrected answer.

[0009] The exemplary embodiment recognizes that prompt refinement using LLM reflection is currently limited in how refinement is performed and in the scope of refinement. For example, currently LLM reflection may involve iterative cycles of prompt refinement, where the model reinterprets the user's prompt or intent after the initial response. The model can then generate a more coherent response based on deeper reflection on what the user is asking.

[0010] Current LLM-based workflows often use multi-step prompts to form the workflow. The basic process of a multi-step or multi-stage workflow is that instead of having to write all prompts for all stages at once, prompt inputs are completed step by step, and then the results are combined with the next prompt as new inputs to the LLM until the final result is obtained. However, the exemplary embodiment recognizes that this method greatly increases the complexity of the prompt project. The exemplary embodiment recognizes that if the next prompt is generated or a prompt is modified in a multi-stage workflow, the impact of that generation or modification on the preceding and succeeding prompts must also be considered. The exemplary embodiment recognizes that this overall process is delicate, and a single move affects the whole. The exemplary embodiment recognizes that a slight modification in prompt-1 leads to a different result, and this different result has a greater impact on the generation of the next result, ultimately resulting in the LLM producing a completely different answer. This phenomenon is also known as the "butterfly effect" of the prompt chain.

[0011] The exemplary embodiments described herein provide a novel method of prompt tuning using LLM reflection. The novel method described herein is self-executing by LLM and adaptive to several preliminary prompts, which trigger automatic tuning of prompts at different stages while simultaneously tuning the parameter weights of the neural network until an optimal result is achieved. [Overview of the project] [Problems that the invention aims to solve]

[0012] Currently, prompt refinement using LLM introspection is limited in both the method and the scope of refinement. [Means for solving the problem]

[0013] Exemplary embodiments provide LLM reflection-based adaptive prompt correction in a multi-stage workflow. One embodiment includes a step of identifying an error in the output of the model in response to processing a prompt through the model. The embodiment further includes a step of converting the error into a score. The embodiment further includes a step of triggering the model to convert the prompt into an argument structure (argument). The embodiment further includes a step of using the score and the argument to cause the model to identify the portion of the argument to be replaced. The embodiment further includes a step of generating the replacement portion corresponding to the portion from the model, such that the revised prompt results in a lower revised score by replacing the portion with a replacement portion. The embodiment further includes a step of using the model to form a revised argument and replacing the portion within the argument with the replacement portion. The embodiment further includes a step of using the model to generate the revised prompt from the revised argument.

[0014] Other embodiments of this model include a corresponding computer system, apparatus, and computer program recorded on one or more computer storage devices, each configured to perform the actions of the embodiment.

[0015] One embodiment includes a computer-readable program product. The computer-readable program product includes a computer-readable storage medium and program instructions stored in the storage medium.

[0016] One embodiment includes a computer system, the computer system comprising a processor, computer-readable memory, and a computer-readable storage medium, and program instructions stored on the storage medium for execution by the processor via the memory.

Brief Description of the Drawings

[0017] The novel features considered characteristic of the present invention are set forth in the appended claims. However, the invention itself, and the preferred mode of use, further objects, and advantages thereof will be best understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings.

[0018] [Figure 1] A block diagram of a computing environment according to an illustrative embodiment is shown.

[0019] [Figure 2] A block diagram of a configuration for adaptive prompt correction based on LLM introspection in a multi-stage workflow according to an illustrative embodiment is shown.

[0020] [Figure 3] The configuration and operation of error detection and quantification according to an illustrative embodiment are shown.

[0021] [Figure 4A] One part of a five-paragraph argument form of a prompt tokenized for argument adjustment according to an illustrative embodiment is shown.

[0022] [Figure 4B] Another part of a five-paragraph argument form of a prompt tokenized for argument adjustment according to an illustrative embodiment is shown.

[0023] [Figure 4C] An exemplary part of a tokenized five-paragraph argument form of a prompt according to an illustrative embodiment is shown.

[0024] [Figure 5] The process of argument adjustment according to an illustrative embodiment is shown.

[0025] [Figure 6] This diagram illustrates an exemplary process for adaptive prompt correction based on LLM reflection in a multi-stage workflow, using an exemplary embodiment. [Modes for carrying out the invention]

[0026] One embodiment identifies an error in the output of the model in response to processing a prompt through the model; converts the error into a score; triggers the model to convert the prompt into an argument structure (argument); uses the score and the argument to cause the model to identify the portion of the argument to be replaced; generates the replacement portion from the model corresponding to the portion, by replacing the portion with the replacement portion, such that the revised prompt results in a lower revised score; uses the model to form a revised argument, replacing the portion within the argument with the replacement portion; and uses the model to generate the revised prompt from the revised argument. Thus, the embodiment autonomously revises the prompt.

[0027] Another embodiment further identifies a revised error in the revised output of the model in response to the revised prompt; determines that the revised score corresponding to the revised error is below a threshold; and creates the revised output as the final output. Thus, the embodiment autonomously improves the output of the model through automatic and autonomous prompt correction.

[0028] In another embodiment, the prompt, the revised prompt, the output, and the revised output relate to a stage in a multi-stage workflow, and the final output is the final output of the stage, which is (i) the input to the next stage, and (ii) the final output of the multi-stage workflow. Thus, the embodiment autonomously improves the output of the model through automatic and autonomous prompt correction in a multi-stage workflow.

[0029] In another embodiment, the argument has a set of parts, the set of parts including a background part, a target part, a constraint part, an input part, and an output part. Thus, the embodiment provides an efficient method for modifying the argument part by part in order to perform prompt revision.

[0030] Another embodiment further compares the output of the model with a benchmark output provided in a supervised mode of the model's operation as part of identifying the error, and the comparison results in the error. Thus, the embodiment autonomously improves the output of the model through automatic and autonomous prompt correction in supervised mode.

[0031] Another embodiment further compares the output of the model with the prompt requirements as part of identifying the error in an unsupervised mode of the model's operation, and the comparison results in the error. Thus, the embodiment autonomously improves the output of the model through automatic and autonomous prompt correction in an unsupervised mode.

[0032] Another embodiment further activates a model-based agent (agent); the agent causes the model to quantify the error into a score. Thus, the embodiment uses the model-based agent to evaluate the scoring error.

[0033] Another embodiment further causes the agent to cause the model to generate an inference of the score. Thus, the embodiment uses a model-based agent to evaluate errors for scoring and to provide an inference about the agent's score.

[0034] Another embodiment further activates a group of model-based agents (agents); configures each agent within the group with a different personality; causes each agent within the group to have the model quantify the error into a score corresponding to the agent; and outputs a group score for the error from the group. Thus, the embodiment evaluates the error for scoring and reaches a consensus among the agents on the score using a consensus-based architecture of the group of model-based agents.

[0035] In another embodiment, the personality of a particular agent within the group influences the manner in which that particular agent causes the model to quantify the error. Thus, the embodiment evaluates the error for scoring using a consensus-based architecture of a group of model-based agents, such that different agents score the error in different ways depending on the personality assigned to each agent.

[0036] Another embodiment further causes each agent in the group to cause the model to output a corresponding inference for the score corresponding to that agent. Thus, the embodiment provides inference about why the agent scored the error in that way.

[0037] Another embodiment further causes a first agent in the group, the first agent having a first score and a first inference for the error, to receive a second inference for the error corresponding to a second score corresponding to a second agent in the group; the first agent causes the model to perform a requantification of the error using the second inference, the requantification outputting a revised first score corresponding to the first agent, where the revised first score is closer to the second score than to the first score. Thus the embodiment provides a way for different agents in the group to consider the inferences of other agents and re-evaluate their own scores in consideration of the inferences of other agents.

[0038] Another embodiment further causes the second agent to receive the first inference about the error from the first agent; the second agent causes the model to perform a second requantification of the error using the first inference, the second requantification of which outputs a revised second score corresponding to the second agent, where the revised second score is closer to the revised first score than the separation between the first score and the second score. Thus the embodiment provides a way for each agent in the group to consider the inferences of all the other agents and to re-evaluate its own score taking into account the inferences of the other agents.

[0039] In another embodiment, the group score is the statistical mean of all scores corresponding to all agents within the group. Thus, the embodiment provides a method for obtaining a unified score from a group of agents.

[0040] Another embodiment further identifies a revised error in the revised output of the model in response to the revised prompt; determines that the revised score corresponding to the revised error does not fall below a threshold; and uses the revised score to create a second revised prompt for another iteration of prompt revision. Thus, the embodiment provides an iterative process of autonomous and incremental prompt correction.

[0041] In another embodiment, the iteration of prompt revision is performed for a prompt stage in a multi-stage workflow. Thus, the embodiment provides an autonomous and incremental iterative process of prompt correction in a multi-stage workflow.

[0042] An exemplary embodiment provides a method for adaptive prompt correction based on LLM reflection in a multi-stage workflow, in a manner similar to training a neural network. The exemplary embodiment processes the entire workflow as if emulating a multi-layer neural network, where the parameter layers of each layer of the neural network are considered prompts at a certain stage. Using network architectures such as neural network backpropagation, gradient descent optimization mechanisms, and fast networks, the exemplary embodiment automatically adjusts the prompts at different stages while simultaneously tuning the parameter weights of the neural network using several benchmark data samples until an optimal final output is achieved.

[0043] Using this method, one embodiment requires only a few preliminary prompts and some sample outputs at the start of a multi-stage task, with subsequent tasks being completed by an LLM based on a reflection mechanism. An exemplary embodiment allows the LLM to autonomously modify the prompts at each stage. Another embodiment allows a human user to participate in prompt adjustments if necessary.

[0044] An exemplary embodiment follows a specific set of steps, which can be summarized as follows:

[0045] 1. Error measurement mechanism based on introspection (This stage is similar to the concept of "loss" in conventional neural networks.)

[0046] 2. Token-based prompt argument splitting (prompt arguments can be considered similar to parameters in conventional neural networks.)

[0047] 3. Error-based prompt argument tuning (Prompt argument tuning can be considered similar to parameter tuning in conventional neural networks.)

[0048] In one embodiment of the error measurement process, the embodiment establishes an introspective error measurement mechanism in a manner similar to conventional neural network loss. The objective of the error measurement process is to find the error between the actual output value and the ideal or benchmark output value. The embodiment uses this error to form a quantitative indicator of the error and then performs backpropagation based on the quantitative indicator.

[0049] In one embodiment, the error measurement mechanism may be implemented as supervised operation. The supervised error measurement mechanism according to the embodiment is supplied with a "correct answer" (compatiblely also called a benchmark or ideal output value) as a reference. The embodiment measures the error between the actual output / answer and the correct output / answer.

[0050] In another embodiment, error measurement is implemented as unsupervised operation. The unsupervised measurement mechanism according to this embodiment does not have access to a "correct answer" or ideal output value as a reference criterion, but instead evaluates the error based on the degree of direct logical agreement and the degree of causality between the actual answer and the questions asked in the input prompt.

[0051] One embodiment sets up a scoring system for error measurement in order to quantify the error measurement. The scoring system can be set up in two scoring modes, namely supervised and unsupervised, which correspond to supervised and unsupervised error measurement mechanisms, respectively.

[0052] The score may be constructed on one or more bases of scoring, called scoring dimensions. For each scoring dimension, one embodiment initializes a group of several LLM agents that score those dimensions. The embodiment initializes the agent group with scoring as themes. The embodiment configures each agent with a different initial personality and scoring angle or logic.

[0053] The agent group is set up to perform several iterations of scoring and scoring adjustment. In one embodiment, the number of iterations to be performed is determined by the agent group's collaborative consultation over those iterations, and an adjusted score is output after each iteration.

[0054] In the first or initial iteration, each agent outputs an initial score corresponding to the error, based on its own LLM-based cognitive construct. Each agent also outputs an inference about its score output. In the remaining iterations, each agent reflects on and modifies its own score based on the scores of the other agents. In other words, the agent re-examines or re-evaluates its own score output and inference in consideration of the score outputs and inferences of the other agents in the group. This re-examination or re-evaluation results in the agent adjusting its own score and inference output. Each agent in the group goes through this reflection process, taking into account the score and inference output of the other agents in the group.

[0055] Each iteration generates a group score, which is a score calculated using a statistical method on the set of scores output by the individual agents within the group in that iteration. For example, in one embodiment, the group score may be the statistical mean of the individual agent scores. In another embodiment, the group score may be a weighted mean of the individual agent scores based on the weights associated with a particular agent. From this disclosure, a number of other methods for calculating a group score from individual agent scores can be conceived, and such methods are intended to be within the scope of the exemplary embodiments. In any given iteration, the group score of each agent group, the score outputs of the agents within the group, or both may change until the end of the iteration.

[0056] In one non-restrictive example, consider a group of three agents configured to score on a scoring dimension called "completeness" of the LLM output in response to a prompt. The three agents give scores of 10, 3, and 7, respectively, and their corresponding inferences in the first iteration. Each of the three agents receives the scores and inferences given by one or more other agents in the group, reshapes its own score based on its own cognitive construction (the cognitive perspective of the LLM-based agent based on the prompt), and gives a (revised) inference on the (revised) score. After several iterations, the scores created by the different agents in the group converge, but this does not necessarily mean they are the same score; rather, each score is within each other's acceptable limits. The group score, based on the converged individual agent scores, is output as the final score of error in the LLM output corresponding to the prompt.

[0057] One embodiment uses error scores to perform prompt argument adjustment as described herein. To perform prompt argument adjustment, one embodiment uses a self-reflection mechanism to dynamically and autonomously adjust prompts at each stage of the workflow.

[0058] The process of adjusting prompt arguments begins by identifying the set of “arguments” in the prompt. This process is analogous to backpropagation in a neural network, where the smallest units that can be modified and adjusted are the parameters within the neural network, which are organized together in the form of a weight matrix. However, to distinguish it from a neural network, in a prompt, the smallest units that can be modified and adjusted are tokens, as proposed by the exemplary embodiment, represented by the concept of “arguments.”

[0059] According to exemplary embodiments, an argument is a logical flow within a prompt, which logically forms a sequential relationship; that is, it follows a five-part structure called a “five-paragraph argument.” The five parts, or “paragraphs” of an “argument” as defined by exemplary embodiments, are “Background,” “Objective,” “Constraint,” “Input,” and “Output.” The term “paragraph” as used herein in the context of an “argument” is not to be confused with the conventional usage of the term paragraph in relation to the construction of language. As used herein, the term “paragraph” in the context of an “argument” is a part or portion of the argument structure, which acts to provide a particular type of information contained in the argument and may have any preferred length or linguistic structure. Paragraphs of an argument may, but do not necessarily, be identified by markers or delimiters, labels containing text or symbols, both markers and labels, or any other form that is neither a marker nor a label but indicates a different paragraph of the argument that the LLM is trained to understand.

[0060] When performing argument splitting of a prompt, one embodiment parses a given prompt and converts it into a five-paragraph argument format by designating one or more groups of tokens within the prompt as one of the five paragraphs of an argument. A prompt can have any number of tokens. Tokens can be all or part of a word, or all or part of a phrase containing an entire sentence. Tokens in a prompt can be organized into one or more arguments. An argument can have zero or more paragraphs of a particular type from the five types of paragraphs described above. A paragraph can have one or more tokens. Continuing the analogy with neural networks, each paragraph can be considered a matrix in a neural network. If a prompt needs to be modified, one embodiment modifies the prompt at a token-level granularity, but in units of arguments.

[0061] Next, one embodiment performs error-based prompt argument adjustment. Using a quantified error (from one embodiment that can be implemented as an LLM-based error generation module as described herein) and a token-based prompt argument split (from one embodiment that can be implemented as an argument construction module as described herein), the embodiment adjusts one or more arguments of the prompt as follows:

[0062] The embodiment causes the LLM to determine whether the prompt in the final prompt flow needs to be modified. If the prompt needs to be modified, the embodiment causes the LLM to determine which argument in the prompt should be modified. After narrowing the scope of the correction to the argument level, the embodiment causes the LLM to modify the argument and generate the modified prompt. Optionally, one embodiment may also cause the LLM in the modified prompt to indicate which aspect of the original prompt (or a previous version of the prompt if there have been multiple revisions) the embodiment is attempting to improve.

[0063] Prompt adjustments may be performed over several iterations before proceeding to the next stage in a multi-stage workflow. The iterative revisions can be configured to continue until the prompt in the prompt flow produces an error quantification below a threshold quantification (closing condition), at which point they stop. Each stage of prompting in the workflow follows this automated and autonomous LLM-based prompt revision process proposed by the exemplary embodiment. When the final revision of the final prompt in the multi-stage workflow satisfies the closing condition, the output created by the LLM is accepted as the output of the entire workflow, and the LLM reflection-based adaptive prompt correction process in the multi-stage workflow according to the exemplary embodiment is completed.

[0064] To address the problems associated with currently available prompt correction technologies and similar issues, the exemplary embodiment provides LLM-based adaptive prompt correction in a multi-stage workflow. The exemplary embodiment describes a method for an automated and autonomous LLM-based prompt revision or correction process.

[0065] For clarity of explanation, exemplary embodiments are described using several illustrative configurations without implying any limitation. From this disclosure, those skilled in the art will be able to recognize many changes, adaptations, and modifications of the described configurations to achieve the described objectives, which are intended within the scope of the exemplary embodiments.

[0066] Furthermore, simplified diagrams of the data processing environment are used in the figures and exemplary embodiments. In actual computing environments, additional structures or components not shown or described herein, or structures or components different from those shown but for similar functions to those described herein, may exist without departing from the scope of the exemplary embodiments.

[0067] Furthermore, the exemplary embodiments are described merely as examples with respect to specific actual or hypothetical components. Any particular mention of these and other similar artifacts is not intended to limit the invention. Any preferred mention of these and other similar artifacts can be selected within the scope of the exemplary embodiments.

[0068] The examples in this disclosure are for illustrative purposes only and are not intended to limit the scope to exemplary embodiments. Any advantages listed herein are illustrative only and are not intended to limit the scope to exemplary embodiments. Additional or different advantages may be realized by specific exemplary embodiments. Furthermore, specific exemplary embodiments may include some or all of the advantages listed above, or none of them.

[0069] Furthermore, exemplary embodiments may be implemented with respect to any type of data, data source, or access to a data source via a data network. Within the scope of the present invention, any type of data storage device may provide data to one embodiment of the present invention, either locally in a data processing system or via a data network. If one embodiment is described using a mobile device, any type of data storage device suitable for use with a mobile device may, within the scope of the exemplary embodiment, provide data to such embodiment, either locally in the mobile device or via a data network.

[0070] The exemplary embodiments are described using specific code, computer-readable storage media, high-level features, designs, architectures, protocols, layouts, diagrams, and tools merely as examples, and are not limited to the exemplary embodiments. Furthermore, for clarity of explanation, the exemplary embodiments are described, merely as examples, in several cases using specific software, tools, and data processing environments. The exemplary embodiments may be used in conjunction with other equivalent or similar structures, systems, applications, or architectures. For example, other equivalent mobile devices, structures, systems, applications, or architectures may be used in conjunction with such embodiments of the present invention within the scope of the present invention. The exemplary embodiments may be implemented in hardware, software, or a combination thereof.

[0071] The examples in this disclosure are for illustrative purposes only and are not limited to exemplary embodiments. Additional data, behaviors, actions, tasks, activities, and operations may be recognized in this disclosure and are contemplated within the scope of the exemplary embodiments.

[0072] Various aspects of this disclosure are described by computer system descriptions, flowcharts, block diagrams, and / or block diagrams of machine logic included in embodiments of computer program products (CPPs). With respect to any flowchart, depending on the technology involved, operations may be performed in a different order than those shown in a given flowchart. For example, in this case as well, depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated stage, simultaneously, or in a manner that at least partially overlaps in time.

[0073] Computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also called "mediums") that are collectively comprised of a set of one or more storage devices that collectively comprise machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. Computer-readable storage media may be, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any suitable combination of those described above. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (such as pits / lands formed on the main surface of a punch card or disk), or any suitable combination of those described above. When the term "computer-readable storage medium" is used in this disclosure, it shall not be interpreted as storage in the form of a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses passing through optical fiber cables, electrical signals communicated through wires, and / or other transmission media.As will be understood by those skilled in the art, data is normally moved at several intermittent points during the normal operation of a storage device, such as during access, defragmentation, or garbage collection; however, since data is not transient while it is stored, this does not mean that a storage device is transient.

[0074] Referring to Figure 1, this figure shows a block diagram of the computing environment 100. The computing environment 100 includes an example of an environment for executing at least some of the computer code involved in performing the method of the present invention, for example, an application 200 that runs in the computing environment 100 and can implement one or more embodiments for adaptive prompt correction based on LLM reflection in a multi-stage workflow as described herein. In addition to block 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end-user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including processing circuits 120 and cache 121), a communication fabric 111, volatile memory 112, persistent storage 113 (including an operating system 122 and block 200 as shown above), a peripheral device set 114 (including a user interface (UI) device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a host physical machine set 142, a virtual machine set 143, and a container set 144.

[0075] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device currently known or to be developed in the future that is capable of running programs, accessing networks, or querying databases such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, the execution of a computer implementation method may be distributed among multiple computers and / or multiple locations. On the other hand, in this presentation of computing environment 100, in order to keep the presentation as simple as possible, the detailed discussion focuses on a single computer, specifically computer 101. Computer 101 may be located in the cloud, although it is not shown in the cloud in Figure 1. On the other hand, computer 101 is not required to be located in the cloud, except to any extent that can be definitively shown.

[0076] The processor set 110 includes one or more computer processors of any type currently known or to be developed in the future. The processing circuitry 120 may be distributed across multiple packages, for example, multiple coordinated integrated circuit chips. The processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for high-speed access by threads or cores running on the processor set 110. The cache memory is typically organized into multiple levels, depending on its relative proximity to the processing circuitry. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 110 may operate with qubits and be designed to perform quantum computing.

[0077] Computer-readable program instructions typically cause the processor set 110 of computer 101 to execute a series of operational steps, thereby loading them onto computer 101 to implement a computer implementation method. As a result, the instructions thus executed instantiate the methods defined in the flowcharts and / or descriptions of the computer implementation methods contained herein (collectively referred to as the "Methods of the Invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as the cache 121 and other storage media considered below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the Methods of the Invention. In the computing environment 100, at least some of the instructions for executing the Methods of the Invention may be stored in blocks 200 in persistent storage 113.

[0078] The communication fabric 111 is a signal conduction path that enables various components of the computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive paths, such as buses, bridges, physical input / output ports, and similar switches and conductive paths. Other types of signal communication paths, such as optical fiber communication paths and / or wireless communication paths, may be used.

[0079] The volatile memory 112 is any type of volatile memory currently known or to be developed in the future. Examples include dynamic type random access memory (RAM) or static type RAM. Typically, the volatile memory 112 is characterized by random access, but this is not required unless explicitly stated. In computer 101, the volatile memory 112 is located in a single package and resides inside computer 101, but alternatively or additionally, the volatile memory may be distributed across multiple packages and / or located externally to computer 101.

[0080] The persistent storage 113 is any form of non-volatile storage for a computer, currently known or to be developed in the future. The non-volatility of this storage means that the stored data is maintained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage 113. The persistent storage 113 may be read-only memory (ROM), but typically at least a portion of the persistent storage allows for writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 may take several forms, such as various known proprietary operating systems or open-source portable operating system interface (CSI) type operating systems that employ a kernel. The code contained in block 200 typically includes at least a portion of computer code involved in performing the method of the present invention.

[0081] The peripheral device set 114 includes a set of peripheral devices for the computer 101. Data communication connections between the computer 101's peripheral devices and other components may be implemented in various ways, such as Bluetooth® connections, Near-Field Communication (NFC) connections, connections made by cables (e.g., Universal Serial Bus (USB) type cables), insert-type connections (e.g., Secure Digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the Internet. In various embodiments, the UI device set 123 may include components such as a display screen, speaker, microphone, wearable devices (such as goggles and smartwatches), keyboard, mouse, printer, touchpad, game controller, and haptic devices. Storage 124 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 is required to have a large amount of storage (for example, when computer 101 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another may be a motion detector.

[0082] The network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN)), the control and forwarding functions of the network module 115 are performed on physically separate devices, such that the control function manages multiple different network hardware devices. Computer-readable program instructions for performing the methods of the present invention can typically be downloaded from an external computer or external storage device to computer 101 via a network adapter card or network interface included in the network module 115.

[0083] WAN102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances by any currently known or future-developed technology for transmitting computer data. In some embodiments, WAN102 may be replaced and / or complemented by a local area network (LAN), such as a Wi-Fi network, designed to transmit data between devices located in a local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0084] An end-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 101) and can take any of the forms described above in relation to computer 101. Typically, EUD 103 receives useful and valuable data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to an end-user, these recommendations would typically be communicated from the computer 101's network module 115 to EUD 103 via WAN 102. In this way, EUD 103 can display or otherwise present the recommendations to the end-user. In some embodiments, EUD 103 may be a client device such as a thin client, heavy client, mainframe computer, desktop computer, and the like.

[0085] The remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. The remote server 104 may be controlled and used by the same entity that operates computer 101. The remote server 104 represents a machine that collects and stores useful and valuable data for use by other computers, such as computer 101. For example, in a hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from the remote database 130 of the remote server 104.

[0086] The public cloud 105 is any computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities, particularly data storage (cloud storage) and computing power, without direct active management by the user. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that make up the computers of the host physical machine set 142, which is the universe of physical computers available in and / or to the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. These VCEs may be stored as images and are understood to be transferable either as images or after instantiation of the VCE, among and between various physical machine hosts. The cloud orchestration module 141 manages the transfer and storage of images, deploys new instances of VCE, and manages active instances of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate over the WAN 102.

[0087] Here, some further explanation of virtualized computing environments (VCEs) is provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature in the kernel that allows for the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave as actual computers from the perspective of the programs running in them. Computer programs running on a normal operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and the devices allocated to the container; this feature is known as containerization.

[0088] The private cloud 106 is similar to the public cloud 105, except that its computing resources are available only for use by a single enterprise. While the private cloud 106 is shown as being in communication with the WAN 102, in other embodiments, the private cloud may be completely isolated from the internet and accessible only through a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. Each of the multiple clouds remains a separate discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple configuration clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0089] Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metric capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, reported, and billed, providing transparency to both service providers and consumers.

[0090] Referring to Figure 2, this figure shows a block diagram of a configuration for adaptive prompt correction based on LLM reflection in a multi-stage workflow, according to an exemplary embodiment. Application 202 may be implemented as Application 200 in Figure 1.

[0091] Application 202 operates with LLM 204 to implement a portion of one embodiment described herein and to provide the features described herein. Application 202 includes error metric module 206 and module 208. Error metric module 206 includes the error generation module described herein and implements an error quantification mechanism.

[0092] Prompt-1 210 is the first prompt in a multi-stage workload of prompts ranging from Prompt-1 to Prompt-n. A version of Prompt-1 is entered into the system as the first prompt to start the workflow. For each version of Prompt-1, including the initial version and iteratively revised versions, LLM204 produces Result 212.

[0093] In one embodiment, module 206 determines the error in an unsupervised manner using result 212 and prompt-1, as described herein. In another embodiment, module 206 determines the error in a supervised manner using result 212 and benchmark (B) 214, as described herein. Benchmark result B may or may not be available for each stage of the workflow. Within the scope of exemplary embodiments, any given workflow stage can operate in supervised or unsupervised mode, so that all stages may be supervised, all stages may be unsupervised, or some stages may be supervised and the rest unsupervised.

[0094] Prompt-1 is iteratively revised in the first stage in the manner described herein. Specifically, Module 208 works together with LLM 204 and Module 206 to construct the argument form of the latest version of Prompt-1 used (which may be the first prompt in the first iteration, or an earlier version of Prompt-1 in subsequent iterations). Module 208 works together with LLM 204, as described herein, to identify the arguments and select the paragraphs of the arguments to be modified. Module 208 works together with LLM 204, as described herein, to modify / revise / correct the identified arguments to form the revised or corrected Prompt-1.

[0095] If module 206 determines that the termination condition for the revision iteration has been met, the workflow proceeds to the next stage. The next stage of the workflow uses another prompt, shown here as prompt-2, and the latest result from the previous stage, shown here as result 212.

[0096] Subsequent stages use prompt-2 and result 212 as input to LLM204. Module 206 uses the result produced by LLM204 and benchmark B, if available in that stage, to determine and quantify the error in the result. Module 208, together with LLM204, uses error quantification to identify the arguments and paragraphs of prompt-2, revise prompt-2 into a new version, and send it to LLM for the new result in the next iteration.

[0097] The workflow iterates through this stage to revise the latest version of the prompt for this stage. As described herein, when the closing condition is met for this stage, the workflow proceeds to the next stage. In the final stage of the workflow (the nth stage), the result of the previous stage (the (n-1)th stage), shown here as result 214, and the prompt for the nth stage, shown here as prompt-n, are sent to LLM204 as input. The revision of prompt-n is iterated using the latest result 216 from the nth stage, and when the closing condition is met for that iteration, the configuration produces output 218, which is the final output of this configuration. Thus, output 218 is produced by the automatic and autonomous revision or correction of prompts in a multi-stage workflow according to one embodiment.

[0098] Referring to Figure 3, this figure shows the configuration and operation of error detection and quantification according to an exemplary embodiment. For illustrative and concise purposes only, and without implying any limitation thereto, we consider configuration 300-1 operating in the first stage of Figure 2. Thus, we consider input 302 to be the latest prompt-1 210 used by LLM204, and output 304 to be the latest result 212 produced from input 302. Component 306 determines the error (e) using input 302, output 304, and optionally benchmark B (not shown) if available.

[0099] Agents 308A, 308B, and 308C are three agents as non-restrictive examples developed for quantifying the error (e) determined by component 306. Assume that agent 308A scores the error e as 10 and provides an inference 310A of that score. Similarly, agent 308B scores the error e as 7 and provides an inference 310B of that score; and agent 308C scores the error e as 3 and provides an inference 310C of that score.

[0100] In the next iteration, the three agents receive each other's scores and inferences and reconsider their own scores and inferences. The three agents 308A-C output new scores and inferences 310A-C, respectively, for (n-1) iterations. In the (n-1)th iteration, shown as configuration 300-(n-1), agent 308A produces the (n-1)th score 9 and the corresponding (n-1)th inference 310A, agent 308B produces the (n-1)th score 8 and the corresponding (n-1)th inference 310B, and agent 308C produces the (n-1)th score 6 and the corresponding (n-1)th inference 310C.

[0101] The nth iteration of this agent group consultation is shown in configuration 300-n. In the nth iteration, a consensus is reached among agents 308A-C, each producing scores of 8.0, 7.8, and 8.2, respectively, which are considered to be within each other's tolerance (assuming an exemplary tolerance of 0.5). With a group score of 8, the error e is quantified and presented in module 208 in Figure 2, which uses this score for further operations as described herein. Finally, through prompt revisions, the overall error of the output is reduced, so that the group score falls below the threshold score by agent consensus. Once the group score falls below the threshold score, the stage output is considered correct and final for that stage, and further iterations of prompt revisions for that stage are terminated, and the workflow continues with output to the next stage.

[0102] Referring to Figure 4A, this figure shows one portion of a five-paragraph argument format of a prompt tokenized for argument adjustment, according to an exemplary embodiment. Figure 4A shows the first portion of argument 400.

[0103] Referring to Figure 4B, this figure shows another portion of the five-paragraph argument form of the prompts to be tokenized for argument adjustment, according to an exemplary embodiment. Figures 4A to 4B collectively form the five-paragraph argument, and Figure 4B shows the second portion of argument 400.

[0104] Referring to Figure 4C, this figure shows an exemplary portion of a prompt in tokenized five-paragraph argument format according to an exemplary embodiment.

[0105] In Figure 4A, argument 400 represents paragraphs 402, 404, and 406. Label 402A identifies paragraph 402 as a "purpose" type paragraph in the five-paragraph structure of argument 400. Label 404A identifies paragraph 404 as a "definition," i.e., a "background" type paragraph in the five-paragraph structure of argument 400. Multiple labels 406A form paragraph 406, which is a "constraint" type paragraph in the five-paragraph structure of argument 400.

[0106] Following Figure 4B, argument 400 shows paragraphs 408 and 410. Label 408A identifies paragraph 408 as an "input" type paragraph in the five-paragraph structure of argument 400. Label 410A identifies paragraph 410 as an "output" type paragraph in the five-paragraph structure of argument 400. In paragraph 408, the actual user query or input prompt may be represented in the form of a non-restrictive example of query 408B, as shown. As a non-restrictive example, single and double asterisks are optionally used as paragraph markers in argument 400 in several instances.

[0107] The tokenized form 500 of argument 400 is shown in Figure 4C. The various tokens identified in argument 400 and its paragraphs are shaded in different ways for the sole purpose of visual representation in the drawing, for example, tokens 502, 504, 506, 508, 510, 512, 514, and 516.

[0108] Referring to Figure 5, this figure illustrates the argument adjustment process according to an exemplary embodiment. Application 502 is an example of Application 202 in Figure 2. LLM 504 is an example of LLM 204 in Figure 2. Operation 506 proceeds substantially as described in Figure 2.

[0109] At any stage, the prompt must be revised or corrected. For example, in stage n, the latest version of prompt-n, as shown, is selected for revision. Application 502 works with LLM 504 to cause LLM 504 to organize prompt-n into a five-paragraph argument 520, namely background 522, objective 524, constraint 526, input 528, and output 530. Application 502 causes LLM 504 to identify the paragraphs to be modified and their tokenized forms. For example, as shown in this non-exclusive description, LLM 504 identifies the "background" paragraph 522 and the "objective" paragraph 524 for revision. LLM504 outputs a revised / new / replaced "background" paragraph 522A and a revised / new / replaced "purpose" paragraph 524A, and replaces paragraph 522 with paragraph 522A and paragraph 524 with paragraph 524A in argument 520. The revised argument 520 is then formed into a new version of prompt-n and entered into LLM for the next iteration of stage n of the workflow.

[0110] Referring to Figure 6, this figure shows a flowchart of an exemplary process for adaptive prompt correction based on LLM reflection in a multi-stage workflow, according to an exemplary embodiment. Process 600 can be implemented using application 502 in Figure 5.

[0111] Process 600 begins in the agent consensus setup (block 602) by calculating an error score using the stage output. The process determines whether the group score of the error is below a threshold (block 604). If the score is below the threshold (the "yes" path in block 604), the process proceeds to the next stage with the output (block 606). The process then terminates and may be similarly repeated in another stage.

[0112] If the score does not fall below the threshold (the "No" path in block 604), the process uses LLM to convert the prompt into a five-argument format (block 608). The process uses LLM to determine one or more arguments to be replaced (block 610). The process uses the replacement arguments in the five-argument format to reconstruct a new prompt (block 612). The process sends the new prompt through the LLM for that stage (block 614). The process creates a new output from LLM in response to the new prompt (block 616). The process returns to block 602 with the new output to repeat the automatic and autonomous prompt correction once more.

[0113] The following definitions and abbreviations are used for interpretation of the claims and specification. When used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” “contains,” or “containing,” or any other variations thereof, are intended to refer to non-exclusive inclusion. For example, a component, mixture, process, method, article, or apparatus containing a list of elements is not necessarily limited to those elements alone, and may include other elements not expressly listed or that are inherent to such component, mixture, process, method, article, or apparatus.

[0114] Additionally, the term “exemplary” is used herein to mean “serving as an example, case, or illustration.” Any embodiment or design described herein as “exemplary” should not necessarily be construed as being preferable or advantageous to other embodiments or designs. The terms “at least one” and “one or more” are understood to include any integer greater than or equal to 1, i.e., 1, 2, 3, 4, etc. The term “multiple” is understood to include any integer greater than or equal to 2, i.e., 2, 3, 4, 5, etc. The term “connection” may include indirect “connection” and direct “connection.”

[0115] References in this specification such as “one embodiment,” “a certain embodiment,” and “exemplary embodiment” indicate that the embodiments described may include certain features, structures, or characteristics, but not all embodiments may include or may not include such features, structures, or characteristics. Furthermore, such phrases do not necessarily refer to the same embodiment. Moreover, if certain features, structures, or characteristics are described in relation to one embodiment, it is considered within the knowledge of those skilled in the art that such features, structures, or characteristics may be affected in relation to other embodiments, whether or not they are explicitly described.

[0116] The terms “about,” “substantially,” and “approximately,” and their variations, are intended to include the degree of error associated with the measurement of a particular quantity based on the equipment available at the time of filing this application. For example, “about” may include a range of ±8%, 5%, or 2% of a given value.

[0117] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive or to limit oneself to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best describe the principles of the embodiments, their practical applications or technical improvements to the art found in the market, or to enable other persons skilled in the art to understand the embodiments described herein.

[0118] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive or to limit oneself to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best describe the principles, practical applications, or technical improvements to the technologies available on the market of the embodiments, or to enable other persons skilled in the art to understand the embodiments described herein.

[0119] Therefore, computer implementations, systems, or devices and computer program products are provided in exemplary embodiments for managing engagement with online communities and for other related features, functions, or operations. Where an embodiment or part thereof is described in relation to a certain type of device, the computer implementation, system, or device, computer program product, or part thereof is adapted or configured for use with a preferred and equivalent indication of that type of device.

[0120] Where an embodiment is described as being implemented in an application, the delivery of the application in a Software-as-a-Service (SaaS) model is intended to be within the scope of the exemplary embodiment. In a SaaS model, the ability of an application implementing an embodiment is provided to the user by running the application on a cloud infrastructure. The user can access the application using a variety of client devices through a thin client interface such as a web browser (e.g., web-based email) or other lightweight client applications. The user does not manage or control the underlying cloud infrastructure, including the network, servers, operating system, or storage of the cloud infrastructure. In some cases, the user may not need to manage or control the capabilities of the SaaS application. In some other cases, the SaaS implementation of the application may allow possible exceptions for limited user-specific application configuration settings.

[0121] Embodiments of the present invention may be supplied as part of a service engagement with a client corporation, non-profit organization, government agency, internal organizational structure, or similar entity. Aspects of these embodiments may include configuring a computer system for execution and deploying software, hardware, and web services that implement some or all of the methods described herein. Aspects of these embodiments may also include analyzing client behavior, generating recommendations in response to the analysis, building a system that implements a portion of such recommendations, integrating the system into existing processes and infrastructure, measuring system usage, allocating costs to system users, and billing for system usage. While each of the above embodiments of the present invention has been described by stating its own individual advantages, the present invention is not limited to any particular combination thereof. Conversely, such embodiments may be combined in any way and number according to the intended development of the present invention without losing their own advantageous effects.

Claims

1. A step of identifying errors in the output of the model in response to processing prompts through the model; The step of converting the aforementioned error into a score; A step in which the model is triggered to convert the prompt into an argument structure (argument); A step in which the model is made to identify the portion of the argument to be replaced, using the score and the argument; A step of generating the replacement portion corresponding to the portion from the model such that the revised prompt results in a lower revised score by replacing the portion with the replacement portion; The step of using the model to form a revised argument and replacing the portion of the argument with the replacement portion; and The step of generating the revised prompt from the revised argument using the aforementioned model. A computer implementation method comprising the above.

2. In response to the revised prompt, the step of identifying the revised error in the revised output of the model; The stage in which it is determined that the revised score corresponding to the aforementioned revised error falls below the threshold; and The stage of creating the revised output as the final output. The computer implementation method according to claim 1, further comprising:

3. The computer implementation method according to claim 2, wherein the prompt, the revised prompt, the output, and the revised output relate to stages in a multi-stage workflow, and the final output is the final output of the stage, and is (i) an input to the next stage, and (ii) one of the final outputs of the multi-stage workflow.

4. The computer implementation method according to any one of claims 1 to 3, wherein the argument has a set of parts, the set of parts including a background part, a target part, a constraint part, an input part, and an output part.

5. As part of the step of identifying the error, the step of comparing the output of the model with a benchmark output provided in a supervised mode of the operation of the model, the comparison step of which the error occurs A computer implementation method according to any one of claims 1 to 3, further comprising the above.

6. As part of the steps to identify the error in the unsupervised mode of the operation of the model, the steps include comparing the output of the model with the requirements of the prompt, the comparison step which results in the error, A computer implementation method according to any one of claims 1 to 3, further comprising the above.

7. The step of activating a model-based agent (agent); and The agent causes the model to quantify the error into the score. A computer implementation method according to any one of claims 1 to 3, further comprising the above.

8. The step of causing the agent to cause the model to generate inferences for the score. The computer implementation method according to claim 7, further comprising:

9. The stage of activating a group of model-based agents; The stage of configuring each agent within the aforementioned group with a different personality; A step in which each agent within the group is instructed to have the model quantify the error into a score corresponding to the agent; and Steps to output a group score for the error from the aforementioned group. A computer implementation method according to any one of claims 1 to 3, further comprising the above.

10. The computer implementation method according to claim 9, wherein the individuality of a particular agent within the group affects the manner in which the particular agent causes the model to quantify the error.

11. This step involves causing each agent within the group to further output corresponding inferences to the model for the score corresponding to that agent. The computer implementation method according to claim 9, further comprising:

12. A step of causing a first agent in the group, the first agent having a first score and a first inference for the error, to receive a second inference for the error corresponding to a second score corresponding to a second agent in the group; and The first agent causes the model to perform a requantification of the error using the second inference, the requantification outputting a revised first score corresponding to the first agent, wherein the revised first score is closer to the second score than the first score. The computer implementation method according to claim 11, further comprising:

13. The step of causing the second agent to receive the first inference about the error from the first agent; and The second agent causes the model to perform a second requantification of the error using the first inference, the second requantification outputting a revised second score corresponding to the second agent, wherein the revised second score is closer to the revised first score than the separation between the first score and the second score. The computer implementation method according to claim 12, further comprising:

14. The computer implementation method according to claim 9, wherein the group score is the statistical mean of all scores corresponding to all agents within the group.

15. In response to the revised prompt, the step of identifying the revised error in the revised output of the model; The stage in which it is determined that the revised score corresponding to the aforementioned revised error does not fall below the threshold; and For another iteration of prompt revision, the step is to create a second revised prompt using the revised score. The computer implementation method according to claim 1, further comprising:

16. The computer implementation method according to claim 15, wherein the iteration of prompt revision is performed for a prompt stage in a multi-stage workflow.

17. On the computer, A procedure for identifying errors in the output of the model in response to processing prompts through the model; Procedure for converting the aforementioned error into a score; A procedure for triggering the model to convert the aforementioned prompt into an argument structure (argument); A procedure for causing the model to identify the portion of the argument to be replaced, using the score and the argument; A procedure for generating the replacement portion corresponding to the portion from the model such that the revised prompt results in a lower revised score by replacing the portion with the replacement portion; A procedure for using the model to form a revised argument and replacing the portion within the argument with the replacement portion; and A procedure for generating the revised prompt from the revised argument using the aforementioned model. A computer program designed to execute something.

18. The computer program according to claim 17, wherein the computer program is stored in a computer-readable storage device in a data processing system, and the computer program is transferred from a remote data processing system via a network.

19. The computer program is stored in a computer-readable storage device in the server data processing system, and the computer program is downloaded for use in a computer-readable storage device associated with the remote data processing system in response to a network request to the remote data processing system. To the aforementioned computer, A procedure for measuring the use of the computer program associated with the request; and Procedure for generating an invoice based on the measured usage described above. A computer program according to claim 17 for further execution of the above.

20. A processor, one or more computer-readable storage media, and program instructions collectively stored in the one or more computer-readable storage media, wherein the program instructions are A procedure for identifying errors in the output of the model in response to processing prompts through the model; Procedure for converting the aforementioned error into a score; A procedure for triggering the model to convert the aforementioned prompt into an argument structure (argument); A procedure for causing the model to identify the portion of the argument to be replaced, using the score and the argument; A procedure for generating the replacement portion corresponding to the portion from the model such that the revised prompt results in a lower revised score by replacing the portion with the replacement portion; A procedure for using the model to form a revised argument and replacing the portion within the argument with the replacement portion; and A procedure for generating the revised prompt from the revised argument using the aforementioned model. A computer system in which the processor is capable of performing an operation including the operation of the processor.

Images