Autonomous workflow generation using vision language models

Abstract

In many embodiments of the invention, a method for directing autonomous interaction with a user interface includes defining an autonomous workflow as subtasks to accomplish an overall task on a user interface, the autonomous workflow defined for a set of input values that are substitutable and including an identification of a starting point, and where each of a plurality of the subtasks includes an identification of a large multimodal model (LMM) and a prompt for the identified LMM, loading the user interface, iterating, until indication of completion of the overall task, an action generation process that includes: capturing a screenshot, providing the captured screenshot and a corresponding prompt to the LMM identified to generate an instruction set to advance the workflow, executing the instruction set, logging whether the action was successful, determining whether the task is complete, and repeating the action generation process when the overall task is not complete.

Description

AUTONOMOUS WORKFLOW GENERATION USING VISION LANGUAGE MODELSFIELD OF THE INVENTION

[0001] The present invention relates generally to autonomous user interface interaction and more specifically to generating autonomous workflows using machine learning models.BACKGROUND OF THE INVENTION

[0002] Websites and applications often have common paths of interaction that users follow to complete a task or achieve an outcome. These can include logging in to a user account, logging in to a user account with multi-factor authentication (MFA), logging out, changing a user account password (password rotation), etc.

[0003] Interaction with a website or application typically can be done in one of several ways: 1) executing another application that accesses the website or application using an application programming interface (API), 2) manually by human interaction on a user interface, or 3) autonomously by an application executing a workflow to interact with the user interface in a way that simulates human interaction. Applications generally have public APIs, but are often limited in what can be done. More features can typically be accessed by manually navigating the webpage or application. Humans can do this intuitively, but at the expense of time and attention of a human user. Application automations that simulate the human interaction conserves human time and attention, but are complex to design and execute successfully.SUMMARY OF THE INVENTION

[0004] In many embodiments of the invention, a method for directing autonomous interaction with a user interface includes defining an autonomous workflow as a set of subtasks to accomplish a particular overall task with respect to a particular user interface by interacting with the user interface, the autonomous workflow being defined for a set of input values that are substitutable and including an identification of a starting point for the user interface, and where each of a plurality of the subtasks includes an identification of a large multimodal model (LMM) and a prompt for the identified LMM, loading the user interface using the identification of the starting point for the user interface, iterating, until indication of completion of the overall task is received, an action generation process that includes: capturing a screenshot of the loaded user interface, providing the captured screenshot and a corresponding prompt from a subtask of the autonomousworkflow to the LMM identified in the subtask to generate an instruction set that represents an action to perform on the loaded user interface to advance the workflow, executing the generated instruction set on the loaded user interface to accomplish the action, logging whether the action was successful, determining whether the overall task is complete, returning confirmation of completion when the overall task is complete, and repeating the action generation process on a next user interface when the overall task is not complete.

[0005] In additional embodiments of the invention, the identification of a starting point for the user interface is a webpage URL.

[0006] Further embodiments of the invention include interrupting and redirecting the workflow when a risk is detected.

[0007] In more embodiments of the invention, at least one subtask utilizes a template.

[0008] In still additional embodiments of the invention, the template contains placeholders for a username and a password to be entered at runtime.

[0009] In still further embodiments of the invention, at least one subtask utilizes retrieval augmented generation (RAG) to direct retrieval of a current state of the user interface for context of the LMM.

[0010] In still more embodiments of the invention, a prompt includes a screenshot image of the user interface, a format for an output of the LMM, and values to be included in the output depending on success of the subtask.

[0011] In additional embodiments of the invention, a prompt also includes past history of the workflow, labels within the screenshot image, and examples of input / output pairs.

[0012] In further embodiments of the invention, a prompt includes examples of thought-action pairs and utilizes an iterative loop to generate a thought-action pair that the LMM observes to inform a next thought-action pair.

[0013] In more embodiments of the invention, the LMM utilizes shelf-reflection to review its own plans and previous outputs to identify better alternatives.

[0014] In still additional embodiments of the invention, the LMM utilizes tree-of-thoughts to explore diverse possibilities.

[0015] In still further embodiments of the invention, the user interface is a website and the identification of the starting point is a URL for an initial webpage of the website.

[0016] In still more embodiments of the invention, capturing a screenshot of the loaded user interface includes identifying interactable elements within the loaded user interface.

[0017] In additional embodiments of the invention, capturing a screenshot of the loaded user interface includes generating a document object model (DOM) representation of the webpage.

[0018] In further embodiments of the invention, the prompt requests a response in JavaScript Object Notation (JSON) format.

[0019] In more embodiments of the invention, the generated instruction set indicates elements in the user interface to manipulate and what action to perform on the elements.

[0020] In still additional embodiments of the invention, the action generation process further includes capturing a post-action screenshot of the user interface and presenting the post-action screenshot to the LMM to verify the action was performed correctly.BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Fig. 1 illustrates a workflow generation system in accordance with an embodiment of the invention.

[0022] Fig. 2 illustrates a portion of an example workflow in accordance with an embodiment of the invention.

[0023] Fig. 3 illustrates a prompt and a response in accordance with an embodiment of the invention.

[0024] Fig. 4 illustrates a process for autonomous workflow generation in accordance with embodiments of the invention.

[0025] Figs. 5A-5E illustrate screenshots of webpages during a process for autonomous workflow generation in accordance with embodiments of the invention.DETAILED DISCLOSURE OF THE INVENTION

[0026] Turning now to the drawings, autonomous workflow generation using artificial intelligence, particularly vision-based large multimodal models (LMMs) or vision language models (VLMs), in accordance with various embodiments of the invention are disclosed. As discussed above, application automations that simulate human interaction with interfaces in computing environments can conserve human time and attention, but are complex to design and execute successfully. Computing environments can include, but are not limited to, websites / web browsers, applications, desktop, mobile, and cloud. Some software tools, such as Playwright andPuppeteer for websites and other tools for other types of interfaces, can automate manual interaction for purposes such as user interface testing. They can, for example, capture and manipulate Document Object Model (DOM) representations of a user interface at each stage of interaction. However, the workflow for these tools must still be designed to provide instructions for the path of interaction.

[0027] An automated workflow can be defined as a series of steps (subtasks) that are performed by a computer system to accomplish an overall task with respect to a website or other type of user interface (e.g., on a desktop or mobile application). Workflows can be written by a human or can be generated programmatically by application code. Human creation is an intensive process. Embodiments of the invention provide for autonomous workflow generation using machine learning, such as by methods that employ large multimodal models (LMM) to analyze user interfaces and generate data structures that define autonomous workflows. An LMM can help design the actions or subtasks of a workflow. In particular, vision-based LLMs or vision language models (VLMs) (e.g., Azure OpenAI GPT4 Vision), a type of Large Multimodal Models (LMMs), can be suitable as they can accept as input screenshots of a website (or other user interface) at stages during the interaction. Further embodiments provide an autonomous workflow agent that can execute a developed autonomous workflow based upon a data structure defining the autonomous workflow created using an appropriately configured LMM.

[0028] Some prior academic work on programmatically creating workflows for interacting with websites utilized LMMs. However, such attempts were addressed to different use cases and not particularly useful for security operations. Moreover, they typically had low success rates and therefore were of limited usefulness. Many embodiments of the invention take advantage of two insights that drastically improve accuracy over the prior work. First, it is recognized that creating a single workflow embodying an overall task is cumbersome to be solved as a whole in one iteration by an LMM and can give results that are nonfunctional or inaccurate. In several embodiments of the invention, an initial process is performed to divide the overall task into subtasks or steps. Each step can then be addressed progressively by giving a prompt to an LMM, where the prompt can include a screenshot and, in many instances, a prior history of the current workflow as context. Additional information can be included in a prompt for better results.

[0029] Second, a more rigorous prompt increases effectiveness of the LMM in producing a useful output data structure defining the steps of an autonomous workflow for interacting with a userinterface. Many embodiments of the invention utilize multi-shot prompting, where the prompt includes examples including an example prompt and the correct data structure that should be output in response to the example prompt. Multi-shot prompting is particularly useful when the output is desired to be structured in a specific way. In many embodiments of the invention, the output of an LMM is structured (for example in JSON format) as a data structure that describes a set of instructions that other software tools may execute to perform the website (or other user interface) interaction.

[0030] Autonomously creating workflows for website (or other user interface) interaction in accordance with embodiments of the invention can enable at least two features: 1) carrying out multi-step tasks that would be intuitive for a person but challenging for a conventional automation to discover the proper course “blind” (e.g., changing password of an account) and 2) providing additional functionality that the website does not already possess (e g., enabling Single Sign On (SSO) when the website does not have SSO option).

[0031] Systems and methods for workflow generation in accordance with various embodiments of the invention are described further below.Systems for Workflow Generation

[0032] A workflow generation system in accordance with an embodiment of the invention is illustrated in Fig. 1. The workflow generation system 10 includes a workflow generation server 12, a web server 14, and a user computer 18. The workflow generation server 12 may perform processes to autonomously create workflows such as those described further below. In the course of these processes, the workflow generation server 12 may communicate with web server 14 that serves pages of the website of interest. Aspects of the workflow generation processes may be configured or controlled by user computer 18. Although a workflow generation system is described here with respect to Fig. 1, one skilled in the art will recognize that any of a variety of workflow generation systems may be utilized in accordance with embodiments of the invention.Workflows

[0033] An automated workflow can be defined as a series of steps (subtasks) that are performed to accomplish an overall task with respect to a website or other type of user interface (e.g., on an application). Different types of workflows can include, but are not limited to, logging in to a useraccount, logging in to a user account with multi-factor authentication (MFA), logging out, changing a user account password (password rotation), etc. Workflows may be specific to a website or service, e.g., Facebook, Instagram, etc., but are designed to be repeatable, for example by other users or using other values (e.g., login credentials) on the same website or service.

[0034] Fig. 2 illustrates a portion of an example workflow in accordance with an embodiment of the invention. The workflow specifies the starting URL of the webpage and a set of information for each step of the workflow. The information can include pre and post action screenshots, a description of the step, values to use, elements within the screen, and locations of the elements.

[0035] In additional embodiments of the invention, workflows may invoke a supervisor mode that can detect risk. Carrying out workflows on some websites can encounter cyber vulnerabilities, e.g., making a typographic error that can result in a phishing website, being redirected to a malicious DNS, etc.). When such risks are detected, the workflow can be interrupted or can branch to inspect the code of the website. Similarly, a supervisor mode may detect correctness. If it determines the workflow is proceeding to an inaccurate end, it can be interrupted. Prompts to an LMM to develop workflows are described next.Action Generation Prompts

[0036] As discussed above, embodiments of the invention can design an automated workflow with high accuracy by prompting an LMM for what action to take with a customized prompt at each step within the workflow. Each step may appear as a webpage with fillable or selectable content to interact with to proceed to the next webpage. A well-designed prompt helps the LMM understand how to interpret an image (screenshot of the webpage) and what to provide as an output. A more precise prompt can instruct the LMM to use information within an image to provide an answer and ensure the response is limited to what is in image. While VLMs and LMMs generally are mentioned here, other types of machine learning models (e.g., deep learning models, computer vision models, etc.) that are suitable for accepting images may be utilized in accordance with embodiments of the invention.

[0037] Processes for autonomously generating workflows in many embodiments of the invention, such as those described below can create or select from templates for tasks, where templates can assist with creating prompts. A template can contain logic to identify what subtasks are required for a workflow. The template can include some basic elements of prompts and / or identification ofan LMM for subtasks. A particular LMM may be more effective than others for a specific task. An example template for a change password task can list subtasks as 1) login, 2) navigate to account settings, 3) find password change page, 4) update password.

[0038] Templates may also be used in a modular fashion having subtemplates. For example, a first step subtemplate for a login subtask may state “What is the first step to login to...” and the next step subtemplate may state “what is the next step to login...” The contents of templates and subtemplates can be used to help form prompts that are effective for accomplishing subtasks and the overall task.

[0039] In several embodiments of the invention, prompts and templates can utilize techniques such as retrieval augmented generation (RAG). RAG is a technique for providing additional custom data when querying an LMM model, rather than relying solely on knowledge derived from the model’s training data. Relevant information can be retrieved and used as augmented context for input to the LMM. RAG allows a response to use newer data than the model was trained on, reduces hallucinations, provides domain-specific responses, and is efficient in not having to retrain a model.

[0040] As an example, a prompt often has better effectiveness when suppling the url of the website, along with the current screenshot to the LMM. Specifically, part of the template may read: "Given the attached screenshot showing the current state of the page from $URL, what is the next step to change the password? Use the labels or icons visible in the screenshot and website context to describe each action." The SURL variable may be supplied at runtime, along the with the text instructing the LMM to use both the screenshot and the website context.

[0041] Additionally, some embodiments of the invention can implement per-instantiation flexibility via input variables to increase effectiveness for certain websites or applications. Input variable can provide additional context by indicating additional conditions that are present at the time of execution.

[0042] For example, a template might read "The Task is to login to a website with username <USERNAME> and password <PASSWORD> $CONTEXT,. The previous step was to... " The USERNAME and PASSWORD text are placeholders whereas SCONTEXT is a runtime variable.

[0043] If the SCONTEXT variable is not supplied at runtime, the instantiated text reads "The Task is to login to a website with username <USERNAME> and password <PASSWORD>, . The previous step was to . .." If the SCONTEXT variable is populated with a value, e.g.,CONTEXT="If a MFA prompt is encountered respond with 123456", then the instantiated text reads: "The Task is to login to a website with username <USERNAME> and password <PASSWORD> If a MFA prompt is encountered respond with 123456,. The previous step was to II

[0044] Some embodiments of the invention can include automatically generating templates. For example, machine learning models such as LMMs or other computer programs can be used to generate templates using past history of which prompts were more successful than others at accomplishing their task.

[0045] A prompt in accordance with several embodiments of the invention can include: a screenshot image of the webpage at that step of the workflow; a format for the response; values to be included in the output when the tasks is complete and successful, when the task is complete and unsuccessful, and when the task is incomplete; and / or a request for a text description of the step, the label of the element in the screenshot to be used in the step, and / or an optional value of the action.

[0046] A prompt can include additional context for the step, such as directing the model to use past history of the workflow, information in the images such as labels, additional information about the website, the URL of the website / webpage, and / or examples.

[0047] Multi-shot prompting refers to a style of prompting that shows the model more examples of what it is requested to do. It often works better than zero-shot for more complex tasks where pattern replication is wanted, or when the output is desired to be structured in a specific way that is difficult to describe. Many embodiments of the invention utilize multi-shot prompting to request that an LMM generate an action. The prompt can include an input / output example. If the outputs have a similar format, then the LMM can derive the schema for the desired output.

[0048] Additional embodiments of the invention may utilize Chain-of-Thought (CoT) Prompting, ReAct (Reason-Act Framework), multi-shot prompting, Self-Reflection / Self-Critique, Retrieval Augmented Generation (RAG), Tree-of-Thoughts, Ensemble Learning, and / or Multi-Agent approaches.

[0049] ReAct (Reason-Act Framework) provides an iterative loop where the LMM explicitly generates a reasoning trace (thought) followed by an action, then observes the outcome to inform the next thought-action pair.

[0050] Chain-of-Thought (CoT) Prompting involves instructing the LMM to "think step-by-step" within its 'Thought' component (or before providing an answer / action), improving deliberation.

[0051] Self-Reflection / Self-Critique involves prompting the LMM to review its own plans, past actions, 'Thoughts', or outcomes to identify errors or better alternatives, often to refine its process.

[0052] Retrieval Augmented Generation (RAG) involves providing the LMM with access to external knowledge sources at inference time to ground its responses and 'Thoughts' in relevant, often up-to-date, factual information.

[0053] Tree-of-Thoughts (ToT) / Graph-of-Thoughts (GoT) involves exploring multiple reasoning paths or action sequences simultaneously (often as part of generating more robust 'Thoughts' or plans), evaluating them, and selecting the most promising. This can be a method for generating the diverse outputs considered by ensemble approaches.

[0054] Ensemble Learning combines multiple model instances, prompts, or outputs (potentially generated via methods like ToT) to improve robustness, accuracy, and error detection through evaluation, voting, or selection.

[0055] Multi-Agent Approaches embodies decomposing tasks among multiple specialized agents that collaborate or work in parallel, each potentially using a combination of the above techniques (ReAct, CoT, RAG, Self-Reflection, ToT) to fulfdl its role.

[0056] When using few-shot (or multi-shot) prompting with a ReAct framework that incorporates Chain-of-Thought (CoT), the prompt includes examples. Each example showcases an 'Observation' (acting as the input for that step), a 'Thought' component where step-by-step CoT reasoning is explicitly demonstrated, and the resulting 'Action.' This method is particularly useful for ensuring the LMM outputs are structured correctly (as Thought- Action pairs) and that the 'Thought' component itself embodies the desired detailed reasoning process. This reasoning and the resulting action plans can be further refined through Self-Reflection / Self-Critique, where the agent analyzes its own outputs or past performance to identify errors or better alternatives. Crucially, this reasoning can be significantly enhanced and grounded by Retrieval Augmented Generation (RAG), which supplies relevant, often up-to-date, external knowledge to inform the 'Thought' process and subsequent 'Action' selection, making the agent more knowledgeable and less prone to hallucination.

[0057] Furthermore, this RAG-enhanced, self-critiqued, clearly defined, and reasoned Thought- Action cycle, taught via few-shot examples, becomes a crucial building block for more advancedstrategies. These include ensemble learning (where multiple such reasoned outputs — generated via advanced methods like Tree-of-Thoughts (ToT) to explore diverse possibilities — are generated and evaluated) and multi-agent systems (where individual, specialized agents each operate on this robust, knowledge-augmented reasoning paradigm to perform their tasks effectively).

[0058] As a process for generating a workflow is performed, prompts may ask an LMM what conditions are present in order to determine which templates are applicable and / or which subtasks to implement. For example, an LMM may be given a screenshot and asked if an MFA prompt is shown. If it is, then subtasks appropriate to satisfying the MFA prompt are enacted. It may be necessary to access another authenticator system to retrieve a token and provide the token to pass the MFA prompt. The authenticator system may have a separate login.

[0059] Fig. 3 illustrates a prompt and a model’s response in accordance with an embodiment of the invention. The prompt requests a set of steps to complete a task where the response is in ISON format and provides additional guidance on the format and contents of the response. The response contains values indicating that the webpage captures entry of an email or username and the action is to fdl the entry.

[0060] Although characteristics of prompts for workflow generation are discussed above, one skilled in the art will recognize that any of a variety of techniques for prompting may be utilized in accordance with embodiments of the invention.Processes for Autonomously Generating Workflows

[0061] Processes for autonomously generating workflows in accordance with embodiments of the invention can incorporate screen capture, well developed LMM prompting for each step, and tools for performing website interaction given a structured instruction set as mentioned further above. Fig. 4 illustrates a process for autonomous workflow generation in accordance with an embodiment of the invention.

[0062] The process 400 includes generating (402) the initial structure of a workflow that represents an overall task. A workflow can be created by a user or by a monitor system. The workflow can include a workflow type (e.g., user login, user logout, password rotation, etc.) and an identification of a starting point for the user interface. In some embodiments, the user interface is a webpage and the identification of a starting point is a starting website URL (e.g., facebook.com). The process can then commence iterating an action generation process to create steps that make up theworkflow. The action generation process proceeds from prompting an LMM for an action to performing the action as follows.

[0063] The workflow generation system loads (404) the user interface, e.g., an initial webpage using the URL. In some embodiments of the invention, any popup windows can be dealt with by having a step where the system recognizes and dismisses the popup, or by letting the LMM handle the popup. Often it is more efficient to let the LMM handle the popup.

[0064] The workflow generation system also takes (404) a screenshot of the user interface, e.g., webpage that is loaded by the URL. Several embodiments of the invention can utilize tools such as, but not limited to, Playwright and Puppeteer to perform website interaction. In additional embodiments, other mechanism may be utilized to interact with different types of user interfaces, such as Microsoft UI Automation in .NET Framework and other accessibility tools designed for users with disabilities.

[0065] Some such tools can locate fillable, clickable, or otherwise interactable elements within the webpage. Optical character recognition can be used to visual pixel-by-pixel match of a region (button's visual template) within a larger image. Elements may be identified by an identifier, e.g., accessibility Id’s in Windows Desktop. Image location and position ca be determined within known locations in the screenshot or image.

[0066] The tool(s) may further generate a document object model (DOM) representation of the webpage based upon assets downloaded based upon a website URL such as (but not limited to) HTML documents, javascript files, and / or CSSs. In a number of embodiments, a web rendering engine can be utilized to construct a DOM from website assets downloaded using a website URL.

[0067] The workflow generation system provides the screenshot and an initial prompt to the selected LMM to generate an instruction set that represents an action to take on that webpage to advance the workflow.

[0068] The workflow generation system determines (406) a multi-shot prompting for an LMM to generate an interaction to attempt on the website and identifies an LMM to use. In some embodiments, a template can be identified that is appropriate to the workflow type and / or the current step within the workflow. The template can include some basic elements of a prompt and / or identification of an LMM to use to generate the action. The prompt can be based on the specific workflow provided and can incorporate contextual information as discussed further above. Anexample of a prompt is shown in Fig. 3. In many embodiments of the invention, the prompt requests a response in JavaScript Object Notation (JSON) format.

[0069] Returning to Fig. 4, the prompt is provided (406) to the LMM. The LMM outputs an instruction set, which may be in JSON format.

[0070] The workflow generation system executes (408) the instruction set to interact with website. As mentioned above, web driver tools such as Playwright and Puppeteer can be used for the website interaction, e.g., using the JSON description of the instruction set. In many embodiments of the invention, the instruction set indicates the element(s) in the webpage to select and values that should be used to fill in fields. If values would contain sensitive information, rather than provide the values to the LMM, placeholders like <user name> can be used and replaced later by the system when performing the actual interaction.

[0071] Whether the performed action was a success or failure can be logged (410) by the system.

[0072] If the browser proceeds to a next webpage that requires an interaction, the process can repeat from the screen capture step.

[0073] Further embodiments of the invention can validate individual steps of the workflow. In some embodiments, a post-action screenshot can be taken and presented to the LMM to verify that the designed action was performed correctly. Completion of the workflow may be recognized by a webpage that indicates finality or when criteria for completion is defined in the original workflow, in the template or by the LMM. For example, a user login workflow may end when the screen shows a home page or an account settings page. A password rotation workflow may end when text on the webpage describes a completion or confirmation that the password was changed.

[0074] Upon completion of the workflow, additional measures may be performed to verify that the task was accomplished. For example, a portion of the website can be entered that requires account login or the changed password is tested on a login.

[0075] Further embodiments of the invention can output and store the final developed workflow that includes steps that were successful. Other client devices and / or users may use the workflow to automate their own interaction with the website, and may substitute their own values (e.g., credentials) at individual steps.

[0076] Example screen shots of a login workflow in accordance with an embodiment of the invention are shown in Figs. 5A-5E. An initial login screen of a website is shown in Fig. 5A and may appear as loaded by the URL in a workflow. As the system obtains actions from an LMM, itcan proceed to the screen shown in Fig. 5B (filling in an email credential), Fig. 5C (showing a password field), Fig. 5D (filling in a password credential), and Fig. 5E (login completion to home page).

[0077] Although a specific process is discussed above with respect to Fig. 4 and 5A-5E, one skilled in the art will recognize that any of a variety of processes may be utilized in accordance with embodiments of the invention. Processes may be used to interact with applications or other software rather than webpages.

[0078] While the above description contains many specific embodiments of the invention, these should not be construed as limitations on the scope of the invention, but rather as an example of one embodiment thereof. For example, many of the examples described above involve screenshots of websites. However, systems and methods in accordance with various embodiments of the invention can be utilized to programmatically develop automatic workflows for any of a variety of graphical user interfaces and should be understood as not limited to applications solely involving graphical user interfaces rendered via websites. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

CLAIMSWhat is claimed is:

1. A method for directing autonomous interaction with a user interface, the method comprising: defining an autonomous workflow as a set of subtasks to accomplish a particular overall task with respect to a particular user interface by interacting with the user interface, the autonomous workflow being defined for a set of input values that are substitutable and including an identification of a starting point for the user interface, and where each of a plurality of the subtasks includes an identification of a large multimodal model (LMM) and a prompt for the identified LMM; loading the user interface using the identification of the starting point for the user interface; iterating, until indication of completion of the overall task is received, an action generation process that includes: capturing a screenshot of the loaded user interface; providing the captured screenshot and a corresponding prompt from a subtask of the autonomous workflow to the LMM identified in the subtask to generate an instruction set that represents an action to perform on the loaded user interface to advance the workflow; executing the generated instruction set on the loaded user interface to accomplish the action; logging whether the action was successful; determining whether the overall task is complete; returning confirmation of completion when the overall task is complete; and repeating the action generation process on a next user interface when the overall task is not complete.

2. The method of claim 1, where the identification of a starting point for the user interface is a webpage URL.

3. The method of claim 1, further comprising interrupting and redirecting the workflow whena risk is detected.

4. The method of claim 1, wherein at least one subtask utilizes a template.

5. The method of claim 4, wherein the template contains placeholders for a username and a password to be entered at runtime.

6. The method of claim 1, wherein at least one subtask utilizes retrieval augmented generation (RAG) to direct retrieval of a current state of the user interface for context of the LMM.

7. The method of claim 1, wherein a prompt includes a screenshot image of the user interface, a format for an output of the LMM, and values to be included in the output depending on success of the subtask.

8. The method of claim 7, wherein a prompt also includes past history of the workflow, labels within the screenshot image, and examples of input / output pairs.

9. The method of claim 1, wherein a prompt includes examples of thought-action pairs and utilizes an iterative loop to generate a thought-action pair that the LMM observes to inform a next thought-action pair.

10. The method of claim 1, wherein the LMM utilizes shelf-reflection to review its own plans and previous outputs to identify better alternatives.

11. The method of claim 1, wherein the LMM utilizes tree-of-thoughts to explore diverse possibilities.

12. The method of claim 1, wherein the user interface is a website and the identification of the starting point is a URL for an initial webpage of the website.

13. The method of claim 1, wherein capturing a screenshot of the loaded user interface includesidentifying interactable elements within the loaded user interface.

14. The method of claim 1, wherein capturing a screenshot of the loaded user interface includes generating a document object model (DOM) representation of the webpage.

15. The method of claim 1, wherein the prompt requests a response in JavaScript Object Notation (JSON) format.

16. The method of claim 1, wherein the generated instruction set indicates elements in the user interface to manipulate and what action to perform on the elements.

17. The method of claim 1, wherein the action generation process further includes capturing a post-action screenshot of the user interface and presenting the post-action screenshot to the LMM to verify the action was performed correctly.

Images