Agent model selection for individual workflow steps in agent-based automation, based on benchmarking and historical data.
The hyperautomation system optimizes agent model selection for workflow steps using benchmarking and historical data, enhancing automation efficiency and accuracy by integrating AI agents and RPA robots.
Patent Information
- Application Number
- JP2025250910
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2025-12-15
- Publication Date
- 2026-11-19
AI Technical Summary
Traditional automation methods fail to provide appropriate agent model selection for each individual workflow step, leading to inefficiencies and potential inaccuracies in agent-based automation.
A hyperautomation system that selects agent models based on benchmarking and historical data, using context grounding and self-benchmarking to optimize model selection for each workflow step, incorporating AI agents and RPA robots for dynamic and deterministic automation.
Improves workflow processing efficiency, reduces the need for user input, and prevents inaccuracies by selecting optimal AI models, resulting in a more intelligent and context-aware automation system.
Smart Images

Figure 2026182718000007 
Figure 2026182718000008 
Figure 2026182718000009
Abstract
Description
[Technical Field]
[0001] The present invention generally relates to agent-based automation, and more specifically to agent model selection for individual workflow steps based on benchmarking and historical data for agent-based automation. [Background technology]
[0002] Traditional automation methods fail to provide the appropriate agent model selection for each individual workflow step. Therefore, improved and / or alternative approaches may be beneficial. [Overview of the project]
[0003] In certain embodiments of the present invention, alternatives or solutions to technical problems and needs that are not yet sufficiently identified, recognized, or resolved by the current art, and / or useful alternatives thereto. For example, some embodiments of the present invention relate to agent-based automation.
[0004] A method for selecting agent models for a hyperautomation system that performs one or more agent-based automations is provided in one embodiment. The hyperautomation system includes memory for storing computer code for the one or more agent-based automations and computer code for selecting from a plurality of available models. The hyperautomation system includes at least one processor that runs the computer code to improve agent model selection. The method includes analyzing a workflow to identify each step of the workflow that requires a model as one or more model-required steps. The method includes automatically selecting one or more artificial intelligence models from the plurality of available models for each of the one or more model-required steps based on one or more model capabilities, one or more model performance, one or more model applications, or one or more benchmarking capabilities of the one or more artificial intelligence models. The method includes assigning the one or more artificial intelligence models to each of the one or more model-required steps in order to generate a multimodal agent-based automation for the workflow. The method includes running the multimodal agent-based automation to implement the workflow. [Brief explanation of the drawing]
[0005] To facilitate understanding of the advantages of specific embodiments of the present invention, a more specific description of the present invention, as briefly outlined above, will be made with reference to specific embodiments illustrated in the accompanying drawings. It should be understood that these drawings only illustrate typical embodiments of the present invention and should not be considered limiting its scope; however, the present invention is described and explained in more specific and detail with reference to the accompanying drawings. Herein, in the accompanying drawings,
[0006] Figure 1 is an architectural diagram showing a hyperautomation system configured to perform agent-based automation and orchestration in one or more embodiments.
[0007] Figure 2 illustrates some of the combined capabilities of artificial intelligence (AI) agents and robotic process automation (RPA) robots in one or more embodiments.
[0008] Figure 3 shows an AI agent, an RPA robot, an agent-based orchestration process (AOP), and a pool of applications according to one or more embodiments.
[0009] Figure 4 shows an exemplary AI agent service interface in one or more embodiments.
[0010] Figure 5 shows an exemplary AOP development interface in one or more embodiments.
[0011] Figure 6 shows an exemplary RPA development interface according to one or more embodiments.
[0012] Figure 7 shows one or more embodiments of an end-to-end AI agent, RPA robot, and AOP development and deployment system.
[0013] Figure 8 is an architectural diagram showing one or more embodiments of agent-based automation and RPA systems.
[0014] Figure 9 is an architecture diagram showing a deployed RPA system according to one or more embodiments.
[0015] Figure 10 is an architectural diagram showing the relationships between the designer, activity, and driver in one or more embodiments.
[0016] Figure 11 is an architectural diagram illustrating a computing system according to one or more embodiments.
[0017] Figure 12A illustrates an example of a neural network according to one or more embodiments.
[0018] Figure 12B illustrates an example of a neuron according to one or more embodiments.
[0019] Figure 13 is an architectural diagram illustrating a reference architecture for a generative AI model according to one or more embodiments.
[0020] Figure 14 is a flowchart illustrating a process for training an AI / ML model according to one or more embodiments.
[0021] Figure 15 is a flowchart illustrating a process according to one or more embodiments.
[0022] Unless otherwise specified, like reference numerals consistently indicate corresponding features throughout the accompanying drawings. [Detailed Description of the Invention]
[0023] Some embodiments relate to agent-based automation. More specifically, embodiments relate to agent model selection for individual workflow steps based on benchmarking and historical data for agent-based automation. Furthermore, embodiments relate to self-benchmarking and artificial intelligence (AI) center access for agent-based automation.
[0024] According to one or more embodiments, a hyperautomation system performs agent-based automation with agent model selection, which includes context grounding for multiple model capabilities and benchmarking, used to select a specific model for each step of the workflow that requires an AI model. The agent model selection of the hyperautomation system may include context grounding for selecting a specific model for each step of the workflow performed in the multimodal agent-based automation. Furthermore, the agent model selection of the hyperautomation system may include using benchmarking and historical job data in line with the context grounding to select a model for each step of the workflow. The agent model selection of the hyperautomation system may select the optimal model for each step that requires an AI model. Prior to the optimal model selection, the agent model selection of the hyperautomation system may perform self-benchmarking, access an artificial intelligence (AI) center, and run within a sandbox. Thus, the agent model selection provides improved control over the multimodal agent-based automation to achieve technical effects, benefits, and advantages such as improved workflow processing, increased efficiency of step execution, and prevention of hallucination due to incorrectly selected AI models. Therefore, the agent model selection in the hyperautomation system generates a more intelligent and context-aware hyperautomation system.
[0025] Figure 1 is an architectural diagram showing a hyperautomation system 100 configured to perform agent-based automation and orchestration in one or more embodiments. As used herein, “hyperautomation” refers to an automation system that brings together components of process automation, agent-based automation, integration tools, and technologies that amplify the ability to automate work. Examples of such components include, but are not limited to, artificial intelligence (AI) agents, agent-based orchestration processes (AOPs), and robotic process automation (RPA) robots.
[0026] Generally, as used herein, “AI agent” refers to AI-enhanced probabilistic automation that operates independently, dynamically, makes decisions, performs actions, and adaptively. In some examples, an AI agent operates in this manner by using a large-scale language model (LLM) or other AI model, which itself is also typically probabilistic in nature. According to one or more embodiments and as described herein, such one or more AI agents may implement context grounding techniques (e.g., search augmentation generation (RAG), extraction, and semantic memory), advanced agent-based search (semantic search and search retrieval, hybrid search, and broad search with re-ranking), tethering (including automatic tethering), and query decomposition.
[0027] Generally, as used herein, "AOP" refers to a dynamic yet predictable automation that combines probabilistic and deterministic methods. In some examples, AOP is automation that allows a user to describe an overall business process. The user is, but is not limited to, any person who has access to the system running the automation (e.g., a developer, engineer, customer, etc.). AOP can be created using an interface that enables the creation of business flowcharts described in Business Process Model Notation (BPMN), which is an Extensible Markup Language (XML) description of a business process (see Figure 5 for example).
[0028] Generally, as used herein, "RPA robot" refers to rule-based deterministic automation that operates predictably and makes deterministic decisions.
[0029] For example, in some embodiments, one or more RPAs may be used at the core of a hyperautomation system, and in certain embodiments, automation capabilities may be extended by AI / machine learning (ML), process mining, analytics, agent-based automation, and / or other advanced tools. For example, as the hyperautomation system learns processes, trains AI / ML models, and leverages analytics, more and more knowledge work may be automated, and computing systems within an organization, both those used by individuals and those operating autonomously, may be involved as participants in the hyperautomation process. Some embodiments of hyperautomation systems enable users and organizations to discover, understand, and extend automation efficiently and effectively.
[0030] In such embodiments, the AI agent "coexists" with the RPA robots and AOPs that perform the RPA. As described herein, the AI agent is an AI-skilled automation that can operate independently and dynamically make decisions, perform actions, and adapt to its performance. The AI agent can dynamically leverage the tools available through these RPA robots to perform tasks such as document processing (see, for example, U.S. Patent Application Publication No. 2021 / 0097274), user interface (UI) automation (see, for example, U.S. Patents No. 10,654,166, 10,990,876, 11,080,548, 11,507,259, 11,733,668, and 11,748,069), semantic copy and paste between source and target (see, for example, U.S. Patent No. 12,124,806 and U.S. Patent Application Publications No. 2023 / 0107316, 2023 / 0415338, and 2024 / 0220581). The AI agent can dynamically select these tools and execute them in a pipeline.
[0031] Generally, agent-based automation is probabilistic automation performed by one or more AI agents. Agent-based automation expands an organization's automation potential by focusing not only on individual tasks but on the entire end-to-end process. A team of RPA robots and / or AOPs directed by AI agents can enable a single employee to accomplish the work of many employees. Agent-based automation via AI agents and / or AOPs gives managers room for guidance, doctors more time to care for patients, developers the ability to fine-tune their work, engineers the freedom to innovate, and customers a seamless, personalized experience.
[0032] A variety of technical effects, benefits, and advantages can be achieved through agent-based automation. Agent-based automation improves memory usage by reducing the amount of storage required for data, and improves processor efficiency by reducing the number of calls and actions. Agent-based automation also potentially offers the ability to process gigabytes, terabytes, petabytes, or more of data that would be impossible with human implementation processes that are mental or manual. Agent-based automation also potentially enables the use of fewer triggers and models through dynamic decision-making. While traditional software automation alone may require 100 actions in an exemplary scenario, agent-based automation in the same exemplary scenario can drastically reduce the number of actions required (e.g., to 15 actions). Agent-based automation can also improve the efficiency of the LLM or AI model by using context grounding to tether the AI agent to the desired context, thereby "constraining" the LLM or AI model to the relevant context.
[0033] An AI agent may have agent memory that evolves to store user interactions, feedback, corrections, and solutions (e.g., dynamic and / or direct user input from human-in-the-loop actions). As used herein, human-in-the-loop actions or "human-in-the-loop" may include AI agents and RPA robots collaborating with users to receive such dynamic and / or direct user input. As the agent memory grows, the AI agent may become increasingly autonomous, reducing the need for dynamic and / or direct user input and improving efficiency. The AI agent may also learn to become more efficient based on the agent memory if more efficient solutions are contained in or derived from the agent memory. For example, the AI agent may periodically process the agent memory to analyze patterns in order to achieve greater autonomy.
[0034] As used herein, “agent memory” refers to a dynamic caching (i.e., memory) system for managing escalations and tool calls. For example, if an AI agent encounters a problem during execution, the AI agent may prompt or otherwise request user interaction or feedback regarding overcoming the problem, remember / cached such interaction or feedback, and learn from it to reduce the need for repetitive user input. According to one or more technical effects, benefits, and advantages, agent memory provides improved efficiency by remembering solutions to common problems and minimizing potentially costly tool calls. The collaborative operation between the AI agent and the agent memory potentially “bends the curve” as the AI agent continuously learns through the agent memory, requiring less and less user interaction.
[0035] Generally, agent-based orchestration is implemented by a conductor application that implements one or more AOPs and / or AI agents to orchestrate AI agents (e.g., UiPath Agents®), third-party agents, RPA robots (e.g., UiPath Robots®), AOPs, and users (e.g., where user interaction is required or requested) that execute a workflow. Agent-based orchestration enables agent-based automation, modeling, and monitoring of complex business processes from start to finish. Agent-based orchestration also offers a unique ability to orchestrate RPA robots, AI agents, AOPs, third-party agents, and users across the entire end-to-end workflow. Agent-based orchestration is beneficial for the successful scaling of agent-based automation.
[0036] AI agents for agent-based automation are AI model-based, as discussed herein, enabling them to operate independently of the user and implement these agent-based automations. AI agents are also goal-oriented, making probabilistic decisions using context. Furthermore, AI agents are well-suited for ad-hoc tasks requiring high adaptability. AI agents learn how to perform tasks and improve over time. AI agents can use and select a variety of tools (often through RPA robots used as tools by the AI agent) to accomplish tasks, gather context, and perform actions. In some embodiments, an AI agent can build workflows for RPA robots and / or other AI agents to execute and generate automations, for example, by leveraging UiPath Autopilot™ for developers or another application that helps developers accelerate the creation and testing of agent-based automations. For example, an AI agent can use a designer application via an API to generate another AI agent or RPA robot to execute a portion of a workflow, and can also trigger human-in-the-loop actions to escalate issues with the workflow. Where appropriate, the workflow can then be deployed by an AI agent. The AI agent may also have varying degrees of autonomy, managed by agent-based orchestration.
[0037] The AI agent generates a dynamic plan to achieve the objective for each instruction, using the provided tools and context, by executing an "agent loop." Once the dynamic plan is generated, the AI agent utilizes an efficient execution path for the dynamic plan. If the dynamic plan has two or more steps that can be executed in parallel, the AI agent executes those two or more steps in parallel, based on the available resources. After each step is completed, the AI agent retrieves the output from that step and regenerates the next step or multiple steps. Thus, the agent loop continues until the objective is achieved. Executing the steps of the dynamic plan in parallel, and using ecosystem tools and context grounding, are advanced capabilities of agent-based orchestration.
[0038] As described herein, RPA robots are rule-based automation that operates predictably and makes deterministic decisions. RPA robots are highly reliable and efficient and are suitable for routine tasks. RPA robots, along with AI agents, may use human-in-the-loop behavior for exception handling. According to one or more embodiments, AI agents are more flexible, more abstract, and more autonomous than RPA robots and AOP. Furthermore, RPA robots are more stable, more specific, and more manageable than AI agents and AOP. Moreover, AOP processes occupy an intermediate position between the flexibility / stability, abstraction / specificity, and autonomy / manageability properties of AI agents and RPA robots, respectively.
[0039] As will be further explained herein with respect to Figure 3, AI agents, AOPs, and RPA robots can discover and use each other as tools to accomplish tasks. AI agents, AOPs, and RPA robots can also access and use various applications (e.g., via Application Programming Interfaces (APIs)). Tools may be manually configured by developers for automation, and / or AI agents and RPA robots may discover and use tools at runtime.
[0040] In some embodiments, AI agents, AOPs, and RPA robots can collaborate with users (e.g., human-in-the-loop), enabling them to make faster, more consistent, and more informed decisions. Furthermore, the use of AI agents, AOPs, and RPA robots allows users to achieve more, as they can handle additional repetitive, monotonous, and ad-hoc tasks at a scale unmanageable by humans. Users can make decisions when the AI agent, AOP, or RPA robot encounters an exception. Thus, users can be elevated to and focus on roles as supervisors, decision-makers, and organizational leaders.
[0041] AI models provide AI agents with the ability to reason, plan, create, and make autonomous decisions. AI models can also be used by RPA robots for task-specific activities, such as document processing or data analysis. AI models can be enhanced with business-specific content and context (e.g., from enterprise-grade context repositories), which improves the accuracy and results of the AI models. AI models can be applied individually or simultaneously, depending on the complexity of the task. AI model selection can be done from the RPA vendor's model library, third-party models, and BYOM (Bring Your Own Model) options (see, for example, U.S. Patents 11,738,453 and 11,748,479).
[0042] The hyper-automation system 100 includes user computing systems such as a desktop computer 102, a tablet 104, and a smartphone 106. However, any desired user computing system, including but not limited to smartwatches, laptop computers, servers, and Internet of Things (IoT) devices, may be used without departing from the scope of this disclosure. Also, although three user computing systems are shown in Figure 1, any appropriate number of user computing systems may be used without departing from the scope of this disclosure. For example, in some embodiments, tens, hundreds, thousands, or millions of user computing systems may be used. The user computing systems may be used actively by a user or may be executed automatically by an AI agent, AOP, and / or RPA robot with much or no user input.
[0043] As disclosed herein, in some embodiments, there are three types of automation: (1) agent-based automation implemented by each AI agent, (2) RPA implemented by each RPA robot, and (3) composite automation achieved by a combination of AI agents and RPA robots to accomplish a more complex overall task. Automations 110, 112, and 114 may include, but are not limited to, those performed by RPA robots and / or AI agents, whether individually or to achieve a larger composite automation. Other processes, such as listeners, may also be implemented. These processes may be standalone applications, subprocesses of another application, parts of an operating system, any other suitable software and / or hardware, or any combination thereof, and will not deviate from the scope of this disclosure. In fact, in some embodiments, the logic of the process(s) is partially or completely implemented by physical hardware.
[0044] Each user computing system 102, 104, and 106 has automations 110, 112, and 114 that run on it, each implemented by an RPA robot, an AI agent, an AOP, etc. In some embodiments, automations 110, 112, and 114 may be stored remotely (e.g., stored on a server 130 or in a database 140 and accessed via a network 120) and loaded by an RPA robot and / or an AI agent to implement automations 110, 112, and 114. The database 140 may store structured data and / or unstructured data, the former of which is typically required for RPA. RPA automations may exist as scripts (e.g., Extensible Markup Language (XML), Extensible Application Markup Language (XAML), etc.) or be compiled into machine-readable code (e.g., as a digital link library). In the case of an AI agent, agent-type automation may be generated based on a plain text description of a desired target, for example.
[0045] The listener monitors and records data regarding user interactions with each computing system and / or the operation of the unattended computing system, and transmits this data to the core hyperautomation system 120 via a network (e.g., a local area network (LAN), a mobile communications network, a satellite communications network, the internet, or any combination thereof). The data may include, but is not limited to, which buttons were clicked, where the mouse was moved, text entered into fields, whether one window was minimized and another was opened, and the applications associated with the windows. In certain embodiments, the data from the listener may be transmitted periodically as part of a heartbeat message. In some embodiments, the data may be transmitted to the core hyperautomation system 120 after a predetermined amount of data has been collected, after a predetermined time has elapsed, or both. One or more servers, such as server 130, receive the data from the listener and store it in a database, such as database 140.
[0046] If automations 110, 112, and 114 are RPAs, they can execute logic developed within the workflows at design time. The workflows may include a series of steps, as defined herein as “activities,” which are executed sequentially or in other logical flows. Each activity may include actions such as clicking a button, reading a file, or writing to a log panel. In some embodiments, workflows may be nested or embedded.
[0047] In some embodiments, the long-running RPA workflow is a master project that supports service orchestration, human-in-the-loop, and long-running transactions in an unattended environment. See U.S. Patent No. 10,860,905, which is incorporated herein by reference in its entirety. Human-in-the-loop is involved when a particular process requires user input (e.g., dynamic and / or direct user input) for exception handling, approval, or verification before proceeding to the next activity step. In this situation, process execution is interrupted and the RPA robot is released until the human-in-the-loop portion of the task is completed.
[0048] Long-running workflows can support workflow fragmentation via persistence activities and, in combination with invocation processes and non-user interaction activities, can orchestrate human-in-the-loop behavior alongside RPA robot tasks. In some embodiments, multiple or numerous computing systems may participate in the execution of the logic of the long-running workflow. The long-running workflow may run within a session to facilitate fast execution. In some embodiments, the long-running workflow can orchestrate background processes that may include activities that execute API calls and operate within the long-running workflow session. These activities may, in some embodiments, be invoked by invocation process activities. Processes with user interaction activities that run in a user session may be invoked by starting a job from a conductor activity (conductors will be described in more detail later in this specification). In some embodiments, the user may interact with the task through a form that must be completed within the conductor. Activities may include causing the RPA robot to wait for the form task to be completed and then resume the long-running workflow.
[0049] One or more of the automations 110, 112, and 114 communicate with the core hyperautomation system 120. In some embodiments, the core hyperautomation system 120 may run conductor applications on one or more servers, such as server 130. Although one server 130 is shown for illustrative purposes, multiple or many servers in close proximity to each other, or servers in a distributed architecture, may be employed without departing from the scope of the invention. For example, one or more servers may be provided for conductor functions, AI / ML model serving, authentication, governance, and / or any other appropriate functions without departing from the scope of the invention. In some embodiments, the core hyperautomation system 120 may incorporate, or be part of, a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, etc. In certain embodiments, the core hyperautomation system 120 may host multiple software-based servers on one or more computing systems, such as server 130. In some embodiments, one or more servers of the core hyperautomation system 120, such as server 130, may be implemented by one or more virtual machines (VMs).
[0050] In some embodiments, one or more of the automations 110, 112, 114 may be deployed on or accessible by the core hyper-automation system 120 and may invoke one or more AI / ML models 132 trained to accomplish various tasks. For example, the AI / ML models 132 may include models trained to explore various application versions, models that perform computer vision (CV), models that perform optical character recognition (OCR), models that generate user interface (UI) descriptors, models that provide suggestions for the next activity or activity column in a workflow, models that perform semantic matching, models that perform natural language processing (NLP), and models that generate or modify code and / or workflows. The AI / ML models may be trained using labeled data that includes, but is not limited to, elements from data sources (e.g., web pages, forms, scanned documents, application interfaces, screens, etc.), previously created workflows, screenshots of various application screens for different versions and their corresponding UI elements, and libraries of UI objects. The AI / ML models 132 may be trained not to overfit to a given training dataset while achieving a desired confidence threshold. In general, UI elements, UI descriptors, applications, and application screens can be considered UI objects.
[0051] AI / ML model 132 can be trained for any suitable purpose without departing from the scope of the invention, as will be discussed in more detail later herein. In some embodiments, two or more AI / ML models 132 can be chained together (e.g., in series, parallel, or a combination thereof) so that they collectively provide a collaborative output. AI / ML model 132 can perform or assist in CV, OCR, document processing and / or document understanding, semantic learning and / or analysis, analytical prediction, process discovery, task mining, testing, automated workflow generation, sequence extraction, cluster detection, speech-to-text conversion, NLP, semantic matching, and any combination thereof. However, without departing from the scope of the invention, any desired number and / or types of AI / ML models can be used. By using multiple AI / ML models, the system can, for example, develop a larger picture of what is happening on a given computing system. For example, one AI / ML model can perform OCR, another can detect buttons, and yet another can compare sequences. The patterns may be determined individually by the AI / ML models, or they may be determined collectively by multiple AI / ML models. In certain embodiments, one or more AI / ML models are deployed locally on at least one computing system 102, 104, 106.
[0052] In some embodiments, multiple AI / ML models 132 may be used. Each AI / ML model 132 is an algorithm (or model) that operates on data, and the AI / ML model itself may be, for example, a deep learning neural network (DLNN) of artificial "neurons" trained on training data. In some embodiments, the AI / ML model 132 has multiple layers that perform various functions such as statistical modeling (e.g., Hidden Markov Models (HMMs)) and may utilize deep learning techniques (e.g., Long Short-Term Memory (LSTM) deep learning, encoding of previous hidden states, etc.) to perform the desired function.
[0053] In some embodiments, the hyper-automation system 100 may provide four main sets of functions: (1) discovery, (2) automation construction, (3) management, and (4) engagement. In some embodiments, the automation (e.g., running on a user computing system, server, etc.) may be performed by, for example, an RPA robot, AOP, or AI agent, and may provide any of the functions described herein. For example, an RPA robot may include an attendant robot, an unattended robot, and / or a test robot. An attendant robot works with the user to assist with tasks (e.g., via UiPath Assistant®). An unattended robot operates independently of the user, may run in the background, and may potentially operate without user awareness. A test robot executes test cases against an application or workflow. In some embodiments, test robots may run in parallel on multiple computing systems.
[0054] The discovery function may discover various opportunities for automating business processes and provide automated recommendations for them. Such a function may be implemented by one or more servers, such as server 130. In some embodiments, the discovery function may include an automation hub, process mining, task mining, and / or task capture. An automation hub (e.g., UiPath Automation Hub®) may provide a mechanism for managing automation deployments with visibility and control. For example, automation ideas may be crowdsourced from employees via a submission form. Feasibility and return on investment (ROI) calculations for automating these ideas may be provided, documentation for future automations may be collected, and collaboration may be provided to get from automation discovery to build more quickly.
[0055] Process mining (e.g., via UiPath Automation Cloud® and / or UiPath AI Center®) refers to the process of collecting and analyzing data from applications (e.g., enterprise resource planning (ERP) applications, customer relationship management (CRM) applications, email applications, call center applications, etc.) to identify what end-to-end processes exist within an organization, how to effectively automate them, and what the impact of that automation will be. For example, this data may be collected by listeners from user computing systems 102, 104, and 106 and processed by a server such as server 130. In some embodiments, one or more AI / ML models 132 may be used for this purpose. This information may be exported to an automation hub to expedite implementation and avoid manual information transfer. The objective of process mining may be to increase business value by automating processes within an organization. Examples of goals for process mining include, but are not limited to, increased profits, improved customer satisfaction, compliance with regulations and / or contracts, and improved employee efficiency.
[0056] Task mining (e.g., via UiPath Automation Cloud® and / or UiPath AI Center®) identifies and aggregates workflows (e.g., employee workflows), then applies AI to reveal patterns and variations in daily tasks, and scores such tasks in terms of ease of automation and potential savings (e.g., time savings and / or cost savings). One or more AI / ML models 132 may be used to reveal repetitive task patterns in the data. Repetitive tasks suitable for automation can then be identified. In some embodiments, this information may initially be provided by a listener and analyzed on a server of a core hyperautomation system 120, such as a server 130. Insights from task mining (e.g., XAML process data) may be exported to process documents or designer applications such as UiPath Studio®, thereby enabling faster creation and deployment of automations. In some embodiments, task mining may include taking screenshots with user actions (e.g., mouse click locations, keyboard inputs, application windows and graphical elements the user was interacting with, and timestamps of such interactions), collecting statistical data (e.g., execution time, number of actions, number of text inputs), editing and annotating screenshots, and specifying the types of actions to be recorded.
[0057] Task capture (e.g., via UiPath Automation Cloud® and / or UiPath AI Center®) automatically documents attended processes as users work through them, or provides a framework for unattended processes. Such documentation may include, in the form of a Process Definition Document (PDD), the desired tasks to be automated, a skeletal workflow, captures of actions for each part of the process, recordings of user actions, and the automatic generation of a comprehensive workflow diagram including details of each step, as well as Microsoft Word® documents, XAML files, etc. In some embodiments, the build-ready workflow can be directly exported to a designer application such as UiPath Studio®. Task capture can simplify the requirements gathering process for both subject matter experts describing processes and Center of Excellence (CoE) members providing production-quality automation.
[0058] The construction of automation can be achieved through designer applications (e.g., UiPath Studio®, UiPath StudioX®, or UiPath Studio Web®). For example, developers in the RPA development facility 150 can use the designer application 154 on the computing system 152 to build and test agent-based automation, RPA, AOP, and / or hybrid automation for various applications and environments such as web, mobile, SAP®, and virtualized desktops. Developers can also build AOP. For example, developers can create automations that are executed by RPA robots, AI agents, AOP, or combinations thereof. API integrations can be provided for various applications, technologies, and platforms. Predefined activities, drag-and-drop modeling, and workflow recorders can facilitate automation with minimal coding. Document comprehension capabilities can be provided through drag-and-drop AI skills that invoke one or more AI / ML models 132 for data extraction and interpretation. Such automations can handle virtually any document type and format, including tables, checkboxes, signatures, and handwritten text. When data is validated or exceptions are handled, this information can be used to retrain each AI / ML model, thereby improving its accuracy over time.
[0059] The designer application 152 may be designed to invoke one or more trained AI / ML models 132 on the server 130 and / or generated AI models 172 in a cloud environment via the network 120 (e.g., a local area network (LAN), mobile communication network, satellite communication network, internet, or any combination thereof) to assist in the automated development process. In some embodiments, one or more of the AI / ML models may be packaged in the designer application 152 or otherwise stored locally on the computing system 150.
[0060] In some embodiments, one or more of the designer application 152 and AI / ML models 132 may be configured to use an object repository stored in a database 140. See U.S. Patent No. 11,748,069, which is incorporated herein by reference in its entirety. Generally, an object repository is a storage mechanism used by automation for images, text, semantic data, taxonomic associations, ontological associations, UI objects, etc. For example, an object repository may contain a library of UI objects that can be used to develop workflows through the designer application 152. The object repository may be used for UI automation to add UI descriptors to activities in the workflow of the designer application 152. In some embodiments, one or more of the AI / ML models 132 may generate new UI descriptors and add them to the object repository in the database 140.
[0061] Once automation is complete in the designer application 152, it can be exposed on the server 130 and pushed to computing systems 102, 104, 106, etc. For example, when a new UI descriptor is created and / or an existing UI descriptor is modified, a global repository of UI object libraries can be built that is shareable and collaborative for all automations. Taxonomies and ontologities can be used for the object repository. A taxonomy is a hierarchical structure of subcategories. An ontology is a formal representation of a knowledge domain, including concepts, attributes, and the relationships between them. In an ontology, the relationships between categories are not necessarily hierarchical, and these ontological relationships can span multiple screens of an application.
[0062] For example, integrated services can enable developers to seamlessly combine UI automation and API automation. Automations that require APIs or span both API and non-API applications and systems can be built, such as any type of automation described herein. A repository (e.g., UiPath Object Repository®) or marketplace (e.g., UiPath Marketplace®) for pre-built automation templates and solutions may be provided, thereby enabling developers to automate a wide variety of processes more quickly. Thus, when building automations, the hyperautomation system 100 may provide a user interface, development environment, API integration, pre-built and / or custom-built AI / ML models, development templates, an integrated development environment (IDE), and advanced AI capabilities. In some embodiments, the hyperautomation system 100 may enable the development, deployment, management, configuration, monitoring, debugging, and maintenance of RPA robots, AOPs, and AI agents, which may provide automation for the hyperautomation system 100.
[0063] In some embodiments, components of the hyperautomation system 100, such as designer applications and / or external rule engines, assist in managing and enforcing governance policies to control the various functions provided by the hyperautomation system 100. Governance is the ability of an organization to set policies to prevent users from developing automations (e.g., RPA robots, AOPs, and / or AI agents) that could perform actions that could harm the organization, such as violating the EU General Data Protection Regulation (GDPR), the US Health Insurance Portability and Accountability Act (HIPAA), or the terms of use of third-party applications. In some embodiments, access control and governance restrictions are implemented at the level of robots and / or robot design applications, as developers may, in other cases, create automations that violate privacy laws, terms of use, etc., when executing those automations. This can provide an additional level of security and compliance to the automation process development pipeline by preventing developers from taking on reliances on unauthorized software libraries that could introduce security risks or operate in a manner that violates policies, regulations, privacy laws, and / or privacy policies. See U.S. Patent No. 11,733,668, which is incorporated in its entirety herein by reference.
[0064] The management functions may provide management, deployment, and optimization of automation across the entire organization. In some embodiments, the management functions may include orchestration, test management, AI capabilities, and / or insights. The management functions of the hyperautomation system 100 may also act as an integration point with third-party solutions and applications for automation applications and / or RPA robots. The management capabilities of the hyperautomation system 100 may include, but are not limited to, provisioning, deployment, configuration, queuing, monitoring, logging, and facilitating interoperability of RPA robots, AOP, and / or AI agents.
[0065] Conductor applications such as UiPath Orchestrator® (which in some embodiments may be provided as part of UiPath Automation Cloud®, or on-premises, within a VM, within a private or public cloud, within a Linux® VM, or as a cloud-native single container suite via UiPath Automation Suite®) provide orchestration capabilities for deploying, monitoring, optimizing, scaling, and securing RPA robots, AOP, and / or AI agents. A test suite (e.g., UiPath Test Suite®) may provide test management for monitoring the quality of deployed automations. The test suite may facilitate test planning and execution, requirements fulfillment, and defect traceability. The test suite may include comprehensive test reporting.
[0066] Analytics software (e.g., UiPath Insights®) can track, measure, and manage the performance of deployed automations. The analytics software can align automation behavior with specific key performance indicators (KPIs) and strategic outcomes of the organization. The analytics software can present results in a dashboard format to enhance user understanding.
[0067] A data service (e.g., UiPath Data Service®) can bring data to a single, scalable, and secure location, for example, stored in a database 140 and featuring a drag-and-drop storage interface. Several embodiments can provide automation with low-code or no-code data modeling and storage while ensuring seamless access to data, enterprise-grade security, and scalability. AI functionality can be provided by an AI Center (e.g., UiPath AI Center®), which facilitates the integration of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options can make such functionality accessible even to non-data scientists. Deployed automation (e.g., RPA robots, AOPs, and AI agents) can invoke AI / ML models from the AI Center, such as AI / ML model 132. The performance of AI / ML models can be monitored and trained and improved using user-validated data, such as that provided by a data review center 160. A user acting as a reviewer may provide labeled data to the core hyperautomation system 120 via a review application 152 on the computing system 154. For example, the reviewer may verify the accuracy of predictions made by the AI / ML model 132 and / or the generative AI model 172, or provide corrections if they are not. The user acting as a reviewer may also provide dynamic and / or direct user input to the AI agent (e.g., within the scope of human-in-the-loop operations), and such dynamic and / or direct user input (e.g., responses and corrections provided by the reviewer) may be used to train the LLM used by the AI agent to be more accurate. In other words, the dynamic and / or direct user input may be stored as training data for retraining the AI / ML model 132 and / or the generative AI model 172, for example, in a database such as database 140.Subsequently, the AI Center may schedule and execute training jobs to train a new version of the AI / ML model using the aforementioned training data. Both positive and negative examples are stored and can be used for retraining the AI / ML model 132 and / or the generative AI model 172.
[0068] The aforementioned engagement feature involves automation and users as a single team to enable seamless collaboration on the desired process. In some embodiments, low-code applications can be built (e.g., via UiPath Apps®) and connected to browser tabs and legacy software, even those lacking APIs. For example, applications can be quickly created using a web browser through a rich library of drag-and-drop controls. Applications can be connected to a single automation or multiple automations.
[0069] An action center (e.g., UiPath Action Center®) provides a simple and efficient mechanism for handing over processes from automation to users and vice versa. Users can provide approvals or escalations, make exceptions, etc. The automation can then perform automated functions of a given workflow.
[0070] A local assistant may be provided as a launchpad for the user to launch automations (e.g., UiPath Autopilot®). Such an assistant may also provide semantic cut-and-paste functionality (e.g., UiPath Clipboard AI®). See U.S. Patent No. 12,124,806 and U.S. Patent Application Publications 2023 / 0107316, 2023 / 0415338, and 2024 / 0220581. For example, this functionality may be provided in a tray provided by the operating system and may allow the user to interact with RPA robots, AOPs, AI agents, and automation-driven applications on their computing system. The interface may enumerate automations approved for a given user and allow the user to run them. These may include ready-to-use automations from an automation marketplace, automations from an internal automation store within an automation hub, etc. When automations are run, they may run as local instances in parallel with other processes on the computing system so that the automations can perform their operations while the user is using the computing system. In certain embodiments, the assistant is integrated with a task capture function, which allows the user to document their processes that will soon be automated from the assistant launchpad.
[0071] In some embodiments, end-to-end measurement and governance of automation programs of any size may be provided by the hyper-automation system 100. As described above, analytics may be used to understand the performance of automation (e.g., via UiPath Insights®). Data modeling and analytics using any combination of available business metrics and operational insights may be used for various automation processes. Custom-designed and pre-built dashboards may provide visibility into data across desired metrics, uncover new analytical insights, track performance metrics, discover ROI for automation, perform telemetry monitoring on user computing systems, detect errors and anomalies, and debug automation. An automation management console (e.g., UiPath Automation Ops®) may be provided to manage automation throughout the entire automation lifecycle. Organizations may be able to manage how automations are built, what users can do with them, and which automations users can access.
[0072] In some embodiments, the hyperautomation system 100 provides an iterative platform where processes are discovered, automation is built, tested, and deployed, performance is measured, the use of automation is easily provided to users, feedback is obtained, AI / ML models are trained and retrained, and the process can be repeated. This facilitates a more robust and effective set of automations.
[0073] In some embodiments, the generative AI model 172 is used as described above. For example, an AI agent utilizes the generative AI model. The generative AI model 172 can generate various types of content, such as text, images, audio, and synthetic data. Various types of generative AI models can be used, including but not limited to LLMs, generative adversarial networks (GANs), diffusion models, flow-based models, variational autoencoders (VAEs), and transformers. For example, in the case of LLMs, NLP models such as word2vec, BERT, GPT-3, and ChatGPT may be used in some embodiments to facilitate semantic understanding and provide more accurate and human-like responses. These models may be part of an AI / ML model 132 hosted on server 130. For example, the generative AI model 172 may be trained on a large text information corpus to perform semantic understanding, understand the properties of things present on a screen from text, and automatically generate code. An AI agent may use such a generative AI model 172. In certain embodiments, generative AI models 172 provided by existing cloud ML service providers such as OpenAI®, Google®, Amazon®, Microsoft®, IBM®, Nvidia®, and Meta® may be used and trained to provide such functionality. In embodiments of generative AI where the generative AI model(s) 172 are remotely hosted, the server 130 may be configured to integrate with third-party APIs that enable it to send requests containing necessary input information to the generative AI model(s) 172 and receive responses (e.g., semantic matching of fields between application versions, classification of application types on a screen, responses to natural language queries from users, etc.). Such embodiments may provide a more advanced and sophisticated user experience and may provide access to state-of-the-art NLP and other ML capabilities offered by these companies.
[0074] One aspect of the generative AI model 172 in some embodiments is the use of transfer learning. In transfer learning, a pre-trained generative AI model, such as an LLM, is fine-tuned for a specific task or domain. This allows the LLM to leverage the knowledge it has already learned during its initial training and adapt it to a specific application. In the case of an LLM, the pre-training stage typically involves training the LLM on a large text corpus consisting of billions of words. In this stage, the LLM learns the relationships between words and phrases, thereby enabling it to generate consistent, human-like responses to text-based input. The output of this pre-training stage is an LLM with a high level of understanding of fundamental patterns in natural language.
[0075] In the fine-tuning phase, a pre-trained LLM is adapted to a specific task or domain by training the LLM on a smaller dataset specific to that task. For example, in some embodiments, the LLM may be trained to analyze specific types or combinations of data sources to improve its accuracy regarding their content. This data may include, but is not limited to, prompt tuning or instruction tuning, where the model is specifically trained to better understand and follow specific types of instructions or prompts, thereby improving its ability to perform a particular task when given appropriate instructions. Such information may be provided as part of the training data, and the LLM may learn to focus on these domains and more accurately identify the data elements within them. Fine-tuning allows the LLM to learn the nuances of its task or domain, such as specific vocabulary and syntax used in that domain, without requiring as much data as would be needed to train the LLM from scratch. By leveraging the knowledge learned in the pre-training phase, a fine-tuned LLM can achieve state-of-the-art performance on a specific task with relatively small amounts of training data.
[0076] LLM can utilize vector databases. Vector databases, along with their vector embeddings, index, store, and provide access to structured or unstructured data (e.g., text, images, time-series data). Data such as text can be tokenized, where a single character, word, or sequence of words is parsed from text into tokens. These tokens are then "embedded" in vector embeddings, which are numerical representations of this data. Vector databases enable LLM to quickly and scalably find and retrieve similar objects in a production environment, which is impossible with manual processes.
[0077] AI and ML enable the numerical representation of unstructured data in vector embeddings without losing its semantic meaning. A vector embedding is a long sequence of numbers that describes the features of the data object it represents. Similar objects are grouped closer together in the vector space. In other words, the more similar objects are, the closer their vector embeddings will be. Similar objects can be found using vector search, similarity search, or semantic search and retrieve. The distance between vector embeddings can be calculated using a variety of techniques, including but not limited to squared Euclidean distance or L2-squared distance, Manhattan distance or L1 distance, cosine similarity, dot product, and Hamming distance. It may be beneficial to choose the same metrics used to train the AI / ML model.
[0078] Vector indexing can be used to organize vector embeddings so that data can be retrieved efficiently. Calculating the distance between a vector embedding and all other vector embeddings in a vector database using the k-nearest neighbors (kNN) algorithm can be computationally expensive when the number of data points is large, because the required calculations increase linearly (O(n)) with the number of dimensions and data points. Finding similar objects using an approximate nearest neighbors (ANN) approach is more efficient. The distances between vector embeddings are pre-calculated, and similar vectors are organized and stored close to each other (e.g., in clusters or graphs) so that similar objects can be found more quickly. This process is called "vector indexing." ANN algorithms that can be used in some embodiments include, but are not limited to, cluster-based indexing, proximity-graph-based indexing, tree-structure-based indexing, hash-based indexing, and compression-based indexing.
[0079] Figure 2 shows some of the combined capabilities 200 of an AI agent 210 and an RPA robot 220 in one or more embodiments. The AI agent 210 is configured to process natural language instructions, achieve expected goals 231 based thereon, execute dynamic decision-making or dynamic flow control with self-correction capabilities 233, store information in long-term memory to evaluate its own execution performance 235, and learn from human-in-the-loop and self-performance 237 during execution. The RPA robot 220 can respond to triggers 241 (e.g., from a conductor application such as UiPath Orchestrator®), respond based on context 243 (i.e., the RPA robot 220 may perform deterministic steps such as retrieving information from the context and updating a document based on the information retrieved from the context; or the agent 210 may use the retrieved context to update a dynamic plan and perform the next step to complete the objective according to the instructions), leverage models 245 (e.g., CV models, document processing models, speech-to-text models, OCR models, AI models, etc.), leverage tools 247 (e.g., leveraging tools available within the RPA ecosystem such as completed automations, workflows within automations, integrated service connector calls for third-party and first-party services, RPA designer application activities, LLM calls, automations, etc.), and be leveraged by the AI agent 210 to perform actions 249 that the RPA robot 220 may take based on input from the AI agent 210 (i.e., using the RPA robot 220 as a tool). The AI agent 210 may also perform actions 249 to update its own memory, update the plan to achieve its objectives with each instruction, self-evaluate and learn from its actions, self-repair when it encounters a failure, and escalate to the user when assistance is needed.
[0080] As described above, agent-based automation achieves a variety of technical effects, benefits, and advantages. Agent-based automation improves memory usage by reducing the amount of storage required for data and improves processor efficiency by reducing the number of calls and actions. Agent-based automation provides the ability to process gigabytes, terabytes, petabytes, or more of data that would be impossible with human implementation processes that are mental or manual. Agent-based automation enables the use of fewer triggers and models through dynamic decision-making. For example, and as described herein, while traditional software automation alone may require 100 actions in an exemplary scenario, agent-based automation in the same exemplary scenario can significantly reduce the number of actions required (e.g., to 15 actions). Agent-based automation can also improve the efficiency of the LLM by using context grounding to tether the AI agent 210 to a desired context, thereby "constraining" the LLM to the relevant context.
[0081] As used herein, “context grounding” refers to a methodology that improves models such as LLMs by integrating company-specific information with pre-trained knowledge, thereby enabling accurate responses to specialized or recent queries. In some embodiments, context grounding extends LLM responses using external data to obtain responses that the LLM would not inherently know and respond to queries based on the provided context. For example, because proprietary industry jargon and complex document structures can pose challenges in ensuring effective retrieval and semantic matching, context grounding addresses this challenge by providing accurate chunking of documents, thereby allowing relevant information (e.g., from the proprietary industry jargon and complex document structures) to be passed to the LLM without noise. As an additional example, context grounding improves LLM responses by providing extract and retrieval techniques (e.g., tailored to the proprietary industry jargon and complex document structures) that are adapted to diverse industries and applications.
[0082] Figure 3 shows diagrams 300 of AOP, AI agent, RPA robot, and application according to one or more embodiments.
[0083] AOP pool 310 includes AOPs 1, 2, ..., P that implement business processes. As described above, AOPs can be implemented as BPMN, which are executed by an AOP execution engine such as Temporal®. AOPs can utilize AI agents and / or RPA robots to execute parts of their business processes.
[0084] The AI agent pool 320 includes AI agents 1, 2, ..., I, who are trained to perform various tasks such as investigating claims, pursuing resolutions with employees, and summarizing policies and technical specifications. The RPA robot pool 330 includes RPA robots 1, 2, ..., J, who perform various automations such as UI automation, semantic matching automation, and form entry automation.
[0085] The application pool 340 includes applications 1, 2, ..., K in which the AI agent and / or RPA robot can interact. For example, the applications may include CRM applications, invoicing applications, payroll applications, banking applications, web applications, legacy system applications, word processor applications, spreadsheet applications, email applications, and the like. The AI agent, RPA robot, and applications may reside on a single computing system, or on multiple or numerous computing systems. The AOP is typically located in the cloud or on a server side, and in some embodiments, may be on the same computing system as the conductor application 350.
[0086] The AOP can trigger or invoke AI agents and RPA robots via the conductor application 350. AI agents and RPA robots can also trigger or invoke each other via the conductor application. For example, to invoke an RPA robot, the AI agent may make a "Start Job" call within the conductor application 350. Note that the RPA robot is deployed as automation controlled by the conductor application 350. AI agents, AOP, and RPA robots can also trigger or invoke specific applications. For example, through information obtained from human-in-the-loop actions, the AI agent may dynamically learn which RPA robots, other AI agents, and / or applications to trigger or invoke to accomplish a task. For example, the AI agent may learn to trigger an RPA robot via the conductor application 350 to fill out and submit a web form. The AI agent may also learn to open Microsoft Excel®, enter form information into the appropriate tabs, open and update a payroll application, and so on. The AI agent may also learn to call or trigger an email resolution AI agent via the conductor application 350 to contact a bank customer service representative if a problem occurs. In some embodiments, the technical effects, benefits, and advantages may be similar to those discussed above with respect to Figures 1 and 2.
[0087] To find each other, AI agents, AOPs, and RPA robots may belong to a tenant. A designer application may call a conductor to obtain a list of available RPAs. In some embodiments, there are three ways to obtain the capabilities of an automation: (1) a user providing a description of what the automation does when creating a workflow in a designer application; (2) an AI agent and ML techniques generating a summary of what a given workflow does; and (3) a developer describing what the automation does in a designer application. A conductor application may also have a list of which applications are available for a given AI agent and RPA robot. In other words, descriptions of available AI agents, RPA robots, and / or applications are derived or assigned by the AI agent, ML techniques, or user.
[0088] Figure 4 shows an exemplary agent service interface 400 in one or more embodiments. As shown in Figure 4, the agent answers questions regarding a policy document provided within context grounding. The agent instruction pane 410 contains a natural language description entered by the user about what the AI agent is intended to do. The user prompt 420 allows the developer to enter user prompt content in the content field 422, if necessary. The tool dropdown 430 allows the developer to select the tools the AI agent will use, for example, using an API for the application or calling an RPA robot to perform RPA.
[0089] The context dropdown 440 allows developers to configure the context grounding for the AI agent. The context configuration pane 442 allows developers to provide a description via the description field 444 and an Elastic Common Schema (ECS) index via the ECS index field 446 for a specific policy document containing information such as contracts, regulations, and what to do, in this example. Developers may also add contexts 450 to further supplement the context grounding. User escalation options can be configured via the escalation dropdown 460. The query field 470 allows users to provide queries that the AI agent will respond to. When the user clicks the run button 480, the AI agent executes the queries. The results are then shown in the run pane 490 while the AI agent retrieves and outputs them.
[0090] Figure 5 shows an exemplary AOP development interface 500 in one or more embodiments. The AOP development interface 500 includes an AOP 510, an AI agent 520, and an RPA robot 530 that can be selected by the user when developing a business process. These can be selected by the AI agent, AOP, or developer and dragged to the canvas 540 to develop the AOP. In this example, a credit check 541 is implemented by retrieving customer data 543 from the database 545 based on a credit check request 542. Next, the AI agent 546 is called and determines the customer type (e.g., very likely to pay, likely to default, frequently unemployed, etc.) by analyzing the customer data in the database 545. This type is then provided to the RPA robot 547, which takes this information into account when performing the credit check and generating the credit check result 549. Alternatively, the AI agent, AOP, or developer may enter a description of the business process in the field 550 and select the generate button 560. This explanation is provided to the LLM, which attempts to understand the business process and automatically create the AOP. The AI agent, AOP, or developer can then edit the AOP.
[0091] Figure 6 shows an exemplary RPA development interface 600 in one or more embodiments. The RPA development interface 600 includes components 610 that can be selected by an AI agent, AOP, or developer when developing a workflow for an RPA robot. The AI agent, AOP, or developer may select and drag these components onto the canvas 620. Alternatively, the AI agent, AOP, or developer may enter a description of the workflow in the field 630 and select the generate button 640. This description is provided to the LLM, which attempts to understand the workflow and automatically create an RPA robot. The AI agent, AOP, or developer can then edit the workflow for the RPA robot. Note that the functions shown and described with respect to Figures 4, 5, and 6 may, in some embodiments, be provided in a single designer application.
[0092] Figure 7 shows an end-to-end AI agent, RPA robot, and AOP development and deployment system 700 in one or more embodiments. A designer application 710 allows the AI agent, AOP, and developer to design workflows for subsequent automation (e.g., AOP, AI agent, and / or RPA robot). Once these subsequent AOP, AI agent, and / or RPA robots are tested and validated, the validated and tested AOP, AI agent, and / or RPA robots are packaged and published to the automation database 720.
[0093] The conductor application 730 manages the deployment of these packaged and exposed automations. When a software process 732 requests the execution of an automation, the conductor application 730 sends a start job command to the AOP engine 740, which then selects and starts an automation from the AOP group 742. When executing the AOP group 742, it may encounter steps implemented by the AI agent 750 and / or the RPA robot 760. In some cases, the AOP engine 740 sends a request to the conductor application 730 to interrupt the running AOP group 742 and send a start job request to the appropriate AI agent 750 or RPA robot 760.
[0094] When an AI agent is requested, the conductor application 730 sends a start job request to the appropriate AI agent 750. This request may include natural language text or other information provided from the AOP engine 740 to the conductor application 730. The AI agent 750 then performs the step by running LLM 752 to assist in task execution. The AI agent 750 then sends task-related information (e.g., requested information, an indication that the step was completed, an indication that the step failed, etc.) to the conductor 730, which then provides this information to the AOP engine 740. The AOP engine 740 then resumes its operation.
[0095] When an RPA robot is requested, the conductor application 730 sends a start job request to the appropriate RPA robot 760. The RPA robot 760 then executes the requested RPA 762. The RPA robot 760 then sends task-related information (e.g., requested information, instructions that the step was completed, instructions that the step failed, etc.) to the conductor application 730, which provides this information to the AOP engine 740. The AOP engine 740 then resumes operation.
[0096] According to one or more embodiments, the AOP 742, AI agent 750, or RPA 762 may require user action. In this case, the AOP engine 740, AI agent 750, or RPA robot 760 contacts the user 770 for human-in-the-loop action that contributes to automation. After the user 770 provides user action, the AOP engine 740, AI agent 750, or RPA robot 760 resumes automation.
[0097] Figure 8 is an architectural diagram showing an agent-based automation and RPA system 800 in one or more embodiments. In some embodiments, the agent-based automation and RPA system 800 is part of the hyper-automation system 100 in Figure 1. The agent-based automation and RPA system 800 includes a designer 810 that enables AI agents, AOPs, or developers to design automations (e.g., workflows, natural language instructions for AI agents and AOPs, context grounding, tool configurations, RPA robots, AOPs, AI agents, etc.). The designer 810 may provide solutions for application integration, as well as automation of third-party applications, managed information technology (IT) tasks, and business IT processes. The designer 810 may facilitate the development of automation projects, which are graphical representations of business processes. The designer 810 facilitates the development and deployment of automations (indicated by arrow 811). The designer 810 may be an application that runs on a user's desktop, an application that runs remotely within a VM, a web application, etc.
[0098] An automation project enables the automation of rule-based processes by allowing an AI agent, AOP, or developer to control the execution order and relationships of custom steps developed in a workflow, i.e., the “Activities” described herein. One commercial example of an embodiment of Designer 810 is UiPath Studio®. Each Activity may include actions such as clicking a button, reading a file, or writing to a log panel. In some embodiments, workflows may be nested or embedded.
[0099] Some types of workflows may include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suitable for linear processes, allowing a flow from one activity to another without complicating the workflow. Flowcharts may be particularly suitable for more complex business logic, allowing for the integration of decision-making and more diverse connections of activities through multiple branching logic operators. FSMs may be particularly suitable for large-scale workflows. FSMs may use a finite number of states in their execution, which are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suitable for determining workflow behavior and debugging processes when execution errors are encountered.
[0100] Once automation is developed in the designer 810, the execution of business processes is orchestrated by the conductor 820, which orchestrates one or more robots 830, one or more AI agents 850, and / or one or more AOPs 870 that execute the workflow developed in the designer 810. One commercial example of an embodiment of the conductor 820 is UiPath Orchestrator®. The conductor 820 facilitates the creation, monitoring, and management of resources in the environment. The conductor 820 can function as an integration point with third-party solutions and applications. As described above, in some embodiments, the conductor 820 may be part of the core hyperautomation system 120 in Figure 1.
[0101] It should be noted that the RPA robot 830 can operate independently for deterministic processes. The AI agent 850 and AOP 870 can also operate independently (for example, for non-deterministic processes), or they can utilize the RPA robot 830 or other AI agent 850 as tools to achieve part of their agent-based automation. The AI agent 850 can drive a composite automation that utilizes both the RPA robot 830 and the AI agent 850, or vice versa, and the AOP 870 can include such a composite automation.
[0102] The conductor 820 manages a group of RPA robots 830 and AI agents 850, and can connect and execute the RPA robots 830 and AI agents 850 from a centralized point (for example, as required by an AOP engine implementing AOP) (indicated by arrow 881). The types of RPA robots 830 that can be managed may include, but are not limited to, attendant robots, unattended robots, development robots (similar to unattended robots but used for development and testing purposes), and non-production robots (similar to attendant robots but used for development and testing purposes). Attended robots are triggered by user events and operate alongside the user on the same computing system. Attended robots may be used with the conductor 820 for centralized process deployment and log recording media. Attended robots may assist users in accomplishing various tasks and may be triggered by user events. In some embodiments, the process cannot be started from the conductor 820 on this type of robot and / or run under a locked screen. In certain embodiments, the attendant robot can only be started from the robot tray or from a command prompt. In some embodiments, the attendant robot should run under user supervision.
[0103] Unattended robots can run unattended in a virtual environment and automate many processes. Unattended robots can be responsible for remote execution, monitoring, scheduling, and providing support for work queues. In some embodiments, debugging for all robot types can be performed in Designer 810. Both attended and unattended robots can automate a wide range of systems and applications, including, but not limited to, mainframes, web applications, VMs, enterprise applications (e.g., those provided by SAP®, Salesforce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).
[0104] Conductor 820 may have a variety of capabilities, including but not limited to provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interoperability (indicated by arrow 882). Provisioning may include creating and maintaining connections between RPA robots 830, AI agents 850, and / or AOP 870 and Conductor 820 (e.g., a web application). Deployment may include ensuring that package versions are correctly delivered to RPA robots 830, AI agents 850, and / or AOP 870 assigned for execution. Configuration may include maintaining and delivering environment and process configurations for RPA robots and AI agents. Queuing may include providing management of queues and queue items. Monitoring may include tracking identification data for robots and AI agents and maintaining user permissions. Logging may include storing and indexing logs in a database (e.g., a Structured Query Language (SQL) database or a "Not Only" SQL (NoSQL) database) and / or another storage mechanism (e.g., ElasticSearch®, which provides the ability to store and rapidly query large datasets). Conductor 820 may provide interoperability by acting as a centralized communication point for third-party solutions and / or applications.
[0105] The RPA robot 830 is an execution agent that implements the workflow built in the designer 810. One commercial example of several embodiments of the RPA robot 830 is UiPath Robots®. In some embodiments, the RPA robot 830 installs the Microsoft Windows® Service Control Manager (SCM) management service by default. As a result, such an RPA robot 830 can open an interactive Windows® session under the local system account and has Windows® service privileges.
[0106] In some embodiments, the RPA robot 830 may be installed in user mode. For such an RPA robot 830, this means that it has the same privileges as the user on which the given RPA robot 830 is installed. This functionality may also be available for high-density (HD) robots, thereby ensuring that each machine is fully utilized to its maximum capacity. In some embodiments, any type of RPA robot 830 may be configured in an HD environment.
[0107] In some embodiments, the RPA robot 830 is divided into several components, each dedicated to a specific automation task. These robot components may include, but are not limited to, an SCM management robot service, a user-mode robot service, an executor, an agent, and a command line. The SCM management robot service manages and monitors Windows® sessions and acts as a proxy between the conductor 820 and the execution host (i.e., the computing system on which the RPA robot 830 runs). These services are entrusted with and manage the authentication credentials for the RPA robot 830. The console application is launched by the SCM under the local system.
[0108] In some embodiments, the user-mode robot service manages and monitors Windows® sessions and acts as a proxy between the conductor 820 and the execution host. The user-mode robot service may be entrusted with and manage authentication information for the RPA robot 830. If the SCM management robot service is not installed, the Windows® application may be automatically launched.
[0109] The executor can execute a given job (i.e., execute a workflow) under a Windows® session. The executor can be aware of per-dots-per-inch (DPI) settings for each monitor. AI Agent 850 can be a Windows® Presentation Foundation (WPF) application that displays available jobs in the system tray window. Note that these agents are different from AI Agent 850. AI Agent 850 can be a service client and can request the start or stop of jobs and changes to their settings. The command line is a service client. The command line is a console application that can request the start of a job and wait for its output.
[0110] By dividing the RPA robot 830 into components as described above, developers, support users, and computing systems can more easily execute, identify, and track what each component is doing. This method allows for the configuration of special actions on a component-by-component basis, such as setting different firewall rules for executors and services. In some embodiments, the executor may always be aware of the DPI setting per monitor. As a result, workflows may run at any DPI, regardless of the configuration of the computing system on which they were created. In some embodiments, projects from the designer 810 may also be independent of the browser zoom level. For applications that are not DPI-aware, or applications that are intentionally marked as non-DPI-aware, DPI may be disabled in some embodiments.
[0111] The agent-based automation and RPA system 800 in this embodiment is part of a hyperautomation system, such as the hyperautomation system 100 in Figure 1. Developers can use the designer 810 to build and test RPA, AOP, and AI agents that utilize AI / ML models deployed in the core hyperautomation system 840 (for example, as part of its AI center). Such RPA robots can send inputs for the execution of AI / ML models(s) and receive outputs from them via the core hyperautomation system 840.
[0112] One or more of the RPA robots 830 may be listeners, as described above. These listeners may provide the core hyperautomation system 840 with information about what the user is doing when using its computing system. This information can then be used by the core hyperautomation system for process mining, task mining, task capture, and the like.
[0113] An assistant / chatbot (of the Conductor 820) may be provided on the user computing system, allowing the user to launch a local RPA robot. For example, the assistant / chatbot may be located in the system tray. The chatbot may have a user interface so that the user can see the text within the chatbot. Alternatively, the chatbot may not have a user interface and may run in the background, listening to the user's voice using the computing system's microphone.
[0114] In some embodiments, data labeling may be performed by a user of the computing system on which the RPA robot or AI agent is running, or on another computing system on which the robot or AI agent provides information. For example, if a robot invokes an AI / ML model to perform CV on an image for a VM user, but the AI / ML model does not correctly identify a button on the screen, the user may draw a rectangle around the misidentified or unidentified component and potentially provide text indicating the correct identification. This information is provided to the core hyperautomation system 540, which can then be used later to train a new version of the AI / ML model.
[0115] Figure 9 is an architectural diagram showing a deployed RPA system 900 in one or more embodiments. In some embodiments, the RPA system 900 may be part of the agent-based automation and RPA system 800 in Figure 8, and / or the hyper-automation system 100 in Figure 1. Note that the architecture of the deployed RPA system 900 may not be used in some embodiments. The deployed RPA system 900 may be a cloud-based system, an on-premise system, a desktop-based system, etc., providing enterprise-level, user-level, or device-level automation solutions for automating different computing processes.
[0116] It should be noted that the client side 901, the server side 902, or both may include any desired number of computing systems without departing from the scope of the present invention. In the client side 901, the robot application 910 includes an executor 912, an execution agent 914, and a designer 916. However, in some embodiments, the designer 916 does not have to run on the same computing system as the executor 912 and the execution agent 914. The executor 912 is a running process. Multiple business projects may be running simultaneously. The execution agent 914 (e.g., Windows® service) is, in this embodiment, a single point of contact for all executors 912. The execution agent 914 is also responsible for transmitting the robot's status (e.g., periodically sending "heartbeat" messages indicating that the robot is still functioning) and downloading the necessary versions of packages to be executed.
[0117] The listener 930 monitors and records data relating to user interactions with the attended computing system in which the listener 930 resides, and / or the operation of the unattended computing system. Without departing from the scope of the present invention, the listener 930 may be an RPA robot, an AOP, an AI agent, part of an operating system, a downloadable application for the respective computing system, or any other software and / or hardware. In fact, in some embodiments, the logic of the listener is partially or completely implemented by physical hardware.
[0118] On the server side 902, the presentation layer 933, service layer 934, and persistence layer 935 are provided together with the conductor 940. Furthermore, the presentation layer 933 includes a web application 942, an Open Data Protocol (oData) Representative State Transfer (REST) Application Programming Interface (API) endpoint 944, and notification and monitoring 946, while the service layer 934 includes API implementation / business logic 948. Thus, as shown in Figure 9, the conductor 940 includes the web application 942, the oData REST API endpoint 944, notification and monitoring 946, and API implementation / business logic 948. The persistence layer 935 includes a database server 950, an AI / ML server 960, and an indexer server 970.
[0119] In some embodiments, communication between the execution agent 914 and the conductor 940 is always initiated by the execution agent 914. In notification scenarios, the execution agent 914 may open a WebSocket channel that the conductor 940 will later use to send commands (e.g., start, stop, etc.) to the RPA robot. Note that, although not shown here to reduce the complexity of Figure 9, the AI agent can also interact with the conductor 940, as discussed above with respect to Figures 1 and 8, for example. The conductor 940 can orchestrate the actions of the AI agent, AOP, and RPA robot.
[0120] In this embodiment, all messages are recorded in the conductor 940, which then processes them via the database server 950, the AI / ML server 960, the indexer server 970, or any combination thereof. As discussed herein and with reference to Figure 8, the executor 912 may be a robotic component.
[0121] In some embodiments, the RPA robot represents an association between a machine name and a username. The robot may manage multiple executors simultaneously. On a computing system that supports the simultaneous execution of multiple interactive sessions (e.g., Windows® Server 2012), multiple RPA robots may run concurrently, each within a separate Windows® session using a unique username.
[0122] The conductor 940 can also facilitate interaction between AI agents and AI / ML models via the AI / ML server 960, which can store and / or facilitate access to generated AI models (e.g., generated AI model 172 in Figure 1).
[0123] In some embodiments, most operations performed by the user on the conductor 940 interface (e.g., via the browser 981) are performed by calling various APIs. Such operations may include, but are not limited to, starting jobs on robots, adding / deleting data in queues, and scheduling jobs to be run unattended, without departing from the scope of the invention. The web application 942 is the visual layer of the server platform. In this embodiment, the web application 942 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other form may be used without departing from the scope of the invention. In this embodiment, the user interacts with the web pages of the web application 942 via the browser 941 and performs various operations to control the conductor 940. For example, the user can create RPA robot groups, assign packages to RPA robots, analyze logs per RPA robot and / or per process, and start and stop RPA robots.
[0124] In addition to the web application 942, the conductor 940 also includes a service layer that exposes an oData REST API endpoint 944. However, other endpoints may also be included without departing from the scope of the present invention. The REST API is utilized in this embodiment by both the web application 942 and the execution agent 914. The execution agent 914 is, in this embodiment, the supervisor of one or more robots on a client computer.
[0125] The REST API in this embodiment covers configuration, logging, monitoring, and queuing functions. Configuration endpoints may, in some embodiments, be used to define and configure application users, authorizations, robots, assets, releases, and environments. Logging REST endpoints may be used to log various information, such as errors, explicit messages sent by robots, and other environment-specific information. Deployment REST endpoints may be used by robots to query the package version to be executed when a start job command is used in conductor 940. Queuing REST endpoints may be responsible for managing queues and queue items, such as adding data to queues, retrieving transactions from queues, and setting the state of transactions.
[0126] A monitoring REST endpoint may monitor the web application 942 and the execution agent 914. The notification and monitoring API 946 may be a REST endpoint used for registering the execution agent 914, delivering configuration settings to the execution agent 914, and sending and receiving notifications between the server and the execution agent 914. In some embodiments, the notification and monitoring API 946 may also use WebSocket communication. As shown in Figure 9, one or more activities / operations described herein are represented by arrows 949.
[0127] In some embodiments, APIs within the service layer 934 may be accessed via the configuration of an appropriate API access path, for example, based on whether the conductor 940 and the overall hyperautomation system are deployed on-premises or cloud-based. The API for the conductor 940 may provide custom methods for querying statistics about various entities registered with the conductor 940. Each logical resource may, in some embodiments, be an oData entity. In such entities, components such as robots, processes, and queues may have attributes, relationships, and behaviors. In some embodiments, the API for the conductor 940 may be utilized by a web application 942 and / or execution agent 914 in two ways: (1) a way to obtain API access information from the conductor 940, or (2) a way to register an external application and have it use the oAuth flow.
[0128] In this embodiment, the persistence layer 935 includes three servers: a database server 950 (e.g., an SQL server), an AI / ML server 960 (e.g., a server providing AI / ML model serving services such as AI Center functions), and an indexer server 970. In this embodiment, the database server 950 stores the configuration of robots and AI agents, groups of robots and AI agents, AOP, associated processes, users, roles, schedules, etc. In some embodiments, this information is managed via a web application 942. The database server 950 may manage queues and queue entries. In some embodiments, the database server 950 may store messages recorded by robots and AI agents (in addition to or instead of the indexer server 970). The database server 950 may also store process mining, task mining, and / or task capture-related data received, for example, from a listener 930 installed on the client side. Although no arrow is shown between listener 930 and database 950, it should be understood that in some embodiments, listener 930 can communicate with database 950 and vice versa. This data may be stored in the form of PDDs, images, XAML files, etc. Note that structured and / or unstructured data may be stored. Listener 930 may be configured to intercept user activity, processes, tasks, and performance metrics on each computing system in which listener 930 resides. For example, listener 930 may record user activity on each computing system (e.g., clicks, typed characters, position, applications, active elements, time, etc.) and then convert it into a format suitable for being provided to and stored in database server 950.
[0129] The AI / ML Server 960 facilitates the integration of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options make such functionality accessible even to non-data scientists. Deployed automation (e.g., RPA robots and / or AI agents) can invoke AI / ML models from the AI / ML Server 960. AI / ML model performance can be monitored and trained and improved using user-validated data. The AI / ML Server 960 can schedule and execute training jobs to train new versions of AI / ML models. The AI / ML model server can also store and / or access generated AI models.
[0130] The AI / ML server 960 may store data relating to AI / ML models and ML packages for configuring various ML skills for users during development. ML skills as used herein refer to, for example, pre-built and trained ML models for a given process that may be used by automation. The AI / ML server 960 may also store data relating to document understanding technologies and frameworks, algorithms, and software packages for various AI / ML capabilities, including but not limited to intent analysis, NLP, speech analysis, and different types of AI / ML models.
[0131] In some embodiments, an optional indexer server 970 stores and indexes information recorded by the robot. In certain embodiments, the indexer server 970 can be disabled via configuration settings. In some embodiments, the indexer server 970 uses ElasticSearch®, an open-source full-text search engine. Messages recorded by the robot (e.g., using log messages or activities such as line writing) can be sent to the indexer server 970 via a logging REST endpoint, where they are indexed for future use.
[0132] Figure 10 is an architectural diagram showing the relationships between Designer 1010, Activities 1020, 1030, 1040, 1050, Driver 1060, API 1070, and AI / ML Model 1080 in one or more embodiments. As described above, an AI agent, AOP, or developer uses Designer 1010 to develop workflows for automation (e.g., workflows executed by RPA robots, AI agents, and AOPs).
[0133] The AI agent, AOP, or developer may design and configure workflow 1092, design and configure agent-type automation 1094 for the AI agent (e.g., providing natural language descriptions, context grounding, tools, etc., for the AI agent), and design and configure AOP 1096 (see also Figures 4B, 5, and 6). Various types of activities may be displayed to the developer in some embodiments. The designer 1010 may be local to the user's computing system or remote to it (e.g., accessed via a VM, or via a local web browser interacting with a remote web server). The workflow for the RPA robot may include user-defined activity 1020, API-driven activity 1030, AI / ML activity 1040, and / or UI automation activity 1050. The user-defined activity 1020 and API-driven activity 1040 interact with the application via their APIs. User-defined activity 1020 and / or AI / ML activity 1040 may, in some embodiments, call one or more AI / ML models 1080, which may reside locally with respect to the computing system on which the robot is operating, and / or remotely with respect to it.
[0134] Some embodiments can identify non-textual visual elements within an image, which are referred to herein as CV. However, it should be noted that in some embodiments, CV incorporates OCR. CV may be performed, at least in part, by an AI / ML model 1080. Some CV activities relating to such elements may include, but are not limited to, text extraction from segmented label data using OCR, fuzzy text matching, segmented label data using ML, and comparison of extracted text in the label data with ground truth data. In some embodiments, the number of activities that can be implemented in user-defined activity 1020 may be in the hundreds, or even thousands. However, any number and / or types of activities may be used without departing from the scope of the invention.
[0135] The UI automation activity 1050 is a set of special low-level activities written in low-level code and designed to facilitate interaction with a screen. The UI automation activity 1050 facilitates these interactions via drivers 1060, which enable the robot to interact with desired software. For example, drivers 1060 may include an operating system (OS) driver 1062, a browser driver 1064, a VM driver 1066, an enterprise application driver 1068, and so on. In some embodiments, the UI automation activity 1050 may utilize one or more AI / ML models 1080, thereby performing interactions with a computing system. In certain embodiments, the AI / ML models 1080 may complement or completely replace the drivers 1060. In fact, in certain embodiments, drivers 1060 are not included.
[0136] Driver 1060 can interact with the OS at a low level, such as by exploring hooks and monitoring keys, via OS driver 1062. Driver 1060 can facilitate integration with applications such as Chrome®, IE®, Citrix®, and SAP®. For example, a "click" activity can perform the same role in these different applications via driver 1060.
[0137] Figure 11 is an architectural diagram showing a computing system 1100 in one or more embodiments. In some embodiments, the computing system 1100 may be one or more computing systems illustrated and / or described herein. In some embodiments, the computing system 1100 may be one or more computing systems illustrated and / or described herein. In certain embodiments, the computing system 1100 may be part of a hyperautomation system as shown in Figures 1 and 8. The computing system 1100 includes a bus 1105 or other communication mechanism for communicating information and a processor(s) 1110 coupled to the bus 1105 for processing information. The processor(s) 1110 may be any kind of general-purpose or application-specific processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor(s) 1110 may also have multiple processing cores, of which at least some may be configured to perform specific functions. In some embodiments, multi-parallel processing may be used. In certain embodiments, at least one of the processors 1110 may be a neuromorphic circuit containing processing elements that mimic biological neurons. In some embodiments, the neuromorphic circuit may not require typical components of the Von Neumann computing architecture.
[0138] The computing system 1100 further includes memory 1115 for storing information and instructions to be executed by processor(s) 1110. Memory 1115 may consist of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage devices such as magnetic disks or optical disks, or any other type of non-temporary computer-readable medium, or a combination thereof. The non-temporary computer-readable medium may be any available medium accessible by processor(s) 1110, and may include volatile media, non-volatile media, or both. The medium may be removable, non-removable, or both. The computing system 1100 also includes communication devices 1120, such as transceivers, that provide access to a communication network via wireless and / or wired connections. In some embodiments, the communication device 1120 may include one or more antennas, which are single antennas, array antennas, phased antennas, switching antennas, beamforming antennas, beam steering antennas, combinations thereof, and / or any other antenna configurations that do not depart from the scope of the present invention.
[0139] The processor(s) 1110 are further coupled to the display 1125 via the bus 1105. Any suitable display device and haptic I / O may be used without departing from the scope of the present invention. A keyboard 1130 and a cursor control device 1135 such as a computer mouse or touchpad are further coupled to the bus 1105, thereby enabling the user to interface with the computing system 1100. However, in certain embodiments, a physical keyboard and mouse may not be present, and the user may interact with the device only via the display 1125 and / or touchpad (not shown). Any type and combination of input devices may be used as a matter of design selection. In certain embodiments, there may be no physical input device and / or display. For example, the user may interact with the computing system 1100 remotely via another computing system to which it is communicated, or the computing system 1100 may operate autonomously.
[0140] Memory 1115 stores software modules that provide functionality when executed by processor(s) 1110. These modules include an operating system 1140 for the computing system 1100.
[0141] The module further includes a model selection module 1145 configured to perform all or part of the processes described herein, or a derivative thereof. For example, the model selection module 1145 may perform agent-based automation with agent model selection in agent-based automation that includes context grounding for multiple model capabilities and benchmarking used to select a specific model for each step of the workflow that requires an AI model. The model selection module 1145 may include context grounding for selecting a specific model for each step of the workflow performed in multimodal agent-based automation. Furthermore, the model selection module 1145 may include using benchmarking and historical job data in line with the context grounding to select a model for the steps of the workflow. The model selection module 1145 may select the optimal model for each step that requires an AI model. Prior to the optimal model selection, the model selection module 1145 may perform self-benchmarking, access an artificial intelligence (AI) center, and run within a sandbox. Therefore, the model selection module 1145 provides improved control over the multimodal agent-type automation to achieve technical effects, benefits, and advantages such as improved workflow processing, increased efficiency of step execution, and prevention of hallucination due to incorrectly selected AI models. Thus, the model selection module 1145 generates a more intelligent and context-aware hyperautomation system.
[0142] The module further includes an agent memory module 1147 configured to perform all or part of the processes described herein, or derivatives thereof, including, but not limited to, performing agent memory for storing context grounding and other results for automation. For example, the agent memory module 1147 is configured to dynamically cache (i.e., store) escalations, tool calls, user interactions or feedback, context grounding, etc., to provide improved efficiency and minimize calls.
[0143] The computing system 1100 may include one or more additional function modules 1150 that include additional functions. For example, the one or more additional function modules 1150 may include an extract and search module configured to perform advanced agent-based extract and search for context grounding within the automation.
[0144] Those skilled in the art will understand, without departing from the scope of the invention, that “computing system” can be implemented as a server, embedded computing system, personal computer, console, personal digital assistant (PDA), mobile phone, tablet computing device, smartwatch, quantum computing system, or any other suitable computing device, or a combination thereof. The fact that the above functions are presented as being performed by a “system” is not intended to limit the scope of the invention in any sense, but rather to provide an example of a number of embodiments of the invention. In fact, the methods, systems, and apparatus disclosed herein can be implemented in localized and distributed forms that are consistent with computing technologies, including cloud computing systems. The computing system may be part of, or accessible by, a LAN, mobile communication network, satellite communication network, internet, public or private cloud, hybrid cloud, server farm, or any combination thereof. Any localized or distributed architecture may be used without departing from the scope of the invention.
[0145] It should be noted that some of the system features described herein are presented as modules to more specifically emphasize their implementation independence. For example, modules may be implemented as hardware circuits including custom very large-scale integrated (VLSI) circuits or gate arrays, or commercially available semiconductors (e.g., logic chips, transistors, or other discrete components). Modules may also be implemented as programmable hardware devices such as field-programmable gate arrays, programmable array logic, programmable logic devices, and graphics processing units.
[0146] Modules can also be implemented, at least partially, in software for execution on various types of processors. For example, a specified unit of executable code may include one or more physical or logical computer instruction blocks, which may be organized as, for example, objects, procedures, or functions. However, the executable code of a specified module does not need to reside in a single physical location, and may include distributed instructions stored in different locations that, when logically combined, constitute the module and achieve the purposes described for that module. Furthermore, modules may be stored on a computer-readable medium, which may be, for example, a hard disk drive, a flash device, RAM, tape, and / or any other such non-temporary computer-readable medium used to store data without departing from the scope of the present invention.
[0147] In fact, a module of executable code may be a single instruction, a number of instructions, or even distributed across several different code segments, different programs, and multiple memory devices. Similarly, operational data, while identified and illustrated herein within a module, may be embodied in any suitable form and organized within any suitable type of data structure. Such operational data may be collected as a single data set, or distributed in different locations, including across different storage devices, and may exist at least partially only as electronic signals on a system or network.
[0148] Without departing from the scope of the present invention, various types of AI / ML models can be trained and deployed. For example, Figure 12A shows an example of a neural network 1200, according to one or more embodiments, trained to receive inputs (indicated in column 1210) to input "neurons" 1-I in an input layer (indicated in column 1220). The neural network 1200 includes several hidden layers (indicated in columns 1230 and 1240). Both deep learning neural networks (DLNNs) and shallow learning neural networks (SLNNs) typically have multiple layers, although SLNNs may have only one or two layers, and usually fewer than DLNNs. Typically, a neural network architecture includes an input layer, several intermediate layers (e.g., hidden layers), and an output layer (indicated in column 1250), and the neural network 1200 is an example of such an architecture.
[0149] DLNNs often have a large number of layers (e.g., 10, 50, 200, etc.), and subsequent layers typically reuse features from previous layers to compute more complex and general functions. SLNNs, on the other hand, tend to have fewer layers and are trained relatively quickly because specialized features are pre-generated from raw data samples. However, feature extraction is labor-intensive. DLNNs, on the other hand, typically do not require specialized features, but training takes longer and they tend to have more layers.
[0150] In both approaches, layers are trained simultaneously on the training set and overfitting is typically checked on a separate cross-validation set. Both techniques can yield excellent results, and there is considerable interest in both approaches. The optimal size, shape, and number of individual layers vary depending on the problem addressed by each neural network.
[0151] Returning to Figure 12A, the input layer provides enterprise-specific information, including diverse industries and applications, unique industry terminology, complex document structures, and pre-trained knowledge, which are then supplied as inputs to J neurons in Hidden Layer 1. Various other inputs are also possible, including, but are not limited to, computing system state information, published automations, business rules, information about which workflows and / or tasks relate to what, initial automation definitions, and process automation documents. In this example, all of these inputs are fed to each neuron, but various architectures may be used individually or in combination, including but not limited to feedforward networks, radial basis networks, deep feedforward networks, deep convolutional inverse graphics networks, convolutional neural networks, recurrent neural networks, artificial neural networks, long short-term memory networks, gated recurrent unit networks, generative adversarial networks, liquid state machines, autoencoders, variational autoencoders, denoising autoencoders, sparse autoencoders, extreme learning machines, echo state networks, Markov chains, Hopfield networks, Boltzmann machines, restricted Boltzmann machines, deep residual networks, Kohonen networks, deep belief networks, deep convolutional networks, support vector machines, neural Turing machines, or any other suitable type or combination of neural networks that do not depart from the scope of the present invention.
[0152] Hidden layer 2 (1240) receives input from hidden layer 1 (1230), hidden layer 3 receives input from hidden layer 2 (1240), and so on for all hidden layers up to the last hidden layer (indicated by the ellipsis 1255), the last hidden layer providing its output as input to the output layer. Although multiple proposals are shown as outputs here, in some embodiments only a single output proposal is provided. In certain embodiments, the proposal is ranked based on a confidence score. In this embodiment, the output is an accurate response to a professional or recent query.
[0153] It should be noted that the number of neurons I, J, K, and L are not necessarily equal. Therefore, for a given layer of the neural network 1200, any desired number of layers can be used without departing from the scope of the invention. In fact, in certain embodiments, the types of neurons in a given layer do not all have to be the same.
[0154] The neural network 1200 is trained to assign confidence scores to appropriate outputs. To reduce inaccurate predictions, in some embodiments, only results with confidence scores that meet or exceed a confidence threshold may be provided. For example, if the confidence threshold is 80%, outputs with confidence scores exceeding this may be used, and the rest may be ignored. According to one or more embodiments, the output layer 1250 indicates that two text fields (shown in outputs 1261 and 1262), a text label (shown in output 1263), and a submit button (shown in output 1265) have been found. Without departing from the scope of one or more embodiments herein, the neural network 1200 may provide the location, dimensions, images, and / or confidence scores of these elements, which can then be used by an RPA robot or other automation that uses this output for a given purpose.
[0155] Neural networks are typically probabilistic constructs that have confidence scores. These scores can be learned by an AI / ML model based on how frequently similar inputs were correctly identified during training. Common types of confidence scores include decimal numbers from 0 to 1 (which can also be interpreted as confidence percentages), numbers from negative infinity to positive infinity, and a set of representations (e.g., "low," "medium," and "high"). Various post-processing calibration techniques, such as temperature scaling, batch normalization, weight decay, and negative log-likelihood (NLL), may also be employed in an attempt to obtain more accurate confidence scores.
[0156] In neural networks, "neurons" are typically algorithmically implemented as mathematical functions based on the functions of biological neurons. Neurons receive weighted inputs and have summation and activation functions that govern whether they pass their output to the next layer. These activation functions can be nonlinear threshold activity functions (i.e., normalized linear unit (ReLU) nonlinearity) that do nothing when the value is below a threshold, but respond linearly when the threshold is exceeded. Deep learning uses summation and ReLU functions because real-world neurons can have roughly similar activity functions. Through linear transformations, information can be subtracted, added, etc. Essentially, neurons act as gate functions that pass their output to the next layer, governed by their underlying mathematical functions. In some embodiments, different functions may be used for at least some neurons.
[0157] An example of neuron 1295 is shown in Figure 12B. Inputs from the previous layer x1, x2, ..., x n Each of these has weights w1, w2, ..., w n This is assigned. Therefore, the total input from preceding neuron 1 is w1x1. These weighted inputs are used for the bias-corrected summation function of the neurons, for example, as shown in the following equation.
number
[0158] This sum is compared to the activation function f(x) to determine whether a neuron "fires". For example, f(x) can be given by the following equation:
number
[0159] Therefore, the output y of neuron 1295 can be given by the following equation.
number
[0160] In this case, neuron 1295 is a single-layer perceptron. However, any suitable neuron type or combination of neuron types may be used without departing from the scope of the present invention. It should also be noted that the range of values of the output values(s) of the weights and / or activation function may differ in some embodiments without departing from the scope of the present invention.
[0161] Often, a goal, or "reward function," is used. The reward function guides the exploration of the state space and the search for intermediate transitions and steps with both short-term and long-term rewards in an attempt to achieve the goal (e.g., finding the most accurate answer to a user query based on relevant metrics). During training, various labeled data are fed through the neural network 1200. Successful classifications strengthen the weights of the input to the neurons, while failed classifications weaken them. A cost function, such as mean squared error (MSE) or gradient descent, may be used to punish slightly incorrect predictions much less than very incorrect ones. If the performance of the AI / ML model does not improve after a certain number of training iterations, data scientists may modify the reward function and provide corrections for inaccurate predictions, etc.
[0162] Backpropagation is a technique for optimizing synaptic weights in feedforward neural networks. Backpropagation can be used to "open the hood" of the hidden layers of a neural network, see how much of the loss each node is responsible for, and then update the weights in a way that minimizes the loss, assigning lower weights to nodes with high error rates and vice versa. In other words, backpropagation allows data scientists to iteratively adjust the weights to minimize the difference between the actual output and the desired output.
[0163] The backpropagation algorithm is mathematically based on optimization theory. In supervised learning, training data with known outputs passes through a neural network, an error is calculated from the known target output using a cost function, which provides the error for backpropagation. The error is calculated at the output, and this error is converted into corrections to the network weights that minimize the error.
[0164] An example of backpropagation for the case of supervised learning is shown below. The column vector input x is processed through a series of N nonlinear activation functions f between each layer i=1,…,N of the network i , and the output at a given layer is first multiplied by the synaptic matrix W i and added with the bias vector b i . The network output o is given by the following equation. [Formula]
[0165] In some embodiments, o is compared with the target output t, resulting in an error [Formula] , which is desired to be minimized.
[0166] Optimization in the form of a gradient descent procedure can be used to minimize the error by modifying the synaptic weights W for each layer. i The gradient descent procedure requires calculating the output o given the input x corresponding to the known target output t, and generates the error o--t. This total error is then propagated backward to provide the local error for weight update, the calculation of which is similar to, but not identical to, that used for forward propagation. In particular, the backpropagation step typically requires an activation function of the form p j (n j )=f’ j (n j ), where nj This is the network activity in layer j (i.e., n j =W j o j-1 +b j ) and o j =f j (n j ) and the apostrophe ' represents the derivative of the activity function f.
[0167] Weight updates can be calculated using the following formula:
number
[0168] Here, o represents the Hadamard product (i.e., the element-wise product of two vectors), T represents the matrix transpose, and o j is f j (W j o j-1 +b j ) represents and o0=x. Here, the learning rate η is selected with respect to machine learning considerations. Below, η is associated with the neural Hebb learning mechanism used in the neural implementation. Note that synapses W and b can be joined to a single large synaptic matrix. Here, it is assumed that an additional column representing the b synapse, which has an appended input vector 1, is included in W.
[0169] The AI / ML model can be trained over multiple epochs until it reaches a good level of accuracy (e.g., 97% or higher after approximately 2000 epochs, using an F2 or F4 threshold for detection). This level of accuracy may, in some embodiments, be determined using an F1 score, F2 score, F4 score, or any other suitable technique that does not deviate from the scope of the invention. After being trained on training data, the AI / ML model can be tested on a set of evaluation data that it has not encountered before. This helps to ensure that the AI / ML model is not in a state of "overfitting," where it performs well on the training data but not on other data.
[0170] In some embodiments, the level of accuracy that an AI / ML model can achieve may be unknown. Therefore, if the accuracy of the AI / ML model begins to decline when analyzing evaluation data (i.e., the model performs well on the training data but begins to decline on the evaluation data), the AI / ML model may undergo more training epochs on the training data (and / or new training data). In some embodiments, an AI / ML model is deployed only when its accuracy reaches a certain level, or when the accuracy of the trained AI / ML model is better than that of an existing deployed AI / ML model. In certain embodiments, a set of trained AI / ML models may be used to accomplish a task. For example, one AI / ML model may be trained to recognize images, another to recognize text, and yet another to recognize semantic and / or ontology associations.
[0171] It should be noted that, in addition to, or instead of, neural networks, some embodiments may use transformer networks such as SentenceTransformers™, a state-of-the-art Python™ framework for sentence, text, and image embedding. Such transformer networks learn the associations of words and phrases that have both high and low scores. This trains the AI / ML model to determine which are close to the input and which are not. Transformer networks may also use field lengths and field types, rather than simply using word / phrase pairs.
[0172] As described above, NLP models such as word2vec, BERT, GPT-3, ChatGPT, and other LLMs may be used in some embodiments to facilitate semantic understanding and provide more accurate and human-like answers. Other techniques, such as clustering algorithms, may be used to find similarities between groups of elements. Clustering algorithms may include, but are not limited to, density-based, distribution-based, centroid-based, and hierarchy-based algorithms. Examples of K-means clustering algorithms, DBSCAN clustering algorithms, Gaussian mixture model (GMM) algorithms, and Bircean Equilibrium Iterative Shrinking Clustering (BIRCH) algorithms may be used. Such techniques may also be useful for categorization.
[0173] Figure 13 is an architecture diagram showing a reference architecture 1300 of a generative AI model according to one or more embodiments. The architecture consists of multiple layers, namely, an API plugin, a prompt library, a vector data source ingestion layer, access processing control, a model training pipeline, an evaluation layer for evaluating hallucination / telemetry / evaluation, a BYOM embedding layer, and an LLM orchestration layer. There are also search and retrieve plugins, access control plugins, and API plugins for integration into enterprise systems.
[0174] Reference Architecture 1300 has three main flows.
[0175] Flow 1: Ingestion may include data ingestion and training flows. For example, data is read from multiple data stores, preprocessed, chunked, and trained through an embedded model (e.g., RAG and training pipeline, i.e., fine-tuning). A vector database stores chunked document embeddings (e.g., vector embeddings), enabling better semantic and similarity-based data retrieval.
[0176] Flow 2: Search and retrieve may include prompt extension using a data search and retrieve flow. For example, when a user query reaches the API layer, a prompt is selected, followed by a data search and retrieve via a vector database or API plugin to obtain appropriate contextual data before the prompt is passed to the LLM layer.
[0177] Flow 3: Inference may include an LLM inference flow. For example, here, a choice is made whether to use a general-purpose foundational model or a self-hosted foundational model. Fine-tuned models may be used if they are tailored to a specific task or use case. The response is evaluated for accuracy, including hallucination, and other metrics.
[0178] It should be noted that in some embodiments, generative AI models having multiple "heads" may be used. A head refers to the output layer of the generative AI model. Generative AI models such as generative AI model 172 in Figure 1 typically have a sequence of multiple layers, and each head often shares the first few layers of the model before branching into its own unique layers.
[0179] Figure 14 is a flowchart of a process 1400 for training an AI / ML model (or more) according to one or more embodiments. In some embodiments, the AI / ML model (or more) may be a generative AI model as described above. In the case of a neural network, the architecture typically includes multiple neuron layers, including an input layer, an output layer, and hidden layers. See, for example, Figures 12A and 12B. The hidden layers in between generate an intermediate representation of the input, which is used to process the input data and generate the output. These hidden layers may include various types of neurons, such as convolutional neurons, recurrent neurons, and / or transformer neurons. A generative AI model may also have various layers.
[0180] The process 1400 performed in Figure 14 may be executed by automation as described herein, implemented in a computer program according to one or more embodiments. The computer program may be implemented on a non-temporary computer-readable medium. The computer-readable medium may include, but is not limited to, a hard disk drive, a flash device, RAM, tape, and / or any other such medium or combination of mediums used to store data. The computer program may include coded instructions for controlling the processor(s) of a computing system (see, for example, Figure 11) to implement all or part of the process 1400 shown in Figure 14, and these may also be stored on the computer-readable medium.
[0181] In some embodiments, the training process begins in block 1410 with the provision of data, labeled or unlabeled, such as diverse industry and application data, specific industry terminology, complex document structures, and company-specific information accompanied by pre-trained knowledge. For generative AI models that are often trained in a general manner, the training process may be omitted unless a fine-tuned model is desired, as will be discussed in more detail later. The AI / ML model is then trained over multiple epochs in block 1420, and the results are reviewed in block 1430. While various types of AI / ML models can be used, LLMs and other generative AI models are typically trained (fine-tuned) using a process called "supervised learning," which is also discussed above. Supervised learning involves providing the model with a large dataset, which the model uses to learn the relationships between inputs and outputs. During the training process, the model adjusts the weights and biases of neurons in the neural network to minimize the difference between the predicted output and the actual output in the training dataset.
[0182] One aspect of the model in some embodiments is the use of transfer learning. For example, transfer learning can utilize a pre-trained model such as ChatGPT and fine-tune it for a specific task or domain in block 1420. This allows the model to leverage knowledge already learned from the pre-training stage and adapt it to a specific application through the training stage in block 1420.
[0183] The pre-training phase involves training the model on an initial training data set, which may be more general. In this phase, the model learns the relationships within the data. In the fine-tuning phase (performed in some embodiments, in addition to or instead of the initial training phase, during block 1420, for example, when the pre-trained model is used as the initial foundation for the final model), the pre-trained model is adapted to a specific task or domain by training the model on a smaller dataset specific to that task. For example, in some embodiments, the model may focus on a specific type(s) of data source. This can help the model identify data elements within it more accurately than a generative AI model that is pre-trained only. Fine-tuning allows the model to learn the nuances of the source, such as specific vocabulary and syntax, specific graphical properties, specific data formats, etc., without requiring as much data as would be needed to train the model from scratch. By leveraging the knowledge learned in the pre-training phase, the fine-tuned model can achieve state-of-the-art performance on a specific task with relatively little additional training data.
[0184] In some embodiments, if the AI / ML model does not meet the desired confidence threshold in decision block 1440, process 1400 proceeds to block 1450 (as indicated by the NO arrow). Training data is supplemented and / or the reward function is modified to help the AI / ML model better achieve its goal, and in block 1450, process 1400 returns to block 1420.
[0185] If the AI / ML model meets the confidence threshold in decision block 1440, process 1400 proceeds to block 1460 (as indicated by the YES arrow). In block 1460, the AI / ML model is tested on evaluation data to ensure that the AI / ML model has generalized well and has not overfitted with respect to the training data. The evaluation data includes information that the AI / ML model has not previously processed.
[0186] If the confidence threshold is met in decision block 1470 for the evaluation data, process 1400 proceeds to block 1480 (as indicated by the YES arrow). The AI / ML model is deployed in 1480. Otherwise, process 1400 returns to block 1450 (as indicated by the NO arrow), and the AI / ML model is further trained.
[0187] Next, moving to Figure 15, a process 1500 according to one or more embodiments is shown. Generally, process 1500 is a method for agent model selection in a hyperautomation system that performs one or more agent-type automations. Process 1500 may be performed by the automation described herein, implemented in a computer program according to one or more embodiments. The computer program may be embodied on a non-temporary computer-readable medium (e.g., a memory that stores computer code for the one or more agent-type automations and computer code for selecting from a plurality of available models). The computer program may include coded instructions for controlling processors (or multiple processors) of a computing system (e.g., see Figure 11) (e.g., at least one processor that executes computer code to improve agent model selection) in order to implement all or part of process 1500 as shown in Figure 15, and these may also be stored on the computer-readable medium.
[0188] Process 1500 begins in block 1520, where a hyperautomation system analyzes the workflow. The workflow may include two or more steps, at least one of which is an activity that requires an AI model (e.g., an LLM or other AI model). Thus, the hyperautomation system analyzes the workflow to identify each step of the workflow that requires an AI model. More specifically, the hyperautomation system analyzes the workflow to identify each step of the workflow that requires a model as one or more model-required steps.
[0189] In block 1540, the hyperautomation system automatically selects one or more AI models from a plurality of available models for each of the one or more model-required steps. According to one or more technical effects, benefits, and advantages, the one or more AI models differ from conventional automation which requires extensive feature engineering (e.g., resource-intensive). This is because the one or more models are trained in a more generalized manner to adapt to various tasks with minimal fine-tuning (e.g., resource-efficient). The hyperautomation system may select one or more AI models based on one or more model capabilities, one or more model performance, one or more model applications, or one or more of their benchmarking capabilities.
[0190] For example, the one or more model capabilities include data processing and analysis, forecasting and prediction, decision-making and automation, and content generation. Data processing and analysis may include, but are not limited to, pattern recognition (e.g., an AI model identifying patterns and relationships in data to enable informed decision-making), data interpretation (e.g., an AI model interpreting complex datasets to extract significant insights and trends that humans might miss), data cleansing and preprocessing (e.g., an AI model automating tasks and ensuring data quality and accuracy), and data management (e.g., an AI model managing large datasets for access and analysis beyond human capabilities). Forecasting and prediction may include, but are not limited to, predictive modeling (e.g., an AI model predicting future outcomes based on historical data to enable agents to make informed decisions) and time series analysis (e.g., an AI model analyzing time-based data to identify trends and patterns to facilitate forecasting and planning). Decision-making and automation may include, but are not limited to, automated decision-making (e.g., AI models automating tasks and decision-making processes to improve efficiency and reduce human error) and robotics and control systems (e.g., AI models controlling and automating automation systems to perform complex tasks). Content generation may include, but are not limited to, image generation (e.g., AI models generating images and artwork), text generation (e.g., AI models generating human-like text including articles, stories, and code), and code generation (e.g., AI models generating code snippets and / or programs).
[0191] Furthermore, the performance of one or more models may include the resource efficiency of the hyperautomation system (e.g., CPU and RAM usage). The performance of the models may also include output accuracy (e.g., the ability of LLM output on unknown data) and / or the ability to correct the output.
[0192] The one or more model applications of the one or more AI models include a set of real-world use cases in which the one or more AI models are put into operation. For example, the one or more model applications may include natural language processing, chatbot services, customer service, machine translation, sentiment analysis, content generation, computer vision, object detection, machine learning, error detection, and predictive analytics.
[0193] The benchmarking capability of the one or more AI models includes systematic testing and performance evaluation across tasks and datasets. Furthermore, the benchmarking capability may provide a standardized method for comparing the strengths and requirements of the one or more AI models. For example, the benchmarking capability may include time constraints for the user to try automation (e.g., more options vs. more resource / time savings) to eliminate the debugging process for each iteration.
[0194] Furthermore, in subblock 1545, the hyperautomation system may perform agent selection for each step of the workflow (regardless of whether an AI model is required). In this regard, the hyperautomation system may select one or more AI agents using multiple model capabilities and benchmarking context. The hyperautomation system may select a first AI agent for the aforementioned step, a second AI agent for AI model selection, and utilize the second AI agent to select an AI model for the first AI agent. The hyperautomation system may make the selection of the first AI agent, the second AI agent, and the AI model based on benchmarking and historical job data, as well as available hardware and memory, browser or application versions, etc. Note that the historical data is used to make decisions by the hyperautomation system and its internal AI agents. That is, if benchmarking and information are available to the first and second AI agents, the first and second AI agents may perform benchmarking themselves and provide access in the AI Center or Sandbox. As indicated by arrow 1550, the agent model selection includes selecting the optimal AI model from multiple available models for each of the one or more model-required steps, based on the one or more benchmarking and historical job data. That is, if either the AI agent or model selection fails to meet the benchmarking requirements, a new AI agent or model selection may be made.
[0195] In block 1560, the hyperautomation system assigns the one or more artificial intelligence models to each of the one or more model-required steps. The hyperautomation system can automatically assign the one or more artificial intelligence models to generate a multimodal agent-type automation for the workflow. The multimodal agent-type automation is a probabilistic automation performed by one or more AI agents using multiple models. Furthermore, the multimodal agent-type automation ensures that multiple AI agents and AI models efficiently perform a variety of different task ranges. The multimodal agent-type automation implements context grounding and benchmarking based on the AI model selection for each step of the workflow.
[0196] In block 1580, the hyperautomation system implements a workflow by performing multimodal agent-based automation. The multimodal agent-based automation includes a probabilistic automation of the workflow for implementing context grounding and benchmarking during agent model selection. According to one or more embodiments, the multimodal agent-based automation includes a probabilistic automation of the workflow using the one or more AI agents and the one or more AI models. According to one or more embodiments, the one or more agent-based automations include artificial intelligence-enhanced probabilistic automation that operates independently, dynamically, and adaptively to perform the multimodal agent-based automation. Thus, one or more technical effects, benefits, and advantages of agent model selection for individual workflow steps by the hyperautomation system are the utilization of benchmarking and historical data that are not otherwise used in conventional automation.
[0197] The computer program may be implemented in hardware, software, or a hybrid implementation. The computer program may consist of modules designed to communicate with each other in an operable manner and to pass information or instructions to a display. The computer program may be configured to run on a general-purpose computer, an ASIC, or any other suitable device.
[0198] It will be readily apparent that the components of various embodiments of the present invention can be arranged and designed in a wide variety of different configurations, as generally described herein and shown in the drawings. Therefore, the detailed description of embodiments of the present invention shown in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the present invention.
[0199] The features, structures, or characteristics of the invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, any reference throughout this specification to “a particular embodiment,” “some embodiments,” or similar phrases means that the particular features, structures, or characteristics described in relation to that embodiment are included in at least one embodiment of the invention. Therefore, the phrases “a particular embodiment,” “some embodiments,” “other embodiments,” or similar phrases appearing throughout this specification do not necessarily all refer to the same group of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0200] It should be noted that references to features, advantages, or similar terms throughout this specification do not imply that all features and advantages that may be realized by the present invention should or are provided in any single embodiment of the invention. Rather, terms referring to features and advantages should be understood to mean that certain features, advantages, or characteristics described in relation to a particular embodiment are included in at least one embodiment of the invention. Accordingly, discussions of features and advantages, as well as similar terms, throughout this specification may, but not necessarily, refer to the same embodiment.
[0201] Furthermore, the features, advantages, and characteristics of the invention described may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the invention may be carried out without one or more specific features or advantages of a particular embodiment. In other cases, additional features and advantages not present in all embodiments of the invention may be recognized in a particular embodiment.
[0202] Those skilled in the art will understand that the inventions discussed above can be readily implemented in a different sequence of steps and / or with hardware elements of a different configuration than those disclosed. Therefore, although the present invention is described based on these preferred embodiments, it will be obvious to those skilled in the art that certain modifications, changes, and alternative configurations are apparent while remaining within the spirit and scope of the invention. Accordingly, to determine the boundaries of the invention, one should refer to the appended claims.
Claims
1. A method for selecting agent models for a hyperautomation system that performs one or more agent-based automations, The hyperautomation system includes a memory for storing computer code for one or more agent-type automations and computer code for selecting from a plurality of available models. The hyperautomation system includes at least one processor that executes the computer code to improve agent model selection, The aforementioned method, Analyze the workflow and identify one or more model-required steps for each step of the workflow that requires a model, Based on one or more model capabilities, one or more model performance, one or more model applications, or one or more benchmarking capabilities of the one or more artificial intelligence models, one or more artificial intelligence models are automatically selected from the available multiple models for each of the one or more model required steps. To generate a multimodal agent-type automation for the workflow, assign the one or more artificial intelligence models to each of the one or more model-required steps, In order to implement the aforementioned workflow, the multimodal agent-type automation is executed. Methods that include...
2. The method according to claim 1, wherein the workflow comprises two or more steps, at least one of which is an activity that requires the AI model.
3. The method according to claim 1, wherein the model capabilities of the one or more AI models include data processing and analysis, prediction and forecasting, decision-making and automation, and content generation.
4. The method according to claim 1, wherein the model performance of the one or more AI models includes the resource efficiency of the hyperautomation system, or the output accuracy and the ability to correct the output.
5. The method according to claim 1, wherein the one or more model applications of the one or more AI models include a group of real-world use cases in which the one or more AI models are put into operation.
6. The method according to claim 1, wherein the benchmarking capability of the one or more AI models comprises systematic testing and performance evaluation across tasks and datasets.
7. The method according to claim 1, wherein the multimodal agent-type automation includes probabilistic automation of the workflow for implementing context grounding and benchmarking during agent model selection.
8. The method according to claim 1, wherein the multimodal agent type automation includes probabilistic automation of the workflow using one or more artificial intelligence agents and one or more artificial intelligence models.
9. The method according to claim 1, wherein the agent model selection includes selecting the optimal artificial intelligence model from the available plurality of models for each of the one or more model required steps, based on the one or more benchmarking and historical job data.
10. The method according to claim 1, wherein the one or more agent-type automation includes artificial intelligence-enhanced probabilistic automation, which is performed by one or more AI agents that operate independently, dynamically, and adaptively in order to perform the multimodal agent-type automation.
11. A memory for storing computer code for executing one or more agent-type automations, and computer code for selecting from a plurality of models available by the hyperautomation system, To improve agent model selection for the multiple available models, the hyperautomation system is configured to perform the following: at least one processor that executes the computer code and A hyper-automation system including, The execution described below is: Analyze the workflow and identify one or more model-required steps for each step of the workflow that requires a model, Based on one or more model capabilities, one or more model performance, one or more model applications, or one or more benchmarking capabilities of the one or more artificial intelligence models, one or more artificial intelligence models are automatically selected from the available multiple models for each of the one or more model required steps. To generate a multimodal agent-type automation for the workflow, assign the one or more artificial intelligence models to each of the one or more model-required steps, In order to implement the aforementioned workflow, the multimodal agent-type automation is executed. A hyper-automation system that includes this.
12. The hyperautomation system according to claim 11, wherein the workflow comprises two or more steps, at least one of which is an activity requiring the AI model.
13. The hyperautomation system according to claim 11, wherein the model capabilities of the one or more AI models include data processing and analysis, prediction and forecasting, decision-making and automation, and content generation.
14. The hyperautomation system according to claim 11, wherein the performance of the one or more AI models includes the resource efficiency of the hyperautomation system, or the output accuracy and the ability to correct the output.
15. The hyperautomation system according to claim 11, wherein the one or more model applications of the one or more AI models include a group of real-world use cases in which the one or more AI models are put into operation.
16. The hyperautomation system according to claim 11, wherein the benchmarking capability of the one or more AI models includes systematic testing and performance evaluation across tasks and datasets.
17. The hyperautomation system according to claim 11, wherein the multimodal agent-type automation includes probabilistic automation of the workflow for implementing context grounding and benchmarking during agent model selection.
18. The hyperautomation system according to claim 11, wherein the multimodal agent type automation includes probabilistic automation of the workflow using one or more artificial intelligence agents and one or more artificial intelligence models.
19. The hyperautomation system according to claim 11, wherein the agent model selection includes selecting the optimal artificial intelligence model from the available plurality of models for each of the one or more model required steps, based on the one or more benchmarking and historical job data.
20. The hyperautomation system according to claim 11, wherein the one or more agent-type automations include artificial intelligence-enhanced probabilistic automations performed by one or more AI agents that operate independently, dynamically, and adaptively in order to perform the multimodal agent-type automations.
