Artificial Intelligence Center Access for Self-Benchmarking and Agent-Based Automation
The hyperautomation system addresses the lack of self-benchmarking and AI access in traditional automation by using benchmarks and historical data to select the best AI models for each workflow step, improving efficiency and preventing incorrect model selection, thereby enhancing workflow processing and reducing user interaction.
Patent Information
- Application Number
- JP2025250911
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2025-12-15
- Publication Date
- 2026-11-19
AI Technical Summary
Traditional automation methods lack sufficient self-benchmarking and access to artificial intelligence (AI) centers, limiting the effectiveness and efficiency of agent-based automation systems.
A hyperautomation system that performs agent-based automation using benchmarks and historical data to select the best AI models for each workflow step, incorporating context grounding and self-benchmarking to improve workflow processing and prevent incorrect model selection.
Enhances workflow processing efficiency, reduces the need for user interaction, and prevents AI model hallucination by selecting the most suitable models dynamically, resulting in a more intelligent and context-aware automation system.
Smart Images

Figure 2026182719000007 
Figure 2026182719000008 
Figure 2026182719000009
Abstract
Description
[Technical Field]
[0001] This invention generally relates to agent-based automation, and more specifically to self-benchmarking and artificial intelligence (AI) center access for agent-based automation. [Background technology]
[0002] Traditional automation methods fail to provide sufficient self-benchmarking and access to artificial intelligence (AI) centers. Therefore, improved and / or alternative approaches may be beneficial. [Overview of the Initiative]
[0003] Certain embodiments of the present invention may provide alternatives or solutions to problems and needs in the art that have not yet been adequately identified, recognized, or solved by the current art, and / or may provide useful alternatives thereto. For example, some embodiments of the present invention relate to agent-based automation.
[0004] According to one or more embodiments, a method for a hyperautomation system is provided that performs one or more agent-based automations using benchmarks. The hyperautomation system includes memory for storing computer code for the one or more agent-based automations and the one or more artificial intelligence agents. The hyperautomation system includes at least one processor that executes the computer code and selects and assigns one or more artificial intelligence models to improve the one or more agent-based automations. The method includes tracking a current workflow execution, which includes storing the current result of a model request step in the current workflow, and matching the current workflow with two or more previous workflows based on the current result of the model request step. The method includes comparing the current result with the corresponding result of the two or more previous workflows to generate two or more benchmark results and capabilities. The method includes selecting and assigning one or more artificial intelligence models from an artificial intelligence center based on the best benchmark result among the two or more benchmark results and capabilities, which include context, prediction, or historical data. The artificial intelligence models correspond to the model request steps of the previous workflows among the two or more previous workflows. [Brief explanation of the drawing]
[0005] To facilitate understanding of the advantages of specific embodiments of the present invention, a more detailed description of the invention, as briefly outlined above, will be made with reference to specific embodiments shown in the accompanying drawings. It should be understood that these drawings only illustrate typical embodiments of the invention and should therefore not be considered limiting; however, the invention will be described in more detail and specifically with reference to the accompanying drawings. The drawings are as follows:
[0006] [Figure 1]Figure 1 is an architectural diagram showing a hyperautomation system configured to perform agent-based automation and orchestration in one or more embodiments.
[0007] [Figure 2] Figure 2 shows some of the combined capabilities of artificial intelligence (AI) agents and robotic process automation (RPA) robots in one or more embodiments.
[0008] [Figure 3] Figure 3 shows a pool of AI agents, RPA robots, agent-based orchestration processes (AOPs), and applications, according to one or more embodiments.
[0009] [Figure 4] Figure 4 shows an example of an AI agent service interface according to one or more embodiments.
[0010] [Figure 5] Figure 5 shows an example of an AOP development interface according to one or more embodiments.
[0011] [Figure 6] Figure 6 shows an example of an RPA development interface according to one or more embodiments.
[0012] [Figure 7] Figure 7 shows one or more embodiments of an end-to-end AI agent, RPA robot, and AOP development and deployment system.
[0013] [Figure 8] Figure 8 is an architectural diagram showing one or more embodiments of agent-based automation and RPA systems.
[0014] [Figure 9] FIG. 9 is an architectural diagram illustrating a deployed RPA system according to one or more embodiments.
[0015] [Figure 10] FIG. 10 is an architectural diagram illustrating the relationship between a designer, activities, and drivers according to one or more embodiments.
[0016] [Figure 11] FIG. 11 is an architectural diagram illustrating a computing system according to one or more embodiments.
[0017] [Figure 12A] FIG. 12A is a diagram illustrating an example of a neural network according to one or more embodiments.
[0018] [Figure 12B] FIG. 12B is a diagram illustrating an example of a neuron according to one or more embodiments.
[0019] [Figure 13] FIG. 13 is an architectural diagram illustrating a reference architecture of a generative AI model according to one or more embodiments.
[0020] [Figure 14] FIG. 14 is a flowchart illustrating a process for training an AI / ML model according to one or more embodiments.
[0021] [Figure 15] FIG. 15 is a flowchart illustrating a process according to one or more embodiments.
[0022] Unless otherwise indicated, like reference numerals consistently designate corresponding features throughout the accompanying drawings. [Modes for carrying out the invention]
[0023] Some embodiments of the present invention relate to agent-based automation. More specifically, embodiments relate to agent model selection for individual workflow steps based on benchmark and historical data for agent-based automation. Furthermore, embodiments relate to self-benchmarking and artificial intelligence (AI) center access for agent-based automation.
[0024] According to one or more embodiments, a hyperautomation system performs agent-based automation with agent model selection, which includes context grounding for multiple model capabilities and benchmarks used to select a specific model for each step of the workflow that requires an AI model. The agent model selection of the hyperautomation system may include context grounding for selecting a specific model for each step of the workflow performed in multimodal agent-based automation. Furthermore, the agent model selection of the hyperautomation system may include using benchmarks and historical job data, in addition to the context grounding, to select a model for each step of the workflow. The agent model selection of the hyperautomation system may select the best model for each step that requires an AI model. Prior to the best model selection, the agent model selection of the hyperautomation system may perform self-benchmarking, access an artificial intelligence (AI) center, and run within a sandbox. Thus, the agent model selection provides improved control over the multimodal agent-based automation to achieve technical effects, benefits, and advantages such as improved workflow processing, increased step execution efficiency, and prevention of hallucination due to incorrectly selected AI models. Therefore, the agent model selection of the hyperautomation system generates a more intelligent and context-aware hyperautomation system.
[0025] Figure 1 is an architectural diagram showing a hyperautomation system 100 configured to perform agent-based automation and orchestration in one or more embodiments. As used herein, “hyperautomation” refers to an automation system that combines components of process automation, agent-based automation, integration tools, and technologies that amplify the ability to automate tasks. Examples of such components include, but are not limited to, artificial intelligence (AI) agents, agent-based orchestration processes (AOPs), and robotic process automation (RPA) robots.
[0026] Generally, as used herein, “AI agent” refers to AI-enhanced probabilistic automation that operates independently, dynamically, makes decisions, takes actions, and adaptively. In some cases, an AI agent operates in this manner as a result of using a Large-Scale Language Model (LLM) or other AI model, which themselves typically have a probabilistic nature. According to one or more embodiments and as described herein, the one or more AI agents may implement context grounding techniques (e.g., Search Augmentation Generation (RAG), extraction, and semantic storage), advanced agent-based search (semantic search and retrieval, hybrid search, and wide-area search with re-ranking), tethering (including automatic tethering), and query decomposition.
[0027] Generally, as used herein, "AOP" refers to a dynamic yet predictable automation that combines probabilistic and deterministic methods. In some cases, AOP is automation that allows a user to describe the overall business process. The user is, but is not limited to, any person who has access to the system performing the automation (e.g., developers, engineers, customers, etc.). AOP can be created using an interface that enables the creation of business flowcharts described in Business Process Model and Notation (BPMN), which is an Extensible Markup Language (XML) description of the business process (see Figure 5 for example).
[0028] Generally, the term "RPA robot" as used herein refers to rule-based deterministic automation that operates predictably and makes deterministic decisions.
[0029] For example, in some embodiments, one or more RPAs may be used at the core of a hyperautomation system, and in certain embodiments, automation capabilities may be extended by AI / machine learning (ML), process mining, analytics, agent-based automation, and / or other advanced tools. For example, as the hyperautomation system learns processes, trains AI / ML models, and leverages analytics, more and more knowledge work can be automated, and computing systems within an organization, both those used by individuals and those operating autonomously, may be involved as participants in the hyperautomation process. Some embodiments of the hyperautomation system enable users and organizations to discover, understand, and extend automation efficiently and effectively.
[0030] In such embodiments, the AI agent “coexists” with the RPA robots and AOPs that perform the RPA. As described herein, the AI agent is AI-skilled automation that operates independently, makes dynamic decisions, performs actions, and can adapt its performance. The AI agent can dynamically leverage the tools available through these RPA robots to perform tasks such as document processing, user interface (UI) automation, and semantic copy-and-paste between source and target. The AI agent can dynamically select these tools and execute them in the form of a pipeline.
[0031] Generally, agent-based automation is probabilistic automation performed by one or more AI agents. Agent-based automation expands an organization's automation potential by focusing on the entire end-to-end process, rather than just individual tasks. Teams of RPA robots and / or AOP led by AI agents can enable a single employee to accomplish the work of many. Agent-based automation via AI agents and / or AOP gives managers room for guidance, doctors more time to care for patients, developers the ability to fine-tune their work, engineers the freedom to innovate, and customers a seamless, personalized experience.
[0032] Various technical effects, benefits, and advantages can be achieved through agent-based automation. Agent-based automation improves memory usage by reducing the amount of storage required for data and improves processor efficiency by reducing the number of calls and actions. Agent-based automation also potentially offers the ability to process gigabytes, terabytes, petabytes, or more of data that would be impossible for processes performed by humans, whether mental or manual. Agent-based automation can also enable the use of fewer triggers and models through dynamic decision-making. Agent-based automation can also employ context grounding, which "constrains" LLM or AI models to their relevant context by tethering AI agents to desired contexts, thereby improving the efficiency of LLM or AI models.
[0033] An AI agent may have agent memory that evolves to store user interactions, feedback, corrections, and solutions (e.g., dynamic and / or direct user input from human-in-the-loop actions). Human-in-the-loop actions, or "human-in-the-loop" as used herein, may include AI agents and RPA robots working in conjunction with a user to receive dynamic and / or direct user input. As agent memory increases, AI agents can become increasingly autonomous, reducing the need for dynamic and / or direct user input and improving efficiency. AI agents may also learn to become more efficient based on agent memory if more efficient solutions are contained within or derived from it. For example, an AI agent may periodically process agent memory to analyze patterns in order to achieve greater autonomy.
[0034] As used herein, “agent memory” is a dynamic caching (i.e., memory) system for managing escalations and tool calls. As an example of operation, if an AI agent encounters a problem during execution, the AI agent may prompt or otherwise request user interaction or feedback to overcome the problem, remember / cached that interaction or feedback, and learn from it to reduce the need for repeated user input. According to one or more technical effects, benefits, and advantages, agent memory provides improved efficiency by remembering solutions to common problems and minimizing potentially costly tool calls. The cooperative operation between the AI agent and agent memory potentially “bends the curve” so that the need for user interaction decreases as the AI agent continuously learns through agent memory.
[0035] Generally, agent-based orchestration is implemented by a conductor application that implements one or more AOPs and / or AI agents to orchestrate AI agents, third-party agents, RPA robots, AOPs, and users to execute workflows. Agent-based orchestration enables agent-based automation, modeling, and monitoring of complex business processes from start to finish. Agent-based orchestration also offers a unique ability to orchestrate RPA robots, AI agents, AOPs, third-party agents, and users across the entire end-to-end workflow. Agent-based orchestration is beneficial for the successful scaling of agent-based automation.
[0036] AI agents for agent-based automation are AI model-based, which allows them to operate independently of the user and implement these agent-based automations. AI agents are also goal-oriented, making probabilistic decisions using context. Furthermore, AI agents are well-suited for ad-hoc tasks requiring high adaptability. AI agents learn how to perform tasks and improve over time. AI agents can use and select various tools to accomplish tasks, gather context, and perform actions. In some embodiments, AI agents can construct workflows and generate automation for RPA robots and / or other AI agents to execute.
[0037] The AI agent generates a dynamic plan to achieve the objective for each instruction using the provided tools and context by executing an "agent loop." Once the dynamic plan is generated, the AI agent utilizes an efficient execution path for the dynamic plan. If the dynamic plan has two or more steps that can be executed in parallel, the AI agent executes those two or more steps in parallel based on the available resources. After each step is completed, the AI agent retrieves the output from that step and regenerates the next step or multiple steps. Thus, the agent loop continues until the objective is achieved.
[0038] As described herein, RPA robots are rule-based automations that operate predictably and make deterministic decisions. RPA robots are highly reliable and efficient and are suitable for routine tasks. RPA robots, along with AI agents, may employ human-in-the-loop behavior for exception handling. According to one or more embodiments, AI agents are more flexible, more abstract, and more autonomous than RPA robots and AOPs. Furthermore, RPA robots are more stable, more specific, and more controllable than AI agents and AOPs. Moreover, AOP processes fall within a range between the flexibility / stability, abstraction / specificity, and autonomy / controllability properties of AI agents and RPA robots, respectively.
[0039] As further described herein with respect to Figure 3, the AI agent, AOP, and RPA robot may discover and use each other as tools to accomplish tasks. The AI agent, AOP, and RPA robot may also access and use various applications (e.g., via Application Programming Interfaces (APIs)). Tools may be manually configured by developers for automation, and / or the AI agent and RPA robot may discover and use tools at runtime.
[0040] In some embodiments, AI agents, AOPs, and RPA robots can work in collaboration with users (e.g., human-in-the-loop), enabling them to make faster, more consistent, and more informed decisions. Furthermore, the use of AI agents, AOPs, and RPA robots allows users to accomplish more because they can undertake additional repetitive, monotonous, and ad-hoc tasks on a scale that would be impossible for humans to perform. If an AI agent, AOP, or RPA robot encounters an exception, the user can make the decision. Thus, the user can be elevated to and focus on becoming a supervisor, decision-maker, and organizational leader.
[0041] AI models provide AI agents with the ability to reason, plan, create, and make autonomous decisions. AI models may also be used by RPA robots for task-specific activities such as document processing or data analysis. AI models can be enhanced with business-specific content and context (e.g., from a collection of corporate context repositories), which improves the accuracy and results of the AI models. AI models may be applied individually or simultaneously, depending on the complexity of the task. The selection of AI models can be from the RPA vendor's model library, third-party models, and Bring Your Own Model (BYOM) options.
[0042] The hyper-automation system 100 includes user computing systems such as a desktop computer 102, a tablet 104, and a smartphone 106. However, any desired user computing system, including but not limited to smartwatches, laptop computers, servers, and Internet-of-Things (IoT) devices, may be used without departing from the scope of this disclosure.
[0043] As disclosed herein, in some embodiments, there are three types of automation: (1) agent-based automation implemented by each AI agent, (2) RPA implemented by each RPA robot, and (3) composite automation achieved by a combination of AI agents and RPA robots to accomplish more complex overall tasks. Automations 110, 112, and 114 may include, but are not limited to, those performed individually or by RPA robots and / or AI agents to achieve larger composite automation. Other processes, such as listeners, may also be implemented. These processes may be standalone applications, subprocesses of other applications, parts of operating systems, any other suitable software and / or hardware, or any combination thereof, and will not deviate from the scope of this disclosure. In fact, in some embodiments, the logic of a process is implemented partially or entirely through physical hardware.
[0044] Each user computing system 102, 104, and 106 runs its respective automation 110, 112, and 114, which are implemented by RPA robots, AI agents, AOP, etc. In some embodiments, automations 110, 112, and 114 may be stored remotely (e.g., on a server 130 or in a database 140 and accessed via a network 120) and loaded by RPA robots and / or AI agents to implement automations 110, 112, and 114. The database 140 may store structured data and / or unstructured data, the former of which is typically required for RPA. RPA automations may exist as scripts (e.g., Extensible Markup Language (XML), Extensible Application Markup Language (XAML), etc.) or be compiled into machine-readable code (e.g., as a dynamic-link library). In the case of AI agents, agent-type automations may be generated based on, for example, a plain text description of a desired goal.
[0045] The listener monitors and records data related to user interactions with each computing system and / or the operation of the unattended computing system, and transmits such data to the core hyperautomation system 120 via a network (e.g., a local area network (LAN), a mobile communications network, a satellite communications network, the internet, or any combination thereof). Such data may include, but is not limited to, which buttons were clicked, where the mouse was moved, text entered into fields, whether one window was minimized and another was opened, and applications associated with windows. In certain embodiments, data from the listener may be transmitted periodically as part of a heartbeat message. In some embodiments, such data may be transmitted to the core hyperautomation system 120 after a predetermined amount of data has been collected, after a predetermined period of time has elapsed, or both. One or more servers, such as server 130, receive and store the data from the listener in a database, such as database 140.
[0046] If automations 110, 112, and 114 are RPAs, they can execute logic developed in the workflow at design time. The workflow may include a set of steps, defined herein as “activities,” which are executed in a sequence or some other logical flow. Each activity may include actions such as clicking a button, reading a file, or writing to a log panel. In some embodiments, workflows may be nested or embedded.
[0047] Long-running workflows for RPA in some embodiments are master projects that support long-running transactions in service orchestration, human-in-the-loop, and unattended environments. See, for example, U.S. Patent No. 10,860,905, which is incorporated herein by reference in its entirety. Human-in-the-loop is involved when a particular process requires user input (e.g., dynamic and / or direct user input) to handle exceptions, approvals, or validations before proceeding to the next step in the activity. In this situation, the execution of the process is paused, and the RPA robot is released until the human-in-the-loop portion of the task is completed.
[0048] Long-running workflows can orchestrate human-in-the-loop operations and RPA robot tasks in combination with process invocation activities and non-user interaction activities, facilitating workflow fragmentation via persistence activities. In some embodiments, multiple or numerous computing systems may participate in the execution of the logic of the long-running workflow. Long-running workflows may run within a session to facilitate rapid execution. In some embodiments, long-running workflows may orchestrate background processes that may include activities making API calls and run within the long-running workflow session. These activities may be invoked by process invocation activities in some embodiments. Processes with user interaction activities that run within a user session may be invoked by starting a job from a conductor activity (conductors will be described in more detail later in this specification). In some embodiments, the user may interact through tasks that require forms to be completed within the conductor. Activities may include waiting for the completion of form tasks and then resuming the long-running workflow.
[0049] One or more of the automations 110, 112, and 114 communicate with the core hyperautomation system 120. In some embodiments, the core hyperautomation system 120 may run conductor applications on one or more servers, such as server 130. Although one server 130 is shown for illustrative purposes, multiple or numerous servers in close proximity to each other or in a distributed architecture may be used without departing from the scope of the invention. For example, one or more servers may be provided for conductor functions, AI / ML model serving, authentication, governance, and / or any other appropriate functions without departing from the scope of the invention. In some embodiments, the core hyperautomation system 120 may incorporate or be part of a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, etc. In certain embodiments, the core hyperautomation system 120 may host multiple software-based servers on one or more computing systems, such as server 130. In some embodiments, one or more servers of the core hyperautomation system 120, e.g., server 130, may be implemented via one or more virtual machines (VMs).
[0050] In some embodiments, one or more of the automations 110, 112, 114 may invoke one or more AI / ML models 132 deployed on or accessible by the core hyperautomation system 120 and trained to accomplish various tasks. For example, the AI / ML models 132 may include models trained to explore various application versions, perform computer vision (CV), perform optical character recognition (OCR), generate user interface (UI) descriptors, suggest the next activity or sequence of activities in a workflow, perform semantic matching, perform natural language processing (NLP), generate or modify code and / or workflows, etc. The AI / ML models may be trained using labeled data, including but not limited to elements from data sources (e.g., web pages, forms, scanned documents, application interfaces, screens, etc.), previously created workflows, screenshots of various application screens in various versions with their respective corresponding UI elements, and libraries of UI objects. The AI / ML models 132 may be trained to achieve a desired confidence threshold without overfitting to a given training dataset. In general, UI elements, UI descriptors, applications, and application screens can be considered UI objects.
[0051] As will be discussed in more detail later herein, the AI / ML model 132 can be trained for any suitable purpose without departing from the scope of the invention. In some embodiments, two or more of the AI / ML models 132 can be chained together (e.g., in series, parallel, or a combination thereof) and they can work together to provide output. The AI / ML model 132 can perform or assist in CV, OCR, document processing and / or document understanding, semantic learning and / or analysis, analytical prediction, process discovery, task mining, testing, automated workflow generation, sequence extraction, cluster detection, speech-to-text conversion, NLP, semantic matching, and any combination thereof. However, any desired number and / or types of AI / ML models can be used without departing from the scope of the invention. For example, by using multiple AI / ML models, the system can get a complete picture of what is happening on a given computing system. For example, one AI / ML model may perform OCR, another may detect buttons, and yet another may compare sequences. The patterns may be determined individually by one AI / ML model, or collectively by multiple AI / ML models. In certain embodiments, one or more AI / ML models are deployed locally on at least one computing system 102, 104, 106.
[0052] In some embodiments, multiple AI / ML models 132 may be used. Each AI / ML model 132 is an algorithm (or model) that operates on data, and for example, the AI / ML model itself may be a deep learning neural network (DLNN) of artificial "neurons" trained on training data. In some embodiments, the AI / ML model 132 may have multiple layers that perform statistical modeling (e.g., Hidden Markov Models (HMMs)) and deep learning techniques (e.g., Long Short-Term Memory (LSTM) deep learning, encoding of previous hidden states, etc.) to perform the desired function.
[0053] In some embodiments, the hyper-automation system 100 may provide four main functional groups: (1) discovery, (2) automation construction, (3) management, and (4) engagement. Automation (e.g., executed on a user computing system, server, etc.) may, in some embodiments, be performed by, for example, an RPA robot, AOP, or AI agent, and may provide any of the functions described herein. For example, an RPA robot may include attended robots, unattended robots, and / or test robots. An attended robot works with the user to assist with a task (e.g., via UiPath Assistant®). An unattended robot operates independently of the user and may run in the background without user awareness. A test robot executes test cases against an application or workflow. In some embodiments, test robots may run in parallel on multiple computing systems.
[0054] The discovery function can discover various opportunities for automating business processes and provide automated recommendations for them. Such a function may be implemented by one or more servers, such as server 130. In some embodiments, the discovery function may include providing an automation hub, process mining, task mining, and / or task capture. The automation hub (e.g., UiPath Automation Hub®) may provide a mechanism for managing automation deployments with visibility and control. For example, automation ideas may be crowdsourced from employees via a submission form. Feasibility and return on investment (ROI) calculations for automating these ideas may be provided, documentation for future automations may be collected, and collaboration may be provided to accelerate the process from automation discovery to construction.
[0055] Process mining (e.g., via UiPath Automation Cloud® and / or UiPath AI Center®) is the process of collecting and analyzing data from applications (e.g., enterprise resource planning (ERP) applications, customer relationship management (CRM) applications, email applications, call center applications, etc.) to identify what end-to-end processes exist within an organization, how to effectively automate them, and what impact such automation would have. This data may be collected, for example, by listeners from user computing systems 102, 104, and 106 and processed by servers such as server 130. In some embodiments, one or more AI / ML models 132 may be used for this purpose. This information may be exported to an automation hub to speed up implementation and avoid manual information transfer. The objective of process mining may be to increase business value by automating processes within an organization. Some examples of process mining goals include, but are not limited to, increased profits, improved customer satisfaction, regulatory and / or contractual compliance, and improved employee efficiency.
[0056] Task mining (e.g., via UiPath Automation Cloud® and / or UiPath AI Center®) identifies and aggregates workflows (e.g., employee workflows), then applies AI to reveal patterns and variations in daily operations, and scores such tasks in terms of ease of automation and potential savings (e.g., time savings and / or cost savings). One or more AI / ML models 132 may be used to reveal recurring task patterns in the data. Recurring tasks suitable for automation can then be identified. In some embodiments, this information may be initially provided by a listener and analyzed on a server of the core hyperautomation system 120, such as a server 130. Insights from task mining (e.g., XAML process data) may be exported to a process document or a designer application such as UiPath Studio® for faster creation and deployment of automations. Task mining in some embodiments may include taking screenshots with user actions (e.g., mouse click locations, keyboard input, application windows and graphical elements the user was interacting with, and timestamps of such interactions), collecting statistical data (e.g., execution time, number of actions, number of text inputs), editing and annotating screenshots, and specifying the types of actions to be recorded.
[0057] Task capture (e.g., via UiPath Automation Cloud® and / or UiPath AI Center®) automatically documents attended processes as users work through them, or provides a framework for unattended processes. Such documentation may include automated tasks in the form of process definition documents (PDDs), skeletal workflows, capture of actions for each part of the process, recording of user actions, and automatic generation of comprehensive workflow diagrams including details of each step, Microsoft Word® documents, XAML files, etc. The build-ready workflow can, in some embodiments, be exported directly to a designer application such as UiPath Studio®. Task capture can simplify the requirements gathering process for both subject matter experts describing processes and Center of Excellence (CoE) members providing production-quality automation.
[0058] The construction of automation can be achieved through designer applications (e.g., UiPath Studio®, UiPath StudioX®, or UiPath Studio Web®). For example, a developer in the RPA development facility 150 can use the designer application 154 of the computing system 152 to build and test agent-based automation, RPA, AOP, and / or hybrid automation for various applications and environments such as web, mobile, SAP®, and virtual desktops. The developer can also build AOP. For example, the developer can create automations that are executed by RPA robots, AI agents, AOP, or combinations thereof. API integrations can be provided for various applications, technologies, and platforms. Predefined activities, drag-and-drop modeling, and workflow recorders can facilitate automation with minimal coding. Document understanding capabilities can be provided that invoke one or more AI / ML models 132 via drag-and-drop AI skills for data extraction and interpretation. Such automations can handle virtually any document type and format, including tables, checkboxes, signatures, and handwritten text. Once the data has been validated or exceptions have been handled, this information can be used to retrain each AI / ML model, improving its accuracy over time.
[0059] The designer application 152 may be designed to invoke one or more trained AI / ML models 132 on the server 130 and / or generated AI models 172 in a cloud environment via the network 120 (e.g., a local area network (LAN), mobile communications network, satellite communications network, the internet, or any combination thereof) to support the automated development process. In some embodiments, one or more AI / ML models may be packaged in the designer application 152 or otherwise stored locally on the computing system 150.
[0060] In some embodiments, the designer application 152 and one or more AI / ML models 132 may be configured to use an object repository stored in a database 140. See, for example, U.S. Patent No. 11,748,069, which is incorporated herein by reference in its entirety. Generally, the object repository is a storage mechanism used by automation for images, text, semantic data, taxonomic associations, ontology associations, UI objects, etc. For example, the object repository may include a library of UI objects that can be used to develop workflows through the designer application 152. The object repository may be used for UI automation to add UI descriptors to activities in the workflow of the designer application 152. In some embodiments, one or more AI / ML models 132 may generate new UI descriptors and add them to the object repository in the database 140.
[0061] Once the automation is complete in the designer application 152, it can be exposed on the server 130 and pushed to computing systems 102, 104, 106, etc. For example, as new UI descriptors are created and / or existing UI descriptors are modified, a global repository of a shareable and collaborative UI object library can be built for all automations. Taxonomies and ontologities can be used for the object repository. A taxonomy is a hierarchical structure of subcategories. An ontology is a formal representation of a knowledge domain that includes concepts, characteristics, and the relationships between them. In an ontology, the relationships between categories are not necessarily hierarchical, and ontological relationships can span multiple screens of an application.
[0062] For example, integrated services may enable developers to seamlessly combine UI automation with API automation. Automations of any kind described herein may require APIs or be built to span both API and non-API applications and systems. A repository (e.g., UiPath Object Repository®) or marketplace (e.g., UiPath Marketplace®) for pre-built automation templates and solutions may be provided, enabling developers to automate a wide variety of processes more quickly. Thus, when building automations, the hyperautomation system 100 may provide a user interface, development environment, API integration, pre-built and / or custom-built AI / ML models, development templates, an integrated development environment (IDE), and advanced AI capabilities. In some embodiments, the hyperautomation system 100 enables the development, deployment, management, configuration, monitoring, debugging, and maintenance of RPA robots, AOPs, and AI agents that may provide automation for the hyperautomation system 100.
[0063] In some embodiments, components of the hyperautomation system 100, such as a designer application and / or an external rule engine, assist in managing and enforcing governance policies to control the various functions provided by the hyperautomation system 100. Governance is the ability of an organization to establish policies to prevent users from developing automations (e.g., RPA robots, AOPs, and / or AI agents) that could perform actions that could harm the organization, such as violating the General Data Protection Regulation (GDPR) of the European Union, the Healthcare Portability and Accountability Act (HIPAA) of the United States, or the terms of use of third-party applications. In some embodiments, access control and governance constraints are implemented at the robot and / or robot design application level, because developers could otherwise create automations that violate privacy laws, terms of use, etc., while running their automations. This can provide an additional level of security and compliance to the automation process development pipeline in some embodiments by preventing developers from having dependencies on unauthorized software libraries. Such unauthorized software libraries may introduce security risks or operate in a manner that violates policies, regulations, privacy laws, and / or privacy policies. For example, see U.S. Patent No. 11,733,668, which is incorporated herein by reference in its entirety.
[0064] The management functions can provide management, deployment, and optimization of automation across the entire organization. In some embodiments, these management functions may include orchestration, test management, AI capabilities, and / or insights. The management functions of the hyperautomation system 100 can also serve as an integration point with third-party solutions and applications for automation applications and / or RPA robots. The management capabilities of the hyperautomation system 100 include, but are not limited to, facilitating the provisioning, deployment, configuration, queuing, monitoring, logging, and interoperability of RPA robots, AOPs, and / or AI agents.
[0065] Conductor applications such as UiPath Orchestrator® (which in some embodiments may be provided as part of UiPath Automation Cloud®, or as a cloud-native single-container suite on-premises, in a VM, in a private or public cloud, in a Linux® VM, or via UiPath Automation Suite®) provide orchestration capabilities for deploying, monitoring, optimizing, scaling, and securing RPA robots, AOP, and / or AI agent deployments. A test suite (e.g., UiPath Test Suite®) may provide test management for monitoring the quality of deployed automations. The test suite may facilitate test planning and execution, requirements fulfillment, and defect traceability. The test suite may include comprehensive test reporting.
[0066] Analytics software (e.g., UiPath Insights®) can track, measure, and manage the performance of deployed automations. The analytics software can align automation operations with specific key performance indicators (KPIs) and strategic outcomes for the organization. The analytics software can present results in a dashboard format to enhance user understanding.
[0067] For example, a data service (e.g., UiPath Data Service®) may store data in a database 140, and a drag-and-drop storage interface may allow data to be ingested into a single, scalable, and secure location. Several embodiments may provide low-code or no-code data modeling and storage for automation while ensuring seamless access to data, enterprise-grade security, and data scalability. AI functionality may be provided by an AI Center (e.g., UiPath AI Center®), which facilitates the integration of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options may make such functionality accessible even to non-data scientists. Deployed automations (e.g., RPA robots, AOPs, and AI agents) may invoke AI / ML models from the AI Center, such as AI / ML model 132. The performance of the AI / ML models may be monitored and trained and improved using user-validated data, such as that provided by a data review center 160. As a reviewer, the user may provide labeled data to the core hyperautomation system 120 via a review application 152 on the computing system 154. For example, the reviewer may verify that the predictions by the AI / ML model 132 and / or the generative AI model 172 are accurate, or provide corrections if they are not. As a reviewer, the user may provide dynamic and / or direct user input (e.g., within the scope of human-in-the-loop operations) to the AI agent, and such dynamic and / or direct user input (e.g., responses and corrections provided by the reviewer) may be used to train the LLM used by the AI agent to be more accurate. In other words, such dynamic and / or direct user input may be stored as training data for retraining the AI / ML model 132 and / or the generative AI model 172, and may be stored in a database, for example, database 140.Subsequently, the AI Center may schedule and execute a training job to train a new version of the AI / ML model using the aforementioned training data. Both positive and negative examples are stored and can be used for retraining the AI / ML model 132 and / or the generative AI model 172.
[0068] Engagement features involve automation and users as a single team to seamlessly collaborate on desired processes. In some embodiments, low-code applications can be built to connect browser tabs and legacy software, even those lacking APIs (e.g., via UiPath Apps®). For example, applications can be quickly created using a web browser through a rich library of drag-and-drop controls. A single application can connect to a single automation or multiple automations.
[0069] An Action Center (e.g., UiPath Action Center®) provides a simple and efficient mechanism for handing over processes from automation to the user and vice versa. The user can provide approvals or escalations, handle exceptions, etc. The automation can then execute automated functions of a given workflow.
[0070] A local assistant may be provided as a launchpad for users to initiate automations (e.g., UiPath Autopilot®). Such an assistant may also provide semantic cut-and-paste functionality (e.g., UiPath Clipboard AI®). See, for example, U.S. Patent No. 12,124,806 and U.S. Patent Application Publications 2023 / 0107316, 2023 / 0415338, and 2024 / 0220581. This functionality may be provided, for example, within a tray provided by the operating system, and may allow users to interact with RPA robots, AOPs, AI agents, and automation-driven applications on their own computing systems. The interface may list automations approved for a given user and allow that user to run them. These may include ready-to-run automations from an automation marketplace, or from an internal automation store within an automation hub. Once automation is performed, it may run as a local instance in parallel with other processes on the computing system, allowing the user to continue using the computing system while the automation is performing its actions. In certain embodiments, the assistant is integrated with a task capture function, which allows the user to document processes that are about to be automated from the assistant launchpad.
[0071] In some embodiments, end-to-end measurement and governance of automation programs of any scale may be provided by the hyper-automation system 100. As described above, analytics may be used to understand the performance of automation (e.g., via UiPath Insights®). Data modeling and analysis using any combination of available business metrics and operational insights may be used for various automation processes. Custom-designed and pre-built dashboards enable data visualization across desired metrics, discovery of new analytical insights, tracking of performance indicators, discovery of ROI for automation, execution of telemetry monitoring on user computing systems, detection of errors and anomalies, and debugging of automation. An automation management console (e.g., UiPath Automation Ops®) may be provided to manage automation throughout the entire automation lifecycle. Organizations can control how automation is built, what users can do with it, and which automations users have access to.
[0072] In some embodiments, the hyper-automation system 100 provides an iterative platform on which processes can be discovered, automations can be built, tested, and deployed; performance can be measured; the use of automation can be easily provided to users; feedback can be obtained; AI / ML models can be trained and retrained; and the process can be repeated. This promotes a more robust and effective automation suite.
[0073] In some embodiments, the generative AI model 172 is used as described above. For example, an AI agent utilizes the generative AI model. The generative AI model 172 can generate various types of content, such as text, images, audio, and synthetic data. Various types of generative AI models can be used, including but not limited to LLMs, generative adversarial networks (GANs), diffusion models, flow-based models, variational autoencoders (VAEs), and transformers. For example, in the case of LLMs, NLP models such as word2vec, BERT, GPT-3, and ChatGPT may be used in some embodiments to facilitate semantic understanding and provide more accurate and human-like responses. These models may be part of an AI / ML model 132 hosted on server 130. For example, the generative AI model 172 may be trained on a large text information corpus to perform semantic understanding, understand the properties of things present on a screen from text, and automatically generate code. An AI agent may use such a generative AI model 172. In certain embodiments, generative AI models 172 provided by existing cloud ML service providers such as OpenAI®, Google®, Amazon®, Microsoft®, IBM®, Nvidia®, and Meta® may be used and trained to provide such functionality. In generative AI embodiments where the generative AI model 172 is remotely hosted, the server 130 may be configured to integrate with a third-party API, which allows the server 130 to send requests containing the necessary input information to the generative AI model 172 and receive responses in return (e.g., semantic matching of fields between application versions, classification of application types on a screen, responses to natural language queries from the user, etc.). Such embodiments may provide a more advanced and sophisticated user experience, as well as access to state-of-the-art NLP and other ML capabilities offered by these companies.
[0074] One embodiment of the generative AI model 172 in some embodiments is the use of transfer learning. In transfer learning, a pre-trained generative AI model, such as an LLM, is fine-tuned for a specific task or domain. This allows the LLM to leverage knowledge already learned during initial training and adapt it to a specific application. For example, in the case of an LLM, the pre-training stage typically involves training the LLM on a large text corpus consisting of billions of words. During this stage, the LLM learns the relationships between words and phrases, thereby enabling the LLM to generate consistent, human-like responses to text-based input. The output of this pre-training stage is an LLM with a high level of understanding of fundamental patterns in natural language.
[0075] In the fine-tuning phase, a pre-trained LLM is adapted to a specific task or domain by training the LLM on a smaller dataset specific to that task. For example, in some embodiments, the LLM may be trained to analyze one or more data sources to improve its accuracy regarding their content. This data may include, but is not limited to, prompt tuning or instruction tuning, where the model is specifically trained to better understand and follow certain types of instructions or prompts, thereby improving its ability to perform a particular task when given appropriate instructions. Such information may be provided as part of the training data, and the LLM may learn to focus on these areas and more accurately identify the data elements within them. Fine-tuning allows the LLM to learn the nuances of the task or domain, such as the specific vocabulary and syntax used in that domain, without requiring as much data as would be needed to train the LLM from scratch. By leveraging the knowledge learned in the pre-training phase, a fine-tuned LLM can achieve state-of-the-art performance for a particular task with relatively small amounts of training data.
[0076] LLM can use vector databases. Vector databases index, store, and provide access to structured or unstructured data (e.g., text, images, time-series data) along with their vector embeddings. Data such as text can be tokenized, where single characters, words, or sequences of words are parsed from text into tokens. These tokens are then "embedded" in vector embeddings, which are numerical representations of the data. Vector databases enable LLM to quickly and extensively find and retrieve similar objects in a production environment, which is not possible with manual processes.
[0077] AI and ML enable the numerical representation of unstructured data in vector embeddings without losing its semantic meaning. A vector embedding is a long sequence of numbers that describes the features of the data object it represents. Similar objects are grouped closer together in the vector space. In other words, the more similar the objects, the closer their vector embeddings are to each other. Similar objects can be found using vector search, similarity search, or semantic search and retrieval. The distance between vector embeddings can be calculated using a variety of techniques, including but not limited to squared Euclidean distance or L2-squared distance, Manhattan distance or L1 distance, cosine similarity, dot product, and Hamming distance. It may be beneficial to select the same metrics used to train the AI / ML model.
[0078] Vector indexing can be used to organize vector embeddings so that data can be retrieved efficiently. Calculating the distance between a given vector embedding and all other vector embeddings in a vector database using the k-nearest neighbors (kNN) algorithm can be computationally expensive, as the required computation increases linearly (O(n)) with respect to the number of dimensions and data points when the number of data points is large. Finding similar objects using an approximate nearest neighbor (ANN) approach is more efficient. Since the distances between vector embeddings are pre-calculated and similar vectors are organized and stored close to each other (e.g., in clusters or graphs), similar objects can be found more quickly. This process is called "vector indexing." ANN algorithms that can be used in several embodiments include, but are not limited to, clustering-based indexing, proximity-graph-based indexing, tree-based indexing, hash-based indexing, and compression-based indexing.
[0079] Figure 2 shows some of the combined capabilities 200 of an AI agent 210 and an RPA robot 220 in one or more embodiments. The AI agent 210 is configured to process natural language instructions, achieve expected goals 231 based thereon, execute through dynamic decision-making or dynamic flow control with self-correcting capabilities 233, store information in long-term memory, evaluate its own performance 235, and further learn 237 from human-in-the-loop and its own performance during execution. The RPA robot 220 is leveraged by the AI agent 210 to respond to triggers 241 (e.g., from a conductor application such as UiPath Orchestrator®), to respond based on context 243 (i.e., the RPA robot 220 may take information from the context and perform deterministic steps, for example, to update a document based on the information taken from the context; or the agent 210 may use the acquired context to update a dynamic plan and perform the next step to achieve the goal according to the instructions), to leverage models 245 (e.g., CV models, document processing models, speech-to-text models, OCR models, AI models, etc.), to leverage tools 247 (e.g., to utilize tools available in the RPA ecosystem, such as full automation, workflows within automation, integrated service connector calls for third-party and first-party services, RPA designer application activities, LLM calls, automation, etc.), and to perform actions 249 that the RPA robot 220 can take (i.e., using the RPA robot 220 as a tool) based on input from the AI agent 210. The AI agent 210 can also update its own memory, update its plan to achieve goals according to instructions, self-evaluate and learn from its actions, self-correct when it encounters failures, and take action 249 to escalate to the user when assistance is needed.
[0080] As described above, agent-based automation achieves various technical effects, benefits, and advantages. Agent-based automation improves memory usage by reducing the amount of storage required for data and improves processor efficiency by reducing the number of calls and actions. Agent-based automation provides the ability to process gigabytes, terabytes, petabytes, or more of data that would be impossible for processes performed by humans, whether mental or manual. Agent-based automation enables the use of fewer triggers and models through dynamic decision-making. For example, as discussed herein, while conventional software automation alone may require 100 actions in one example scenario, agent-based automation in the same scenario can significantly reduce the number of actions required (e.g., to as few as 15 actions). Agent-based automation can also employ context grounding, which tethers the AI agent 210 to a desired context, "constraining" the LLM to the relevant context and thereby improving the efficiency of the LLM.
[0081] As used herein, “context grounding” refers to a methodology for improving models such as LLMs by integrating enterprise-specific information with pre-trained knowledge, enabling accurate responses to specialized or recent queries. In some embodiments, context grounding extends LLM responses using external data to obtain responses that the LLM would not inherently know and answer queries based on the provided context. For example, proprietary industry jargon and complex document structures can be challenges in ensuring effective retrieval and semantic matching; context grounding addresses this challenge by providing precise chunking of documents, allowing relevant information (e.g., information derived from proprietary industry jargon and complex document structures) to be passed to the LLM without noise. As an additional example, context grounding improves LLM responses by providing enhanced extraction and retrieval techniques tailored to diverse industries and applications (e.g., tailored to proprietary industry jargon and complex document structures).
[0082] Figure 3 shows a diagram 300 of AOP, AI agent, RPA robot, and application according to one or more embodiments.
[0083] AOP pool 310 includes AOP1, 2, ..., P, which implement business processes. As described above, AOP may be implemented as BPMN, which is executed by an AOP execution engine such as Temporal®. AOP may utilize AI agents and / or RPA robots to execute parts of the business process.
[0084] The AI agent pool 320 includes AI agents 1, 2, ..., I, who are trained to perform a variety of tasks such as billing investigations, seeking solutions with employees, and summarizing policies and technical specifications. The RPA robot pool 330 includes RPA robots 1, 2, ..., J, who perform a variety of automations such as UI automation, semantic matching automation, and form entry automation.
[0085] The application pool 340 includes applications 1, 2, ..., K in which AI agents and / or RPA robots can interact. For example, the applications may include CRM applications, invoice creation applications, payroll applications, banking applications, web applications, legacy system applications, word processor applications, spreadsheet applications, email applications, and the like. The AI agents, RPA robots, and applications may reside on a single computing system, or on multiple or numerous computing systems. The AOP is typically located in the cloud or on a server side, and in some embodiments, may reside on the same computing system as the conductor application 350.
[0086] AOP can trigger or invoke AI agents and RPA robots via conductor application 350. AI agents and RPA robots can also trigger or invoke each other via conductor application. For example, to invoke an RPA robot, an AI agent may make a "Start Job" call in conductor application 350. Note that RPA robots are deployed as automation controlled by conductor application 350. AI agents, AOP, and RPA robots can also trigger or invoke specific applications. For example, through information obtained from human-in-the-loop actions, an AI agent may dynamically learn which RPA robots, other AI agents, and / or applications to trigger or invoke to accomplish a task. For example, an AI agent may learn to trigger an RPA robot via conductor application 350 to fill out and submit a web form. The AI agent may also learn to open Microsoft Excel®, enter form information into the appropriate tab, and open and update a payroll application, etc. Furthermore, it may also learn to call or trigger an email resolution AI agent via the conductor application 350 to contact a bank customer service representative in the event of a problem. In some embodiments, the technical effects, benefits, and advantages may be similar to those discussed above with respect to Figures 1 and 2.
[0087] AI agents may belong to a tenant so that AI agents, AOPs, and RPA robots can find each other. A designer application may call a conductor to obtain a list of available RPAs. In some embodiments, there are three ways to obtain the automation capability: (1) a user provides a description of what the automation will do when creating a workflow in a designer application; (2) an AI agent and ML technology generate a summary of what a given workflow will do; or (3) a developer describes what the automation will do in a designer application. A conductor application may also have a list of which applications are available for a given AI agent and RPA robot. In other words, descriptions of available AI agents, RPA robots, and / or applications are derived or provided by the AI agent, ML technology, or user.
[0088] Figure 4 shows an example of an agent service interface 400 in one or more embodiments. As shown in Figure 4, the agent answers questions regarding policy documents provided within context grounding. The agent instruction pane 410 contains a natural language description entered by the user about what the AI agent is intended to do. The user prompt 420 allows the developer to enter content for the user prompt in the content field 422, if necessary. The tool dropdown 430 allows the developer to select the tools the AI agent will use, such as using an API for the application or calling an RPA robot to perform RPA.
[0089] The context dropdown 440 allows developers to configure context grounding for the AI agent. The context configuration pane 442 allows developers to provide a description via the description field 444 and an Elastic Common Schema (ECS) index via the ECS index field 446 for a specific policy document containing information about contracts, regulations, and what to do, in this example. Developers may also add contexts 450 to further complement the context grounding. User escalation options can be configured via the escalation dropdown 460. The query field 470 allows users to provide queries to be answered by the AI agent. When the user clicks the run button 480, the AI agent executes the query. The results of the AI agent's execution are then displayed in the run pane 490 as the AI agent retrieves and outputs them.
[0090] Figure 5 shows an example of an AOP development interface 500 in one or more embodiments. The AOP development interface 500 includes an AOP 510, an AI agent 520, and an RPA robot 530 that a user can select when developing a business process. These can be selected by the AI agent, AOP, or developer and dragged onto the canvas 540 to develop the AOP. In this example, a credit check 541 is implemented based on a credit check request 542 by retrieving customer data 543 from a database 545. Next, the AI agent 546 is called and determines the customer type (e.g., very likely to pay, likely to default, often between unemployment and employment) by analyzing the customer data in the database 545. This type is then provided to the RPA robot 547, which performs the credit check and takes this information into account when generating the credit check result 549. Alternatively, the AI agent, AOP, or developer may provide a description of the business process in field 550 and select the generate button 560. This explanation is provided to the LLM, which attempts to understand the business process and automatically create the AOP. The AI agent, AOP, or developer can then edit the AOP.
[0091] Figure 6 shows an example of an RPA development interface 600 according to one or more embodiments. The RPA development interface 600 includes components 610 that an AI agent, AOP, or developer can select when developing a workflow for an RPA robot. The AI agent, AOP, or developer can select it and drag it onto the canvas 620. Alternatively, the AI agent, AOP, or developer can enter a description of the workflow in field 630 and select the generate button 640. This description is provided to the LLM, which attempts to understand the workflow and automatically create an RPA robot. The AI agent, AOP, or developer can then edit the workflow for the RPA robot. Note that the functions shown and described with respect to Figures 4, 5, and 6 may, in some embodiments, be provided in a single designer application.
[0092] Figure 7 shows an end-to-end AI agent, RPA robot, and AOP development and deployment system 700 in one or more embodiments. A designer application 710 enables the AI agent, AOP, and developer to design workflows for subsequent automation (e.g., AOP, AI agent, and / or RPA robot). Once these subsequent AOP, AI agent, and / or RPA robots are tested and validated, the validated and tested AOP, AI agent, and / or RPA robots are packaged and published to the automation database 720.
[0093] The conductor application 730 manages the deployment of these packaged and published automations. When a software process 732 requests the execution of an automation, the conductor application 730 sends a start job command to the AOP engine 740, which selects and starts the automation from the AOP group 742. When executing the AOP group 742, it may encounter steps implemented by the AI agent 750 and / or the RPA robot 760. In some cases, the AOP engine 740 pauses the running AOP group 742 and requests the conductor application 730 to send a start job request to the appropriate AI agent 750 or RPA robot 760 to execute the step.
[0094] When an AI agent is requested, the conductor application 730 sends a start job request to the appropriate AI agent 750. This request may include natural language text or other information provided to the conductor application 730 from the AOP engine 740. The AI agent 750 then performs the step by running LLM 752 to assist in task execution. The AI agent 750 then sends information related to the task (e.g., requested information, an indication that the step is complete, an indication that the step has failed, etc.) to the conductor 730, which then provides this information to the AOP engine 740. The AOP engine 740 then resumes its operation.
[0095] When an RPA robot is requested, the conductor application 730 sends a start job request to the appropriate RPA robot 760. The RPA robot 760 then executes the requested RPA 762. The RPA robot 760 then sends task-related information (e.g., requested information, indication that the step was completed, indication that the step failed, etc.) to the conductor application 730, which provides this information to the AOP engine 740. The AOP engine 740 then resumes operation.
[0096] According to one or more embodiments, a user action may be required by the AOP 742, AI agent 750, or RPA 762. In this case, the AOP engine 740, AI agent 750, or RPA robot 760 contacts the user 770 for a human-in-the-loop action that contributes to automation. After the user 770 provides the user action, the AOP engine 740, AI agent 750, or RPA robot 760 resumes automation.
[0097] Figure 8 is an architectural diagram showing an agent-based automation and RPA system 800 in one or more embodiments. In some embodiments, the agent-based automation and RPA system 800 is part of the hyperautomation system 100 in Figure 1. The agent-based automation and RPA system 800 includes a designer 810 that enables an AI agent, AOP, or developer to design automations (e.g., workflows, natural language instructions for AI agents and AOPs, context grounding, tool configurations, RPA robots, AOPs, AI agents, etc.). The designer 810 may provide solutions for application integration, as well as solutions for automating third-party applications, managed information technology (IT) tasks, and business IT processes. The designer 810 may facilitate the development of automation projects, which are graphical representations of business processes. The designer 810 facilitates the development and deployment of automations (indicated by arrow 811). The designer 810 may be an application that runs on a user's desktop, an application that runs remotely in a VM, a web application, etc.
[0098] The automation project enables the automation of rule-based processes by allowing an AI agent, AOP, or developer to control the execution order and relationships of a set of custom steps developed in a workflow, such as “Activities,” as described herein. One commercial example of an embodiment of Designer 810 is UiPath Studio®. Each Activity may include actions such as clicking a button, reading a file, or writing to a log panel. In some embodiments, workflows may be nested or embedded.
[0099] Workflow types may include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suitable for linear processes, allowing a flow from one activity to another without complicating the workflow. Flowcharts may be particularly suitable for more complex business logic, allowing for the integration of decisions and the connection of activities in a more diverse manner through multiple branching logic operators. FSMs may be particularly suitable for large-scale workflows. FSMs may use a finite number of states in their execution, which are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suitable for determining the behavior of a workflow when it encounters an execution error and for debugging the process.
[0100] Once automation is developed in the designer 810, the execution of business processes is orchestrated by the conductor 820, which orchestrates one or more robots 830, one or more AI agents 850, and / or one or more AOPs 870 that execute the workflow developed in the designer 810. One commercial example of an embodiment of the conductor 820 is UiPath Orchestrator®. The conductor 820 facilitates the creation, monitoring, and management of resources in the environment. The conductor 820 can function as an integration point with third-party solutions and applications. As described above, in some embodiments, the conductor 820 may be part of the core hyperautomation system 120 in Figure 1.
[0101] It should be noted that the RPA robot 830 can operate independently for deterministic processes. The AI agent 850 and AOP 870 can also operate independently (for example, for non-deterministic processes), or they can utilize the RPA robot 830 or other AI agent 850 as tools to achieve part of their agent-based automation. The AI agent 850 can drive a composite automation that utilizes both the RPA robot 830 and the AI agent 850, or vice versa, and the AOP 870 may include such a composite automation.
[0102] The conductor 820 manages a fleet of RPA robots 830 and AI agents 850, connecting and executing (indicated by arrow 881) the RPA robots 830 and AI agents 850 from a central aggregation point (for example, as requested by an AOP engine implementing AOP). The types of RPA robots 830 that can be managed may include, but are not limited to, attended robots, unattended robots, development robots (similar to unattended robots but used for development and testing purposes), and non-production robots (similar to attended robots but used for development and testing purposes). Attended robots are triggered by user events and operate in parallel with the user on the same computing system. Attended robots may be used with the conductor 820 for centralized process deployment and log recording media. Attended robots may assist users in accomplishing various tasks and may be triggered by user events. In some embodiments, processes cannot be started from the conductor 820 for this type of robot, and / or they cannot be run under a locked screen. In certain embodiments, attended robots can only be started from the robot tray or from the command prompt. In some embodiments, attended robots should be run under user supervision.
[0103] Unattended robots can run unattended in a virtual environment and automate many processes. They can also provide support for remote execution, monitoring, scheduling, and work queuing. In some embodiments, debugging for any robot type can be performed in Designer 810. Both attended and unattended robots can automate a wide range of systems and applications, including but not limited to mainframes, web applications, VMs, enterprise applications (e.g., those manufactured by SAP®, Salesforce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).
[0104] Conductor 820 may have a variety of capabilities (indicated by arrow 882) including, but not limited to, provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interoperability. Provisioning may include creating and maintaining connections between RPA robots 830, AI agents 850, and / or AOP 870 and Conductor 820 (e.g., a web application). Deployment may include ensuring that the correct package versions are delivered to assigned RPA robots 830, AI agents 850, and / or AOP 870 for execution. Configuration may include maintaining and delivering environment and process configurations for RPA robots and AI agents. Queuing may include providing management of queues and queue items. Monitoring may include tracking identification data for robots and AI agents and maintaining user permissions. Logging may include storing and indexing logs in a database (e.g., a Structured Query Language (SQL) database or a "not only" SQL (NoSQL) database) and / or other storage mechanisms (e.g., ElasticSearch®, which provides the ability to store and rapidly query large datasets). Conductor 820 may provide interoperability by acting as a centralized communication point for third-party solutions and / or applications.
[0105] The RPA robot 830 is an execution agent that implements the workflow built in the designer 810. One commercial example of several embodiments of the RPA robot 830 is UiPath Robots®. In some embodiments, the RPA robot 830 installs the Microsoft Windows® Service Control Manager (SCM) management service by default. As a result, such an RPA robot 830 can open an interactive Windows® session under the local system account and has Windows® service privileges.
[0106] In some embodiments, the RPA robot 830 may be installed in user mode. For such an RPA robot 830, this means that it has the same privileges as the user on which it is installed. This feature may also be available for high-density (HD) robots, which ensure that the robot makes full use of the potential of each machine. In some embodiments, any type of RPA robot 830 may be configured in an HD environment.
[0107] In some embodiments, the RPA robot 830 is divided into multiple components, each dedicated to a specific automation task. These robot components may include, but are not limited to, an SCM management robot service, a user-mode robot service, an executor, an agent, and a command line. The SCM management robot service manages and monitors Windows® sessions and acts as a proxy between the conductor 820 and the execution host (i.e., the computing system on which the RPA robot 830 runs). These services trust and manage the credentials of the RPA robot 830. The console application is launched by the SCM under the local system.
[0108] In some embodiments, the user-mode robot service manages and monitors Windows® sessions and acts as a proxy between the conductor 820 and the execution host. The user-mode robot service can trust and manage the credentials of the RPA robot 830. If the SCM management robot service is not installed, the Windows® application may be automatically launched.
[0109] The executor can execute a given job (i.e., execute a workflow) under a Windows® session. The executor can recognize the dots per inch (DPI) setting for each monitor. The AI agent 850 may be a Windows® Presentation Foundation (WPF) application that displays available jobs in a system tray window. Note that these agents are different from the AI agent 850. The AI agent 850 may be a client of the service and may request the start or stop of jobs and changes to their settings. The command line is a client of the service. The command line is a console application that may request the start of a job and await its output.
[0110] As described above, dividing the components of the RPA robot 830 helps developers, support users, and computing systems more easily execute, identify, and track what each component is doing. In this way, special behavior can be configured for each component, for example, different firewall rules can be set for the executor and services. In some embodiments, the executor may always be aware of the DPI setting per monitor. As a result, workflows can run at any DPI regardless of the configuration of the computing system on which they were created. In some embodiments, projects from the designer 810 are also independent of the browser zoom level. For applications that are not DPI-aware, or applications that are intentionally marked as DPI-insensitive, DPI can be disabled in some embodiments.
[0111] In this embodiment, the agent-based automation and RPA system 800 is part of a hyperautomation system, such as the hyperautomation system 100 in Figure 1. Developers can use the designer 810 to build and test RPA, AOP, and AI agents that utilize AI / ML models (e.g., those as part of its AI center) deployed within the core hyperautomation system 840. Such RPA robots can send inputs for the execution of AI / ML models and receive their outputs via the core hyperautomation system 840.
[0112] As described above, one or more of the RPA robots 830 may be listeners. These listeners may provide the core hyperautomation system 840 with information about what users are doing when using their computing systems. This information can then be used by the core hyperautomation system for process mining, task mining, task capture, and the like.
[0113] To enable users to launch local RPA robots, an assistant / chatbot (of the Conductor 820) may be provided on the user's computing system. For example, the assistant / chatbot may be located in the system tray. The chatbot may have a user interface so that the user can see the text within the chatbot. Alternatively, the chatbot may lack a user interface and run in the background, listening to the user's voice using the computing system's microphone.
[0114] In some embodiments, data labeling may be performed by a user of the computing system on which the RPA robot or AI agent is running, or on another computing system from which the robot or AI agent provides information. For example, if a robot invokes an AI / ML model to perform CV on an image for a VM user, but the AI / ML model does not correctly identify a button on the screen, the user may draw a rectangle around the misrecognized or unrecognized component and also provide text indicating the correct identification. This information is provided to the core hyperautomation system 540 and can then be used later to train a new version of the AI / ML model.
[0115] Figure 9 is an architectural diagram showing a deployed RPA system 900 in one or more embodiments. In some embodiments, the RPA system 900 may be part of the agent-based automation and RPA system 800 in Figure 8 and / or the hyperautomation system 100 in Figure 1. Note that the architecture of the deployed RPA system 900 may not be used in some embodiments. The deployed RPA system 900 may be a cloud-based system, an on-premise system, a desktop-based system providing enterprise-level, user-level, or device-level automation solutions for automating different computing processes, etc.
[0116] Without departing from the scope of the present invention, it should be noted that the client side 901, the server side 902, or both may include any desired number of computing systems. In the client side 901, the robot application 910 includes an executor 912, an execution agent 914, and a designer 916. However, in some embodiments, the designer 916 may not be running on the same computing system as the executor 912 and the execution agent 914. The executor 912 is a running process. Multiple business projects may be running simultaneously. The execution agent 914 (e.g., Windows® service) is a single point of contact for all executors 912 in this embodiment. The execution agent 914 is also responsible for transmitting the robot's status (e.g., periodically sending "heartbeat" messages indicating that the robot is still functioning) and downloading the necessary versions of packages to be executed.
[0117] The listener 930 monitors and records data related to user interaction with the attended computing system in which the listener 930 resides and / or the operation of the unattended computing system. The listener 930 may be an RPA robot, AOP, an AI agent, part of an operating system, a downloadable application for the computing system, or any other software and / or hardware that does not deviate from the scope of the present invention. In fact, in some embodiments, the logic of the listener is partially or completely implemented by physical hardware.
[0118] On the server side 902, the presentation layer 933, service layer 934, and persistence layer 935 are provided, and the conductor 940 is also provided. Furthermore, the presentation layer 933 includes a web application 942, an Open Data Protocol (oData) Representative State Transfer (REST) Application Programming Interface (API) endpoint 944, and notification and monitoring 946, while the service layer 934 includes the API implementation / business logic 948. Thus, as shown in Figure 9, the conductor 940 includes the web application 942, the oData REST API endpoint 944, notification and monitoring 946, and the API implementation / business logic 948. The persistence layer 935 includes a database server 950, an AI / ML server 960, and an indexer server 970.
[0119] In some embodiments, communication between the execution agent 914 and the conductor 940 is always initiated by the execution agent 914. In a notification scenario, the execution agent 914 may open a WebSocket channel which is then used by the conductor 940 to send commands (e.g., start, stop, etc.) to the RPA robot. Note that, although not shown here to reduce complexity in Figure 9, the AI agent can also interact with the conductor 940, as described above with respect to Figures 1 and 8, for example. The conductor 940 can orchestrate the actions of the AI agent, AOP, and RPA robot.
[0120] In this embodiment, all messages are recorded in the conductor 940, which further processes them via the database server 950, the AI / ML server 960, the indexer server 970, or any combination thereof. As discussed herein and as illustrated with reference to Figure 8, the executor 912 may be a robot component.
[0121] In some embodiments, the RPA robot represents an association between a machine name and a username. The robot may manage multiple executors simultaneously. On a computing system that supports multiple concurrently running interactive sessions (e.g., Windows® Server 2012), multiple RPA robots may run simultaneously, each within a separate Windows® session using a unique username.
[0122] The conductor 940 can also facilitate interaction between AI agents and AI / ML models via the AI / ML server 960, which can store and / or facilitate access to generated AI models (e.g., generated AI model 172 in Figure 1).
[0123] In some embodiments, most actions performed by the user on the conductor 940 interface (e.g., via the browser 981) are performed by calling various APIs. Such actions may include, but are not limited to, starting jobs on robots, adding / deleting data in queues, and scheduling jobs to run unattended, without departing from the scope of the invention. The web application 942 is the visual layer of the server platform. In this embodiment, the web application 942 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, without departing from the scope of the invention, any desired markup language, scripting language, or any other form may be used. In this embodiment, the user interacts with the web pages of the web application 942 via the browser 941 to perform various actions to control the conductor 940. For example, the user may create RPA robot groups, assign packages to RPA robots, analyze logs per RPA robot and / or per process, and start and stop RPA robots.
[0124] In addition to the web application 942, the conductor 940 also includes a service layer that exposes an oData REST API endpoint 944. However, other endpoints may also be included without departing from the scope of the present invention. In this embodiment, the REST API is utilized by both the web application 942 and the execution agent 914. In this embodiment, the execution agent 914 is a supervisor of one or more robots on a client computer.
[0125] The REST API in this embodiment covers configuration, logging, monitoring, and queuing functions. Configuration endpoints may be used in some embodiments to define and configure application users, permissions, robots, assets, releases, and environments. Logging REST endpoints may be used to log different information, such as errors, explicit messages sent by robots, and other environment-specific information. Deployment REST endpoints may be used by robots to query the package version to be executed when a start job command is used in conductor 940. Queuing REST endpoints may be responsible for managing queues and queue items, such as adding data to the queue, retrieving transactions from the queue, and setting the state of transactions.
[0126] A monitoring REST endpoint may monitor the web application 942 and the execution agent 914. The notification and monitoring API 946 may be a REST endpoint used for registering the execution agent 914, delivering configuration settings to the execution agent 914, and sending / receiving notifications from the server and the execution agent 914. In some embodiments, the notification and monitoring API 946 may also use WebSocket communication. As shown in Figure 9, one or more activities / actions described herein are represented by arrows 949.
[0127] In some embodiments, APIs within the service layer 934 may be accessed through the configuration of an appropriate API access path, for example, based on whether the conductor 940 and the overall hyperautomation system are deployed on-premises or cloud-based. The API for the conductor 940 may provide custom methods for querying statistics about various entities registered with the conductor 940. In some embodiments, each logical resource may be an oData entity. In such entities, components such as robots, processes, and queues may have characteristics, relationships, and operations. In some embodiments, the conductor 940's API may be utilized by the web application 942 and / or execution agent 914 in two ways: (1) by obtaining API access information from the conductor 940, or (2) by registering an external application to use the oAuth flow.
[0128] In this embodiment, the persistence layer 935 includes three servers: a database server 950 (e.g., an SQL server), an AI / ML server 960 (e.g., a server providing AI / ML model serving services such as AI Center functions), and an indexer server 970. In this embodiment, the database server 950 stores the configurations of robots and AI agents, groups of robots and AI agents, AOP, associated processes, users, roles, schedules, etc. In some embodiments, this information is managed through a web application 942. The database server 950 may manage queues and queue items. In some embodiments, the database server 950 may store messages recorded by robots and AI agents (in addition to or instead of the indexer server 970). The database server 950 may also store process mining, task mining, and / or task capture-related data received, for example, from a listener 930 installed on the client side. Although no arrow is shown between listener 930 and database 950, it should be understood that in some embodiments, listener 930 can communicate with database 950, and vice versa. This data may be stored in the form of PDDs, images, XAML files, etc. Note that structured data and / or unstructured data may be stored. Listener 930 may be configured to intercept user actions, processes, tasks, and performance metrics on the computing system in which listener 930 resides. For example, listener 930 may record user actions on the computing system (e.g., clicks, typed characters, location, application, active element, time, etc.) and then convert them into a suitable format for being provided to and stored by database server 950.
[0129] The AI / ML Server 960 facilitates the integration of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options make such functionality accessible even to non-data scientists. Deployed automations (e.g., RPA robots and / or AI agents) can invoke AI / ML models from the AI / ML Server 960. The performance of AI / ML models can be monitored and trained and improved using user-validated data. The AI / ML Server 960 can schedule and execute training jobs to train new versions of AI / ML models. The AI / ML model server can also store and / or access generated AI models.
[0130] The AI / ML server 960 can store AI / ML models and ML-related data. The AI / ML server 960 can store packages for configuring various ML skills for users during development. As used herein, ML skills refer to, for example, pre-built and trained ML models for a given process that can be used by automation. The AI / ML server 960 can also store data related to document understanding techniques and frameworks, as well as algorithms and software packages for various AI / ML capabilities, including but not limited to intent analysis, NLP, speech analysis, and various types of AI / ML models.
[0131] The indexer server 970 is optional in some embodiments and stores and indexes information recorded by the robot. In certain embodiments, the indexer server 970 can be disabled through configuration settings. In some embodiments, the indexer server 970 uses ElasticSearch®, an open-source full-text search engine. Messages recorded by the robot (e.g., using activity such as log messages or light lines) can be sent to the indexer server 970 via a logging REST endpoint, where they are indexed for future use.
[0132] Figure 10 is an architectural diagram showing the relationships between Designer 1010, Activities 1020, 1030, 1040, 1050, Driver 1060, API 1070, and AI / ML Model 1080 in one or more embodiments. As described above, an AI agent, AOP, or developer uses Designer 1010 to develop workflows for automation (e.g., workflows executed by RPA robots, AI agents, and AOPs).
[0133] An AI agent, AOP, or developer may design and configure a workflow 1092, an agent-based automation 1094 for the AI agent (e.g., providing natural language description, context grounding, tools, etc., for the AI agent), and an AOP 1096 (see also Figures 4B, 5, and 6). Various types of activities may be displayed to the developer in some embodiments. The designer 1010 may be local to the user's computing system or remote to it (e.g., accessed via a VM, or via a local web browser interacting with a remote web server). A workflow for an RPA robot may include a user-defined activity 1020, an API-driven activity 1030, an AI / ML activity 1040, and / or a UI automation activity 1050. The user-defined activity 1020 and the API-driven activity 1040 interact with the application via their APIs. In some embodiments, a user-defined activity 1020 and / or an AI / ML activity 1040 may invoke one or more AI / ML models 1080, which may be located locally with respect to the computing system on which the robot is operating, or remotely with respect to it.
[0134] In some embodiments, it is possible to identify non-textual visual elements within an image, which is referred to herein as CV. However, it should be noted that in some embodiments, CV includes OCR. CV may be performed, at least in part, by the AI / ML model 1080. Some CV activities related to such elements may include, but are not limited to, text extraction from segmented label data using OCR, fuzzy text matching, cropping of segmented label data using ML, and comparison of extracted text in the label data with ground truth data. In some embodiments, the number of activities that may be implemented in user-defined activity 1020 may be in the hundreds or thousands. However, any number and / or types of activities may be used without departing from the scope of the invention.
[0135] The UI automation activity 1050 is a subset of special low-level activities written in low-level code and designed to facilitate interaction with a screen. The UI automation activity 1050 facilitates these interactions via drivers 1060, which enable the robot to interact with desired software. For example, drivers 1060 may include an operating system (OS) driver 1062, a browser driver 1064, a VM driver 1066, an enterprise application driver 1068, and so on. In some embodiments, one or more AI / ML models 1080 may be used by the UI automation activity 1050 to perform interactions with a computing system. In certain embodiments, the AI / ML models 1080 may complement or completely replace the drivers 1060. In fact, in certain embodiments, drivers 1060 are not included.
[0136] Driver 1060 can interact with the OS at a low level via OS driver 1062, for example, to look for hooks and monitor keys. Driver 1060 can facilitate integration with applications such as Chrome®, IE®, Citrix®, and SAP®. For example, a "click" activity can perform the same role in these different applications via driver 1060.
[0137] Figure 11 is an architectural diagram showing a computing system 1100 according to one or more embodiments. In some embodiments, the computing system 1100 may be one or more computing systems described and / or described herein. In some embodiments, the computing system 1100 may be one or more computing systems described and / or described herein. In certain embodiments, the computing system 1100 may be part of a hyperautomation system, such as those shown in Figures 1 and 8. The computing system 1100 includes a bus 1105 or other communication mechanism for communicating information and a processor 1110 coupled to the bus 1105 for processing information. The processor 1110 may be any type of general-purpose or purpose-specific processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), multiple entities thereof, and / or any combination thereof. The processor 1110 may also have multiple processing cores, and at least some of these cores may be configured to perform specific functions. In some embodiments, multi-parallel processing may be used. In certain embodiments, at least one of the processors 1110 may be a neuromorphic circuit that includes processing elements that mimic biological neurons. In some embodiments, the neuromorphic circuit may not require components of a typical von Neumann computing architecture.
[0138] The computing system 1100 further includes memory 1115 for storing information and instructions executed by the processor 1110. Memory 1115 may consist of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage devices such as magnetic disks or optical disks, or any other type of non-temporary computer-readable medium, or any combination thereof. The non-temporary computer-readable medium may be any available medium accessible by the processor 1110, and may include volatile media, non-volatile media, or both. The medium may be removable, non-removable, or both. The computing system 1100 also includes communication devices 1120, such as transceivers, for providing access to a communication network via wireless and / or wired connections. In some embodiments, the communication devices 1120 may include one or more antennas, which are single, arrayed, phase-controlled, switching, beamforming, beam-steering, combinations thereof, or any other antenna configuration that does not depart from the scope of the present invention.
[0139] The processor 1110 is further coupled to the display 1125 via the bus 1105. Any suitable display device and haptic I / O may be used without departing from the scope of the invention. A keyboard 1130 and a cursor control device 1135, such as a computer mouse or touchpad, are further coupled to the bus 1105, enabling the user to interface with the computing system 1100. However, in certain embodiments, there is no physical keyboard and mouse, and the user may interact with such devices only through the display 1125 and / or touchpad (not shown). Any type and combination of input devices may be used as a matter of design selection. In certain embodiments, there is no physical input device and / or display. For example, the user may interact with the computing system 1100 remotely via another computing system that is communicably connected, or the computing system 1100 may operate autonomously.
[0140] Memory 1115 stores software modules that provide functionality when executed by processor 1110. These modules include an operating system 1140 for computing system 1100.
[0141] The module further includes a model selection module 1145 configured to perform all or part of the processes described herein, or derivatives thereof. For example, the model selection module 1145 may perform agent-based automation with agent model selection, which includes context grounding and benchmarking against multiple model capabilities used to select a specific model for each step of the workflow that requires an AI model. The model selection module 1145 may include context grounding for selecting a specific model for each step of the workflow performed in multimodal agent-based automation. Furthermore, the model selection module 1145 may include using benchmarks and historical job data, in addition to the context grounding, to select a model for each step of the workflow. The model selection module 1145 may select the best model for each step that requires an AI model. Prior to the best model selection, the model selection module 1145 may perform self-benchmarking, access an artificial intelligence (AI) center, and run within a sandbox. Therefore, the model selection module 1145 provides improved control over the multimodal agent-based automation to achieve technical effects, benefits, and advantages such as improved workflow processing, increased step execution efficiency, and prevention of hallucination due to incorrectly selected AI models. Thus, the model selection module 1145 generates a more intelligent and context-aware hyperautomation system.
[0142] The module further includes an agent memory module 1147 configured to perform all or part of the processes described herein, or derivatives thereof, including, but not limited to, performing agent memory for context grounding and other results for automation. For example, the agent memory module 1147 is configured to dynamically cache (i.e., store) escalations, tool calls, user interactions or feedback, context grounding, etc., in order to improve efficiency and minimize calls.
[0143] The computing system 1100 may include one or more additional function modules 1150 that include additional functions. For example, the one or more additional function modules 1150 may include an extract and search module configured to perform advanced agent-based extract and search for context grounding within the automation.
[0144] Those skilled in the art will understand that “computing system” can be embodied as a server, embedded computing system, personal computer, console, personal digital assistant (PDA), mobile phone, tablet computing device, smartwatch, quantum computing system, or any other suitable computing device, or any combination of devices that does not depart from the scope of the present invention. The fact that the functions described above are presented as being performed by a “system” is not intended to limit the scope of the present invention in any way, but rather to provide an example of one of the many embodiments of the present invention. In fact, the methods, systems, and apparatus disclosed herein can be implemented in localized and distributed forms that are compatible with computing technologies, including cloud computing systems. The computing system may be part of, or accessible by, a LAN, mobile communication network, satellite communication network, internet, public or private cloud, hybrid cloud, server farm, or any combination thereof. Any localized or distributed architecture may be used without departing from the scope of the present invention.
[0145] It should be noted that some of the system features described herein are presented as modules to more specifically emphasize their implementation independence. For example, modules may be implemented as hardware circuits including custom very large-scale integrated circuits (VLSI) or off-the-shelf semiconductors such as gate arrays, logic chips, transistors, or other discrete components. Modules may also be implemented in programmable hardware devices such as field-programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, or similar.
[0146] Modules can also be implemented, at least partially, in software executed by various types of processors. For example, an identified unit of executable code may include one or more physical or logical computer instruction blocks, which may be organized as, for example, objects, procedures, or functions. Nevertheless, the executable code of an identified module does not necessarily have to be physically located in one place, and may include distributed instructions stored in different locations, which, when logically combined, constitute the module and achieve the purposes described for that module. Furthermore, modules may be stored in a computer-readable medium, which may be, for example, a hard disk drive, a flash device, RAM, tape, and / or any other such non-temporary computer-readable medium used to store data without departing from the scope of the present invention.
[0147] In fact, a module of executable code may be a single instruction, a number of instructions, or even distributed across multiple different code segments, between different programs, and across multiple memory devices. Similarly, operational data may be identified and illustrated within a module herein, embodied in any suitable form, and organized within any suitable type of data structure. Such operational data may be collected as a single dataset, or distributed in different locations, including on different storage devices, and may exist at least partially only as electronic signals on a system or network.
[0148] Various types of AI / ML models can be trained and deployed without departing from the scope of the present invention. For example, Figure 12A shows an example of a neural network 1200 trained to receive inputs (represented in column 1210) to input "neurons" 1-I in an input layer (represented in column 1220) according to one or more embodiments. The neural network 1200 includes several hidden layers (represented in columns 1230 and 1240). Both DLNNs and shallow learning neural networks (SLNNs) typically have multiple layers, although SLNNs may have only one or two layers in some cases, and usually fewer than DLNNs. Typically, a neural network architecture includes an input layer, several intermediate layers (e.g., hidden layers), and an output layer (represented in column 1250), and the neural network 1200 also has such a configuration.
[0149] DLNNs often have a large number of layers (e.g., 10, 50, 200, etc.), and subsequent layers typically reuse features from previous layers to compute more complex and general functions. SLNNs, on the other hand, tend to have fewer layers and are trained relatively quickly because specialized features are pre-created from raw data samples. However, feature extraction is labor-intensive. DLNNs, on the other hand, typically do not require specialized features, but training takes longer and they tend to have more layers.
[0150] In both approaches, each layer is typically trained simultaneously on a training set while overfitting is checked on an independent cross-validation set. Both techniques can yield excellent results, and there is considerable enthusiasm for both approaches. The optimal size, shape, and number of individual layers vary depending on the problem each neural network addresses.
[0151] Returning to Figure 12A, the input layer provides company-specific information, including diverse industries and applications, unique industry terminology, complex document structures, and pre-trained knowledge, which are then supplied as inputs to J neurons in Hidden Layer 1. Various other inputs are also possible, including, but are not limited to, computing system state information, published automations, business rules, information about which workflows and / or tasks are relevant, initial automation definitions, and process automation documentation. In this example, all of these inputs are fed to each neuron, but various architectures are possible in which they can be used individually or in combination, including feedforward networks, radial basis networks, deep feedforward networks, deep convolutional inverse graphics networks, convolutional neural networks, recurrent neural networks, artificial neural networks, long short-term memory networks, gated recurrent unit networks, generative adversarial networks, liquid state machines, autoencoders, variational autoencoders, denoising autoencoders, sparse autoencoders, extreme learning machines, echo state networks, Markov chains, Hopfield networks, Boltzmann machines, restricted Boltzmann machines, deep residual networks, Kohonen networks, deep belief networks, deep convolutional networks, support vector machines, neural Turing machines, or any other suitable type or combination of neural networks that do not deviate from the scope of the present invention.
[0152] Hidden layer 2 (1240) receives input from hidden layer 1 (1230), hidden layer 3 receives input from hidden layer 2 (1240), and so on for all hidden layers up to the last hidden layer (represented by the ellipsis 1255), which provides its output as input to the output layer. Although multiple recommendations are shown as outputs here, in some embodiments only a single output recommendation is provided. In certain embodiments, recommendations are ranked based on confidence scores. In this embodiment, the output is an exact response to a specialized or recent query.
[0153] It should be noted that the number of neurons I, J, K, and L are not necessarily equal. Therefore, any desired number of layers can be used for a given layer of the neural network 1200 without departing from the scope of the invention. In fact, in certain embodiments, the types of neurons in a given layer may not all be identical.
[0154] The neural network 1200 is trained to assign confidence scores to appropriate outputs. To reduce inaccurate predictions, in some embodiments, only results with confidence scores that meet or exceed a confidence threshold may be provided. For example, if the confidence threshold is 80%, outputs with confidence scores exceeding this value may be used, and the rest may be ignored. According to one or more embodiments, the output layer 1250 indicates that two text fields (represented by outputs 1261 and 1262), one text label (represented by output 1263), and one submit button (represented by output 1265) were found. Without departing from the scope of one or more embodiments herein, the neural network 1200 may provide the location, dimensions, images, and / or confidence scores of these elements, which can then be used by an RPA robot or other automation that uses this output for a predetermined purpose.
[0155] A neural network is typically a probabilistic construct that has a confidence score. This can be a score learned by an AI / ML model based on how correctly it identified similar inputs during training. Common types of confidence scores include decimal numbers from 0 to 1 (which can also be interpreted as a confidence percentage), numbers from negative infinity to positive infinity, and sets of representations (e.g., "low," "medium," "high"). Various post-processing calibration techniques such as temperature scaling, batch normalization, weight decay, and negative log-likelihood (NLL) may also be used to attempt to obtain a more accurate confidence score.
[0156] In neural networks, "neurons" are typically implemented algorithmically as mathematical functions based on the functions of biological neurons. A neuron receives a weighted input and has an additive function and activation function that govern whether or not it passes the output to the next layer. This activation function can be a nonlinear thresholded activation function where nothing happens if the value is below a threshold, but the function responds linearly above the threshold (i.e., Rectified Linear Unit (ReLU) nonlinearity). The additive and ReLU functions are used in deep learning because real neurons can generally have similar activation functions. Information can be subtracted, added, etc., via linear transformations. Essentially, a neuron functions as a gating function that passes the output to the next layer, governed by its underlying mathematical function. In some embodiments, different functions may be used for at least some neurons.
[0157] An example of neuron 1295 is shown in Figure 12B. Inputs from the previous layer x1, x2, ..., x n Each of these has weights w1, w2, ..., w n This is assigned. Therefore, the collective input from the previous neuron 1 is w1x1. These weighted inputs are used for the neuron's additive function, which is modified by the following bias: m
number
[0158] This addition is compared to the activation function f(x) to determine whether or not the neuron "fires". For example, f(x) can be given by the following equation:
number
[0159] Therefore, the output y of neuron 1295 can be given by the following equation.
number
[0160] In this case, neuron 1295 is a single-layer perceptron. However, any suitable neuron type or combination of neuron types may be used without departing from the scope of the present invention. It should also be noted that in some embodiments, the range of weight values and / or the range of output values of the activation function may differ.
[0161] A goal, or "reward function," is often used. The reward function guides the exploration of the state space and attempts to achieve the goal (e.g., finding the most accurate answer to a user query based on relevant metrics) by exploring intermediate transitions and steps with both short-term and long-term rewards. During training, various labeled data are fed into the neural network 1200. Successful classifications strengthen the weights of the input to the neurons, while failed classifications weaken them. A cost function, such as mean squared error (MSE) or gradient descent, may be used to punish slightly incorrect predictions much less than very incorrect ones. If the performance of the AI / ML model does not improve after a certain number of training iterations, data scientists may modify the reward function and provide corrections for incorrect predictions, etc.
[0162] Backpropagation is a technique for optimizing synaptic weights in feedforward neural networks. It can be used to examine the hidden layers of a neural network to understand how much of the loss each node is responsible for, and then update the weights to minimize the loss by assigning lower weights to nodes with higher error rates, and vice versa. In other words, backpropagation allows data scientists to iteratively adjust weights to minimize the difference between the actual output and the desired output.
[0163] The backpropagation algorithm has a mathematical foundation in optimization theory. In supervised learning, training data with known outputs passes through the neural network, an error is calculated from the known target output by a cost function, which gives the error for backpropagation. The error is calculated at the output, and this error is converted into corrections to the network weights that minimize the error.
[0164] An example of backpropagation in the case of supervised learning is shown below. The column vector input x is processed through a series of N non-linear activation functions f between each layer i=1,…,N of the network i , and the output at a given layer is first multiplied by the synaptic matrix W i and added with the bias vector b i . The network output o is given by the following equation. [Mathematical formula]
[0165] In some embodiments, o is compared to the target output t, resulting in [Mathematical formula] , which is desired to be minimized.
[0166] Optimization in the form of a gradient descent procedure may be used to minimize the error by modifying the synaptic weights W for each layer i . The gradient descent procedure requires calculation of the output o given the input x corresponding to the known target output t, and generation of the error o--t. This total error is then propagated in the reverse direction, and gives local errors for weight updates by calculations similar to but not identical to those used for forward propagation. In particular, the backpropagation phase typically requires an activation function of the form p j (n j )=f j ′(n j ), where n jThis is the network activity in layer j (i.e., n j =W j o j-1 +b j ) and o j =f j (n j ) and the apostrophe ' represents the derivative of the activation function f.
[0167] Weight updates can be calculated using the following formula:
number
[0168] Here, o represents the Hadamard product (i.e., the element-wise product of two vectors), T represents the matrix transpose, o j is f j (W j o j-1 +b j ) represents and o0 = x. Here, the learning rate η is selected according to machine learning considerations. Below, η is associated with the neural Hebb learning mechanism used in the neural implementation. Note that synapses W and b may be joined to one large synaptic matrix, where it is assumed that the input vector is appended with 1 and that an additional column representing the b synapse is included in W.
[0169] The AI / ML model can be trained over multiple epochs until a good level of accuracy is achieved (e.g., 97% or higher using an F2 or F4 threshold for detection, and over approximately 2,000 epochs). This level of accuracy may, in some embodiments, be determined using an F1 score, F2 score, F4 score, or any other suitable technique that does not deviate from the scope of the invention. After training on training data, the AI / ML model can be tested on a set of evaluation data that the AI / ML model has not previously encountered. This helps to ensure that the AI / ML model is not in a state of "overfitting," where it performs well on the training data but poorly on other data.
[0170] In some embodiments, the level of accuracy that an AI / ML model can achieve may be unknown. Therefore, if the accuracy of the AI / ML model begins to decline when analyzing evaluation data (i.e., the model performs well on training data but begins to perform poorly on evaluation data), the AI / ML model may undergo further training on the training data (and / or new training data) for more epochs. In some embodiments, an AI / ML model is deployed only when its accuracy reaches a certain level, or when the accuracy of the trained AI / ML model is better than that of an existing deployed AI / ML model. In certain embodiments, a set of trained AI / ML models may be used to accomplish a task. For example, one AI / ML model may be trained to recognize images, another to recognize text, and yet another to recognize semantic and / or ontology associations.
[0171] It should be noted that in addition to, or as an alternative to, neural networks, transformer networks such as SentenceTransformers™ may be used in some embodiments. This is a state-of-the-art Python™ framework for sentence, text, and image embedding. Such transformer networks learn the associations of words and phrases that have both high and low scores. This trains the AI / ML model to determine, respectively, what is close to the input and what is not. Transformer networks may also use field length and field type, rather than simply using word / phrase pairs.
[0172] As described above, NLP models such as word2vec, BERT, GPT-3, ChatGPT, and other LLMs can be used in several embodiments to facilitate semantic comprehension and provide more accurate and human-like responses. Other techniques, such as clustering algorithms, can also be used to find similarities between sets of elements. Clustering algorithms may include, but are not limited to, density-based, distribution-based, centroid-based, and hierarchy-based algorithms. Examples include K-means clustering, DBSCAN clustering, Gaussian mixture model (GMM) algorithms, and balanced iterative reduction and clustering with hierarchy (BIRCH) algorithms. Such techniques can also assist in categorization.
[0173] Figure 13 is an architecture diagram showing a reference architecture 1300 of a generative AI model in one or more embodiments. This architecture consists of multiple layers, namely, an API plugin, a prompt library, vector data source ingestion, access processing control, a model training pipeline, an assessment layer for evaluating hallucination / telemetry / evaluation, a BYOM embedding layer, and an LLM orchestration layer. There are also acquisition plugins, access control plugins, and API plugins that are integrated into enterprise systems.
[0174] Reference Architecture 1300 has three main flows.
[0175] Flow 1: Ingestion may include data ingestion and training flows. For example, data is read from multiple data stores, preprocessed, chunked, and trained through an embedded model (e.g., RAG and training pipeline, i.e., fine-tuning). A vector database stores chunked document embeddings (e.g., vector embeddings) that enable better semantic and similarity-based data retrieval.
[0176] Flow 2: Acquisition may include prompt extension using a data acquisition flow. For example, when a user query arrives at the API layer, a prompt is selected, and then, before the prompt is passed to the LLM layer, data acquisition occurs through a vector database or API plugin to obtain the appropriate contextual data.
[0177] Flow 3: Inference may include the LLM inference flow. For example, here a choice is made whether to use a general-purpose foundational model or a self-hosted foundational model. Fine-tuned models may be used if they are tailored to a specific task or use case. The response is evaluated for accuracy, including hallucination, and other metrics.
[0178] It should be noted that in some embodiments, generative AI models having multiple "heads" may be used. A head refers to the output layer of a generative AI model. A generative AI model like generative AI model 172 in Figure 1 typically has a sequence of multiple layers, and each head often shares the first few layers of the model before branching off into its own unique layers.
[0179] Figure 14 is a flowchart of a process 1400 for training an AI / ML model according to one or more embodiments. In some embodiments, the AI / ML model may be a generative AI model as described above. In the case of a neural network, the architecture typically includes multiple neuron layers, including an input layer, an output layer, and hidden layers. See, for example, Figures 12A and 12B. The hidden layers located between these generate an intermediate representation of the input used to process the input data and generate the output. These hidden layers may include various types of neurons, such as convolutional neurons, recurrent neurons, and / or transformer neurons. A generative AI model may also have various layers.
[0180] The process 1400 performed in Figure 14 may be executed by automation as described herein, which is implemented in a computer program according to one or more embodiments. The computer program may be implemented on a non-temporary computer-readable medium. The computer-readable medium may include, but is not limited to, a hard disk drive, a flash device, RAM, tape, and / or any other such medium or combination of mediums used to store data. The computer program may include encoded instructions for controlling a processor of a computing system (see, for example, Figure 11) in order to implement all or part of the process 1400 shown in Figure 14, and such instructions may also be stored on the computer-readable medium.
[0181] In some embodiments, the training process begins in block 1410 by providing data, labeled or unlabeled, such as company-specific information with diverse industry and application data, unique industry terminology, complex document structures, and pre-trained knowledge. For generative AI models, since they are generally extensively trained, the training process may be omitted unless a fine-tuned model is desired, as will be detailed later. The AI / ML model is then trained over multiple epochs in block 1420, and the results are reviewed in block 1430. While various types of AI / ML models can be used, LLMs and other generative AI models are typically trained (fine-tuned) using a process called "supervised learning," which was also discussed above. Supervised learning involves providing the model with a large dataset, which it uses to learn the relationships between inputs and outputs. During the training process, the model adjusts the weights and biases of neurons in its neural network to minimize the difference between predicted outputs and actual outputs in the training dataset.
[0182] One aspect of the model in some embodiments is the use of transfer learning. For example, transfer learning can leverage a pre-trained model such as ChatGPT, which is fine-tuned for a specific task or domain in block 1420. This allows the model to leverage knowledge already learned from the pre-training stage and adapt it to a specific application through the training stage in block 1420.
[0183] The pre-training phase involves training the model on a more general set of initial training data. During this phase, the model learns relationships within the data. In the fine-tuning phase (which, in some embodiments, is performed in addition to or instead of the initial training phase, during block 1420, for example, when the pre-trained model is used as the initial foundation for the final model), the pre-trained model is adapted to a specific task or domain by training the model on a smaller dataset specific to that task. For example, in some embodiments, the model may focus on one or more data sources. This may enable it to more accurately identify data elements within them than a generative AI model that is only pre-trained. Fine-tuning allows the model to learn the nuances of the sources, such as specific vocabulary and syntax, specific geometric properties, specific data formats, etc., without requiring as much data as would be needed to train the model from scratch. By leveraging the knowledge learned in the pre-training phase, the fine-tuned model can achieve state-of-the-art performance for a particular task with relatively little additional training data.
[0184] In some embodiments, if the AI / ML model does not meet the desired confidence threshold in decision block 1440, process 1400 proceeds to block 1450 (as indicated by the NO arrow). Training data is supplemented and / or the reward function is modified to help the AI / ML model better achieve its goal, and then process 1400 returns to block 1420.
[0185] If the AI / ML model meets the confidence threshold in decision block 1440, process 1400 proceeds to block 1460 (as indicated by the YES arrow). The AI / ML model is tested on evaluation data in block 1460 to ensure that the AI / ML model generalizes well and does not overfit with respect to the training data. The evaluation data includes information that the AI / ML model has not previously processed.
[0186] If the confidence threshold is met in decision block 1470 for the evaluation data, process 1400 proceeds to block 1480 (as indicated by the YES arrow). The AI / ML model is deployed in 1480. Otherwise, process 1400 returns to block 1450 (as indicated by the NO arrow), and the AI / ML model is further trained.
[0187] Referring next to Figure 15, one or more embodiments of process 1500 are shown. Generally, process 1500 is a method for selecting agent models in a hyperautomation system that performs one or more agent-type automations using benchmarks. Process 1500 may be performed by the automation described herein, which is implemented in a computer program according to one or more embodiments. The computer program may be embodied on a non-temporary computer-readable medium (e.g., a memory that stores computer code for the one or more agent-type automations and the one or more artificial intelligence agents). The computer program may include encoded instructions for controlling a processor of a computing system (e.g., see Figure 11) (e.g., at least one processor that executes the computer code to select and assign one or more artificial intelligence models to improve the one or more agent-type automations) to implement all or part of process 1500 as described in Figure 15, and such instructions may also be stored on the computer-readable medium.
[0188] Process 1500 begins in block 1510 with the hyperautomation system tracking the current workflow execution. The current workflow may consist of two or more steps. These two or more steps may include at least one model request step, which is an activity that requires an AI model (e.g., an LLM or other AI model).
[0189] According to one or more embodiments, the hyperautomation system analyzes the current workflow to identify each step of the workflow that requires an AI model, and stores such step in the agent memory of the hyperautomation system as a model request step with the current result.
[0190] In block 1530, the hyperautomation system performs matching. The hyperautomation system may match the current workflow to two or more previous workflows. These two or more previous workflows may be stored in agent memory. The hyperautomation system may match the current workflow to two or more previous workflows based on the current result of a model request step. In this way, the hyperautomation system can benchmark the behavior of the current workflow against previous workflows. For example, these two or more previous workflows are stored in agent memory for access by the hyperautomation system. Furthermore, matching the current workflow to the two or more previous workflows based on the current result of a model request step involves utilizing one or more datasets detailing the agents and results of the two or more previous workflows. These one or more datasets may be located in the agent memory of the artificial intelligence center. The artificial intelligence center is a software service or component of the hyperautomation system that enables the deployment, management, and continuous improvement of AI models, AI agents, and agent-based automation.
[0191] In block 1550, the hyperautomation system generates two or more benchmark results and capabilities by comparing the current result with the corresponding result of the two or more previous workflows. For example, the hyperautomation system generates the two or more benchmark results and capabilities by comparing the model request step of the current workflow with two or more model request steps of the two or more previous workflows. The hyperautomation system may also compare and utilize context, predictive, or historical data, for example, by comparing one or more historical information data instances in the artificial intelligence center that detail the operational results of the two or more previous workflows. The one or more historical information data instances, along with other data (such as contextual data, predictive data, etc., as described herein), may be used for agent-based benchmarking (for example, if benchmarks and other information are available to the agent, the agent can perform the benchmarking itself). According to one or more embodiments, the actor agent of the one or more artificial intelligence agents that execute the model request step of the current workflow performs self-benchmarking by matching the current workflow with the two or more previous workflows and comparing the current result with the two or more benchmark results and capabilities. This includes contextual, predictive, or historical data. Please send the next chunk.
[0192] In block 1570, the hyperautomation system accesses the Artificial Intelligence Center. The hyperautomation system accesses the Artificial Intelligence Center to start and run sandboxes for the one or more Artificial Intelligence Models, the one or more Artificial Intelligence Agents, and the one or more agent-based automations. As described herein, a sandbox is a software mechanism for isolating running programs / agents / models / automations to mitigate system failures, errors, and / or software vulnerabilities and to determine scores. Within the sandbox of the Artificial Intelligence Center of the hyperautomation system, the two or more benchmark results and capabilities may be generated by systematic testing and performance evaluation across the entire task and dataset of agent memory related to previous workflows. Access to the Artificial Intelligence Center and execution within the sandbox generate the benchmark results which may be used for selection and allocation.
[0193] In block 1590, the hyperautomation system performs selection and assignment. That is, the hyperautomation system selects and assigns one or more artificial intelligence models from the artificial intelligence center based on the best benchmark result among the two or more benchmark results and capabilities. The best benchmark result includes the result with the highest score among the two or more benchmark results and capabilities. That is, each benchmark result may be assigned a score such as a decimal number from 0 to 1 (which can also be interpreted as a confidence percentage) that identifies the efficiency and / or accuracy of the agent / model, etc. According to one or more embodiments, the artificial intelligence model corresponds to the model required in the model request step of a previous workflow among the two or more previous workflows. Note that the one or more agent-type automations include artificial intelligence-enhanced stochastic automation performed by the one or more AI agents that operate independently, dynamically, and adaptively to perform the selection and assignment of the one or more artificial intelligence models. According to one or more technical effects, benefits, and advantages, the one or more AI models are trained in a more generalized manner (e.g., resource-efficient) to adapt to various tasks with minimal fine-tuning, unlike conventional automation which requires extensive feature engineering (e.g., resource-intensive). A hyper-automation system may select one or more AI models based on one or more model capabilities, one or more model performance, one or more model applications, or one or more of their benchmark capabilities.
[0194] For example, the one or more model capabilities include data processing and analysis, forecasting and prediction, decision-making and automation, and content generation. Data processing and analysis may include, but are not limited to, pattern recognition (e.g., an AI model identifying patterns and relationships in data to enable informed decision-making), data interpretation (e.g., an AI model interpreting complex datasets to extract significant insights and trends that humans might miss), data cleaning and preprocessing (e.g., an AI model automating tasks and ensuring data quality and accuracy), and data management (e.g., an AI model managing large datasets for access and analysis beyond human capabilities). Forecasting and prediction may include, but are not limited to, predictive modeling (e.g., an AI model predicting future outcomes based on historical data to enable agents to make informed decisions) and time series analysis (e.g., an AI model analyzing time-based data to identify trends and patterns to facilitate forecasting and planning). Decision-making and automation may include, but are not limited to, automated decision-making (e.g., AI models automating tasks and decision-making processes to improve efficiency and reduce human error) and robotics / control systems (e.g., AI models controlling and automating automated systems to perform complex tasks). Content generation may include, but are not limited to, image generation (e.g., AI models generating images and artwork), text generation (e.g., AI models generating human-like text including articles, stories, and code), and code generation (e.g., AI models generating code snippets and / or programs).
[0195] Furthermore, the performance of one or more models may include the resource efficiency of the hyperautomation system (e.g., CPU and RAM usage). The performance of the models may also include output accuracy (e.g., the ability of the LLM output to unknown data) and / or the ability to correct the output.
[0196] The one or more model applications of the one or more AI models include a set of real-world applications that put the one or more AI models into practical use. For example, the one or more model applications may include natural language processing, chatbot services, customer service, machine translation, sentiment analysis, content generation, computer vision, object detection, machine learning, error detection, predictive analytics, and the like.
[0197] The benchmark capabilities of one or more AI models include systematic testing and performance evaluation across tasks and datasets. Furthermore, the benchmark capabilities may provide a standardized method for comparing the strengths and requirements of the one or more AI models. For example, the benchmark capabilities may include time constraints (e.g., more options vs. more resource / time savings) for users to attempt automation to eliminate the debugging process for each iteration.
[0198] Furthermore, in subblock 1595, the hyperautomation system may perform agent selection for each step of the workflow (whether or not an AI model is required). In this regard, the hyperautomation system may select one or more AI agents using multiple model capabilities and benchmark contexts. The hyperautomation system may select a first AI agent for the step, a second AI agent for AI model selection, and utilize the second AI agent to select an AI model for the first AI agent. The hyperautomation system may make the selection of the first AI agent, the second AI agent, and the AI model based on benchmark and historical job data, as well as available hardware and memory, browser or application versions, etc. Note that historical data enables the hyperautomation system and its internal AI agents to make decisions. That is, if benchmarks and information are available to the first and second AI agents, the first and second AI agents may perform benchmarks themselves and provide access within the AI center or sandbox.
[0199] The computer program may be implemented in hardware, software, or a hybrid implementation. The computer program may consist of modules designed to communicate with each other in an operable manner and to pass information or instructions to a display. The computer program may be configured to run on a general-purpose computer, an ASIC, or any other suitable device.
[0200] It will be readily apparent that the components of various embodiments of the present invention can be arranged and designed in a wide variety of different configurations, as generally described herein and illustrated in the drawings. Therefore, the detailed description of embodiments of the present invention shown in the accompanying drawings is not intended to limit the claimed scope of the invention, but merely to represent selected embodiments of the invention.
[0201] The features, structures, or characteristics of the present invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, any reference throughout this specification to “specific embodiments,” “several embodiments,” or similar phrases means that the specific features, structures, or characteristics described in relation to such embodiments are included in at least one embodiment of the present invention. Therefore, wherever phrases such as “in specific embodiments,” “several embodiments,” “in other embodiments,” or similar phrases appear throughout this specification, they do not necessarily refer to the same group of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0202] It should be noted that references to features, advantages, or similar terms throughout this specification do not imply that all features and advantages that can be realized by the present invention should or are provided in a single embodiment of the invention. Rather, terms referring to features and advantages should be understood to mean that certain features, advantages, or characteristics described in relation to embodiments are included in at least one embodiment of the invention. Accordingly, discussions of features and advantages, and similar terms throughout this specification may, but do not necessarily, refer to the same embodiment.
[0203] Furthermore, the described features, advantages, and characteristics of the present invention can be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the present invention may be carried out without one or more specific features or advantages of a particular embodiment. In other examples, additional features and advantages that may not be present in all embodiments of the present invention may be recognized in a particular embodiment.
[0204] Those skilled in the art will readily understand that the present invention discussed above may be implemented using hardware elements in a different order of steps and / or configurations different from those disclosed. Therefore, although the present invention is described based on these preferred embodiments, certain modifications, changes, and alternative configurations will be apparent to those skilled in the art, while remaining within the spirit and scope of the invention. Accordingly, to define the technical scope of the present invention, one should refer to the appended claims.
Claims
1. A method for a hyperautomation system that performs one or more agent-based automations using benchmarks, The hyperautomation system includes a memory for storing computer code for one or more agent-type automation and one or more artificial intelligence agents, The hyperautomation system includes at least one processor that executes the computer code to select and assign one or more artificial intelligence models in order to improve the one or more agent-type automation, The aforementioned method, Tracking the current workflow execution, including remembering the current result of the model request step in the current workflow, Based on the current result of the aforementioned model request step, the current workflow is matched to two or more previous workflows, To compare the current result with the corresponding result of two or more previous workflows to generate two or more benchmark results and capabilities, The artificial intelligence models corresponding to the model request step of a previous workflow among the two or more previous workflows, and the selection and assignment of one or more artificial intelligence models from the artificial intelligence center based on the best benchmark result among the two or more benchmark results and capabilities, including context, prediction, or historical data. Methods that include...
2. The method according to claim 1, wherein the hyperautomation system compares the model request step of the current workflow with two or more model request steps of two or more previous workflows to generate two or more benchmark results and capabilities.
3. The method according to claim 1, wherein the two or more prior workflows are stored in agent memory for access by the hyperautomation system.
4. The method according to claim 1, wherein the current workflow includes two or more steps, the model request step which is an activity that requires the artificial intelligence model.
5. The method according to claim 1, wherein the one or more agent-type automation includes artificial intelligence-enhanced stochastic automation performed by the one or more AI agents that operate independently, dynamically, and adaptively to perform the selection and assignment of the one or more artificial intelligence models.
6. The method according to claim 1, wherein the two or more benchmark results and capabilities include systematic testing and performance evaluation across tasks and datasets within a sandbox of the hyperautomation system.
7. The method according to claim 1, wherein the actor agent of one or more artificial intelligence agents that execute the model request step of the current workflow performs self-benchmarking by matching the current workflow to two or more previous workflows and comparing the current results to generate two or more benchmark results and capabilities.
8. The method according to claim 1, wherein matching the current workflow to the two or more previous workflows based on the current result of the model request step includes utilizing one or more datasets in the artificial intelligence center detailing the agents and results of the two or more previous workflows.
9. The method according to claim 1, wherein generating the two or more benchmark results and capabilities by comparing the current result with the corresponding results of the two or more previous workflows includes utilizing one or more historical information data instances in the artificial intelligence center that detail the operational results of the two or more previous workflows.
10. The method according to claim 1, wherein the best benchmark result includes the result that includes the highest score among the two or more benchmark results and capabilities.
11. It is a hyper-automation system, Memory for storing computer code for one or more agent-based automations, for one or more artificial intelligence agents, and for performing the one or more agent-based automations using benchmarks, At least one processor that executes the computer code to cause the hyperautomation system to perform the following actions in order to select and assign one or more artificial intelligence models in order to improve the one or more agent-type automation: Includes, The above and subsequent statements are as follows: Tracking the current workflow execution, including storing the current result of the model request step of the current workflow; Based on the current result of the aforementioned model request step, the current workflow is matched to two or more previous workflows, To compare the current result with the corresponding result of two or more previous workflows to generate two or more benchmark results and capabilities, The artificial intelligence models corresponding to the model request step of a previous workflow among the two or more previous workflows, and the selection and assignment of one or more artificial intelligence models from the artificial intelligence center based on the best benchmark result among the two or more benchmark results and capabilities, including context, prediction, or historical data, A hyper-automation system that includes this.
12. The hyperautomation system according to claim 11, wherein the hyperautomation system compares the model request step of the current workflow with two or more model request steps of two or more previous workflows to generate two or more benchmark results and capabilities.
13. The hyperautomation system according to claim 11, wherein the two or more prior workflows are stored in agent memory for access by the hyperautomation system.
14. The hyperautomation system according to claim 11, wherein the current workflow includes two or more steps, the model request step which is an activity that requires the artificial intelligence model.
15. The hyperautomation system according to claim 11, wherein the one or more agent-type automation includes artificial intelligence-enhanced probabilistic automation performed by the one or more AI agents that operate independently, dynamically, and adaptively to perform the selection and assignment of the one or more artificial intelligence models.
16. The hyperautomation system according to claim 11, wherein the two or more benchmark results and capabilities include systematic testing and performance evaluation across tasks and datasets within a sandbox of the hyperautomation system.
17. The hyperautomation system according to claim 11, wherein the actor agent of one or more artificial intelligence agents that execute the model request step of the current workflow performs self-benchmarking by matching the current workflow to two or more previous workflows and comparing the current results to generate two or more benchmark results and capabilities.
18. The hyperautomation system according to claim 11, wherein matching the current workflow to the two or more previous workflows based on the current result of the model request step includes utilizing one or more datasets in the artificial intelligence center detailing the agents and results of the two or more previous workflows.
19. The hyperautomation system according to claim 11, wherein generating the two or more benchmark results and capabilities by comparing the current result with the corresponding results of the two or more previous workflows includes utilizing one or more historical information data instances in the artificial intelligence center that detail the operational results of the two or more previous workflows.
20. The hyperautomation system according to claim 11, wherein the best benchmark result includes the result that includes the highest score among the two or more benchmark results and capabilities.
