Automated adjustment of user input in processes using AI agents

Abstract

This invention provides a system and method for automatically modifying a process using an AI (artificial intelligence) agent to adjust user input to that process. [Solution] The method includes receiving process data associated with a process, automatically modifying the process based on the process data using an AI agent to adjust user input to the process, and outputting the modified process.

Description

[Technical Field]

[0001] This invention relates in general to automation, and more specifically to the automatic adjustment of user input in a process using an AI (artificial intelligence) agent. [Background technology]

[0002] A process involves multiple activities performed by one or more computers to carry out various tasks. Often, a process includes activities that request user input even though such user input is not required, resulting in longer execution times and frequent interruptions to the user. Requests for user input are typically referred to as human-in-the-loop. Conversely, a process may not include activities that request user input even though such user input is necessary, resulting in reduced accuracy. Therefore, improved and / or alternative approaches may be beneficial. [Overview of the project]

[0003] Certain embodiments of the present invention may provide alternatives or solutions to technical problems and needs that have not yet been adequately identified, recognized, or resolved by existing automation technologies, and / or may provide useful alternatives thereto. For example, some embodiments of the present invention relate to dynamic human-in-the-loop for agent-based automation.

[0004] According to one or more embodiments, a system and method are provided for automatically modifying a process by an AI (artificial intelligence) agent to adjust user input to the process. Process data associated with the process is received. The process is automatically modified by the AI ​​agent based on the process data to adjust user input to the process. The modified process is output.

[0005] In one embodiment, process design data associated with the process is received. During process design, the process is automatically modified based on the process design data.

[0006] In one embodiment, process runtime data associated with the process is received. During a particular execution instance of the process, the process is automatically modified based on the process runtime data associated with that particular execution instance.

[0007] In one embodiment, process history execution data associated with the process is received. After one or more execution instances of the process, the process is automatically modified based on the process history execution data. In one embodiment, the process history execution data is preprocessed, and the process is automatically modified based on the preprocessed process history execution data.

[0008] In one embodiment, one or more activities that request user input from the user are added to the process.

[0009] In one embodiment, one or more activities that require user input from the user are removed from the process.

[0010] In one embodiment, the AI ​​agent receives the process data as context via one or more prompts.

[0011] In one embodiment, the AI ​​agent receives instructions via one or more prompts. The process is then further automatically modified by the AI ​​agent based on these instructions. [Brief explanation of the drawing]

[0012] In order that the advantages of specific embodiments of the present invention may be readily understood, a more specific description of the invention briefly described above will be given with reference to specific embodiments illustrated in the accompanying drawings. It should be understood that these drawings show only typical embodiments of the present invention and therefore should not be considered as limiting the scope thereof, and the present invention will be described and explained more specifically and in detail with the aid of the accompanying drawings.

[0013] [Figure 1] Figure 1 is an architecture diagram showing a hyperautomation system configured to perform agent-based automation and orchestration according to an embodiment of the present invention.

[0014] [Figure 2] Figure 2 is a diagram showing part of the combined functionality of an artificial intelligence (AI) agent and robotic process automation (RPA) robot according to an embodiment of the present invention.

[0015] [Figure 3] Figure 3 is a diagram showing an AI agent, an RPA robot, an agent-based orchestration process (AOP), and a pool of applications according to an embodiment of the present invention.

[0016] [Figure 4A] Figure 4A is a diagram showing an example of an AI agent service interface according to an embodiment of the present invention. [Figure 4B] Figure 4B is a diagram showing an example of an AI agent service interface according to an embodiment of the present invention.

[0017] [Figure 5] Figure 5 is a diagram showing an example of an AOP development interface according to an embodiment of the present invention.

[0018] [Figure 6]FIG. 6 is a diagram illustrating an example of an RPA development interface according to an embodiment of the present invention.

[0019] [Figure 7] FIG. 7 is a diagram illustrating an end-to-end AI agent, RPA robot, and AOP development and deployment system according to an embodiment of the present invention.

[0020] [Figure 8] FIG. 8 is an architectural diagram illustrating an agent-based automation and RPA system according to an embodiment of the present invention.

[0021] [Figure 9] FIG. 9 is an architectural diagram illustrating a deployed RPA system according to an embodiment of the present invention.

[0022] [Figure 10] FIG. 10 is an architectural diagram illustrating the relationship among a designer, activities, and drivers according to an embodiment of the present invention.

[0023] [Figure 11] FIG. 11 is an architectural diagram illustrating a computing system configured to update an AI agent according to an embodiment of the present invention.

[0024] [Figure 12A] FIG. 12A is a diagram illustrating an example of a neural network according to an embodiment of the present invention.

[0025] [Figure 12B] FIG. 12B is a diagram illustrating an example of a neuron according to an embodiment of the present invention.

[0026] [Figure 13] FIG. 13 is an architectural diagram illustrating a reference architecture of a generative AI model according to an embodiment of the present invention.

[0027] [Figure 14] Figure 14 is a flowchart showing the process of training an AI / ML model according to one embodiment of the present invention.

[0028] [Figure 15] Figure 15 shows a method, according to one embodiment of the present invention, for automatically modifying a process using an AI agent to adjust user input to that process.

[0029] [Figure 16] Figure 16 shows an exemplary directed graph 1600 of a process for processing and paying invoices according to one or more embodiments.

[0030] [Figure 17] Figure 17 shows an exemplary event log of process execution according to one or more embodiments.

[0031] [Figure 18] Figure 18 shows an exemplary directed graph of a modified process for processing and paying invoices, according to one or more embodiments.

[0032] Unless otherwise indicated, the same reference numerals consistently indicate the corresponding features throughout the attached drawings. [Modes for carrying out the invention]

[0033] Some embodiments relate to dynamic human-in-the-loop systems for agent-based automation. Specific embodiments will first be described with reference to Figures 1-14, and then further described with reference to Figures 15-18.

[0034] Figure 1 is an architectural diagram showing a hyperautomation system 100 configured to perform agent-based automation and orchestration according to one embodiment of the present invention. As used herein, “hyperautomation” refers to an automation system that brings together components of process automation, agent-based automation, integration tools, and technologies that amplify the ability to automate work. Examples of these components include, but are not limited to, artificial intelligence (AI) agents, agent-based orchestration processes (AOPs), and robotic process automation (RPA) robots.

[0035] Generally, as used herein, “AI agent” refers to AI-enhanced probabilistic automation that operates independently, dynamically, makes decisions, performs actions, and acts adaptively. This may be due to the use of Large-Scale Language Models (LLMs) by the AI ​​agent, which themselves are typically probabilistic. “AOP” refers to automation that enables users to describe the overall business process. AOP can be created using an interface that allows the creation of business flowcharts described in Business Process Models and Notation (BPMN). BPMN is an extensible markup language (XML) description of business processes. See, for example, Figure 5. “RPA robot” refers to rule-based deterministic automation that operates predictably and makes deterministic decisions.

[0036] For example, in some embodiments, RPA may be used at the core of a hyperautomation system, and in certain embodiments, automation capabilities may be extended by AI / machine learning (ML), process mining, analytics, agent-based automation, and / or other advanced tools. For example, as the hyperautomation system learns processes, trains AI / ML models, and uses analytics, more knowledge work can be automated, and computing systems within the organization, both those used by individuals and those that run autonomously, can become participants in the hyperautomation process. Some embodiments of the hyperautomation system enable users and organizations to discover, understand, and extend automation efficiently and effectively.

[0037] In such embodiments, the AI ​​agent "coexists" with the RPA robots that perform RPA and AOP. As noted herein, the AI ​​agent is AI-skilled automation that can operate independently, make decisions dynamically, perform actions, and adapt its performance. The AI ​​agent can dynamically leverage the tools available through these RPA robots to perform tasks such as document processing (e.g., see U.S. Patent Application Publication 2021 / 0097274), user interface (UI) automation (e.g., see U.S. Patents 10,654,166, 10,990,876, 11,080,548, 11,507,259, 11,733,668, and 11,748,069), semantic copy and paste between source and target (e.g., see U.S. Patent 12,124,806 and U.S. Patent Application Publications 2023 / 0107316, 2023 / 0415338, and 2024 / 0220581). The AI ​​agent can dynamically select these tools and execute them in a pipeline.

[0038] Generally speaking, agent-based automation is probabilistic automation performed by one or more AI agents. Agent-based automation expands an organization's automation potential by focusing not only on individual tasks but also on the whole end-to-end process. A team of RPA robots directed by AI agents can enable one employee to accomplish the work of many. Agent-based automation via AI agents gives managers room for guidance, doctors more time for patient care, developers the ability to fine-tune their work, technicians the freedom to innovate, and customers a seamless and personalized experience.

[0039] In some embodiments, agent-based automation can achieve a variety of technical effects, benefits, and advantages. Agent-based automation improves memory usage by reducing the storage required to store data and improves processor efficiency by reducing the number of calls and actions. Agent-based automation also potentially offers the ability to process gigabytes, terabytes, petabytes, or more of data that would be impossible for human processes to perform, whether mental or manual. Agent-based automation can also enable the use of fewer triggers and models through dynamic decision-making. In an exemplary scenario, while RPA alone might require 100 actions, using agent-based automation can significantly reduce this (e.g., to 15 actions). Context grounding can also be used to tie AI agents to the desired context for agent-based automation. Thus, context grounding "constrains" the LLM to the appropriate context.

[0040] An AI agent may have agent memory that evolves to store user interactions, feedback, corrections, and solutions (e.g., dynamic user input from human-in-the-loop operations). As used herein, “human-in-the-loop” or human-in-the-loop operation may include the AI ​​agent and RPA robot working in collaboration with a user to receive dynamic direct user input. As agent memory grows, the AI ​​agent can become increasingly autonomous, reducing the need for dynamic direct human input and improving efficiency. The AI ​​agent may also learn to become more efficient based on agent memory if more efficient solutions are contained in or derived from that agent memory. For example, the AI ​​agent may periodically process agent memory to analyze patterns in order to achieve greater autonomy.

[0041] As used herein, “agent memory” refers to a dynamic caching (i.e., memory) system for managing escalations and tool calls. Exemplary operation, if an AI agent encounters a problem during execution, it may prompt or otherwise request user interaction or feedback to overcome the problem, remember / cached that interaction or feedback, and learn from it to reduce the need for repeated human input. According to one or more technical effects, benefits, and advantages, agent memory improves efficiency by remembering solutions to common problems and minimizing potentially costly tool calls. The collaborative operation between the AI ​​agent and agent memory can “bend the curve” so that as the AI ​​agent continuously learns through agent memory, less and less human interaction is required.

[0042] Generally, agent-based orchestration is performed by a conductor application to implement one or more AOPs, which utilize AI agents and RPA robots. In some embodiments, agent-based orchestration orchestrates AI agents (e.g., UiPath Agents®), third-party agents, RPA robots (e.g., UiPath Robots®), AOPs, and humans (e.g., where human approval is required) to execute agent-based workflows. Thus, agent-based orchestration enables the automation, modeling, and monitoring of complex business processes from start to finish. Agent-based orchestration also offers a unique ability to orchestrate RPA robots, AI agents, third-party agents, and humans across an entire end-to-end agent-based workflow. Agent-based orchestration is beneficial for scaling agent-based automation successfully.

[0043] As described above, AI agents for agent-based automation are AI model-based, enabling them to operate independently of humans and perform these agent-based automations. AI agents are also goal-oriented, making probabilistic decisions using context. Furthermore, AI agents are well-suited for ad-hoc tasks requiring high adaptability. AI agents learn how to perform tasks and improve over time. AI agents can use and select various tools (often via RPA robots, which the AI ​​agent uses as tools) to accomplish tasks, gather context, and perform actions. In some embodiments, AI agents may generate automations by building workflows for RPA robots and / or other AI agents to execute, leveraging UiPath Autopilot® for developers or other applications that help accelerate automation creation and testing. For example, an AI agent may use a designer application via an API to generate another AI agent or RPA workflow, and then perform human-in-the-loop actions to address issues related to the generated workflow. If correct, that workflow can then be deployed. AI agents can also possess varying degrees of autonomy, controlled by agent-based orchestration.

[0044] The AI ​​agent executes an "agent-type loop," using the provided tools and context to generate a dynamic plan to achieve the objective according to the instructions. Once the dynamic plan is generated, the AI ​​agent utilizes an efficient execution path for that plan. If the dynamic plan has two or more steps that can be executed in parallel, the AI ​​agent executes these steps in parallel based on available resources. After each step is completed, the AI ​​agent retrieves the output from that step and regenerates the next step or multiple steps. Thus, the agent-type loop continues until the objective is achieved. Executing steps of the dynamic plan in parallel, as well as using ecosystem tools and context grounding, are advanced features of agent-based orchestration.

[0045] As noted herein, RPA robots are rule-based, operate predictably, and make deterministic decisions. RPA robots are highly reliable and efficient and are suitable for routine tasks. RPA robots, along with AI agents, can use human-in-the-loop behavior for exception handling. According to some embodiments, AI agents are more flexible, more abstract, and more self-deterministic than RPA robots and AOP. RPA robots are generally more stable, more specific, and more controllable than AI agents and AOP. AOP processes typically fall somewhere between the flexibility / stability, abstraction / specificity, and self-determination / controllability characteristics of AI agents and RPA robots, respectively.

[0046] As further explained with respect to Figure 3, AI agents and RPA robots can potentially find and use each other as tools to accomplish tasks. AI agents and RPA robots can also access and use various applications (e.g., via Application Programming Interfaces (APIs)). Tools may be manually configured by developers for automation, and / or AI agents and RPA robots may discover and use tools at runtime.

[0047] In some embodiments, AI agents, AOPs, and RPA robots can work collaboratively with users (e.g., human-in-the-loop), enabling them to make faster, more consistent, and more informed decisions. Furthermore, the use of AI agents, AOPs, and RPA robots allows humans to accomplish more. This is because AI agents, AOPs, and RPA robots can undertake additional repetitive, monotonous, and ad-hoc tasks at a scale unmanageable by human users. When an AI agent, AOP, or RPA robot encounters an exception, a human can make the necessary decision. Thus, humans can be elevated to supervisory roles, decision-making roles, and organizational leadership roles, allowing them to focus on these responsibilities.

[0048] AI models provide AI agents with the ability to infer, plan, generate, and make autonomous decisions. AI models can also be used by RPA robots for task-specific activities such as document processing or data analysis. AI models can be enhanced with business-specific content and context (e.g., from enterprise-specific context repositories), which improves the accuracy and results of the AI ​​models. AI models can be applied individually or simultaneously, depending on the complexity of the task. AI model selection can be done from the RPA vendor's model library, third-party models, and BYOM (bring your own model) options (see, for example, U.S. Patents 11,738,453 and 11,748,479).

[0049] The hyper-automation system 100 includes user computing systems such as a desktop computer 102, a tablet 104, and a smartphone 106. However, any desired user computing system, including but not limited to smartwatches, laptop computers, servers, and Internet of Things (IoT) devices, can be used without departing from the scope of the present invention. Also, although three user computing systems are shown in Figure 1, any appropriate number of user computing systems can be used without departing from the scope of the present invention. For example, in some embodiments, tens, hundreds, thousands, or millions of user computing systems may be used. The user computing systems may be used actively by the user or may operate automatically with little or no user input.

[0050] As disclosed herein, in some embodiments there are three types of automation: (1) agent-based automation performed by each AI agent, (2) RPA performed by each RPA robot, and (3) composite automation achieved by a combination of AI agents and RPA robots to accomplish a more complex overall task. Automations 110, 112, and 114 may include, but are not limited to, those performed by RPA robots and / or AI agents, whether performed individually or to achieve a larger composite automation. Other processes, such as listeners, may also be implemented. These processes may be standalone applications, subprocesses of other applications, parts of operating systems, any other suitable software and / or hardware, or any combination thereof, and will not depart from the scope of the present invention. In fact, in some embodiments, the logic of a process is partially or completely implemented by physical hardware.

[0051] On each user's computing system 102, 104, and 106, the respective automations 110, 112, and 114 are executed, some of which are performed by RPA robots, AI agents, etc. In some embodiments, automations 110, 112, and 114 may be stored remotely (e.g., on a server 130 or in a database 140 and accessed via a network 120) and loaded by RPA robots and / or AI agents to be executed. The database 140 may store structured data and / or unstructured data, but the former is usually required for RPA. RPA automations may exist as scripts (e.g., Extensible Markup Language (XML), Extensible Application Markup Language (XAML), etc.) or may be compiled into machine-readable code (e.g., as a digital link library). In the case of AI agents, agent-type automations may be generated based, for example, on a plaintext description of a desired target.

[0052] The listener monitors and records data relating to user interactions with each computing system and / or the operation of the unattended computing system, and transmits this data to the core hyperautomation system 120 via a network (e.g., a local area network (LAN), mobile communication network, satellite communication network, the internet, or any combination thereof). This data may include, but is not limited to, buttons clicked, mouse movement locations, text entered into fields, whether one window is minimized and another is opened, and applications associated with windows. In certain embodiments, data from the listener may be transmitted periodically as part of a heartbeat message. In some embodiments, data may be transmitted to the core hyperautomation system 120 after a predetermined amount of data has been collected, after a predetermined time has elapsed, or both. One or more servers, such as server 130, receive the data from the listener and store it in a database, such as database 140.

[0053] If automations 110, 112, and 114 are RPAs, they can execute logic developed in the workflow at design time. A workflow may include a series of steps, defined herein as “activities,” which are executed sequentially or in other logical flows. Each activity may include actions such as clicking a button, reading a file, or writing to a log panel. In some embodiments, workflows may be nested or embedded.

[0054] In some embodiments, long-running RPA workflows are master projects that support service orchestration, human-in-the-loop, and long-running transactions in unattended environments. See, for example, U.S. Patent No. 10,860,905, which is incorporated herein by reference in its entirety. Human-in-the-loop is involved when a particular process requires human input (e.g., dynamic direct user input) to handle exceptions, approvals, or validations before proceeding to the next step in the activity. In this situation, process execution is suspended and the RPA robot is released until the human-in-the-loop portion of the task is completed.

[0055] Long-running workflows can support workflow fragmentation via persistent activities and, in combination with invocation processes and non-user interaction activities, can orchestrate human-in-the-loop tasks with RPA robot tasks. In some embodiments, multiple or numerous computing systems may participate in the logic execution of a long-running workflow. Long-running workflows may run in sessions to facilitate fast execution. In some embodiments, a long-running workflow may orchestrate background processes that execute API calls and may include activities that run in the long-running workflow session. These activities may be invoked by invocation process activities in some embodiments. Processes with user interaction activities that run in a user session may be invoked by starting a job from a conductor activity (conductors are described later). In some embodiments, the user may interact with the conductor through tasks that require form completion. Activities may include causing the RPA robot to wait for the form task to complete and then resume the long-running workflow.

[0056] One or more of the automations 110, 112, and 114 are in communication with the core hyperautomation system 120. In some embodiments, the core hyperautomation system 120 may run conductor applications on one or more servers, such as server 130. Although one server 130 is shown for illustrative purposes, multiple or many servers that are close to each other or in a distributed architecture can be used without departing from the scope of the invention. For example, one or more servers may be provided for conductor functions, AI / ML model provisioning, authentication, governance, and / or any other appropriate functions without departing from the scope of the invention. In some embodiments, the core hyperautomation system 120 may incorporate or be part of a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, etc. In certain embodiments, the core hyperautomation system 120 may host multiple software-based servers on one or more computing systems, such as server 130. In some embodiments, one or more servers of the core hyperautomation system 120, such as server 130, may be implemented by one or more virtual machines (VMs).

[0057] In some embodiments, one or more of the automations 110, 112, and 114 are deployed on or accessible from the core hyperautomation system 120 and can invoke one or more AI / ML models 132 trained to perform various tasks. For example, the AI / ML models 132 may include models trained to perform tasks such as searching for various application versions, performing computer vision (CV), performing optical character recognition (OCR), generating user interface (UI) descriptors, suggesting the next activity or set of activities in an RPA workflow, performing semantic matching, performing natural language processing (NLP), generating or modifying code and / or RPA workflows. The AI / ML models may be trained using labeled data including elements from data sources such as web pages, forms, scanned documents, application interfaces, screens, previously created RPA workflows, screenshots of various versions of various application screens and their corresponding UI elements, UI object libraries, etc. The AI / ML models 132 may be trained to avoid overfitting to a given training dataset while achieving a desired confidence threshold. In general, UI elements, UI descriptors, applications, and application screens can be considered UI objects.

[0058] AI / ML model 132 can be trained for any suitable purpose without departing from the scope of the present invention. As will be described in more detail later, in some embodiments, two or more AI / ML models 132 can be linked together (e.g., in series, parallel, or a combination thereof) to provide collaborative output. AI / ML model 132 can perform or assist in CV, OCR, document processing and / or document understanding, semantic learning and / or semantic analysis, analytical prediction, process discovery, task mining, testing, automated RPA workflow generation, sequence extraction, cluster detection, speech-to-text conversion, NLP, semantic matching, and any combination thereof. However, any desired number and / or types of AI / ML models can be used without departing from the scope of the present invention. By using multiple AI / ML models, for example, the system can form a whole picture of what is happening on a given computing system. For example, one AI / ML model may perform OCR, another model may detect buttons, and yet another model may compare sequences. Patterns may be determined individually by one AI / ML model or collectively by multiple AI / ML models. In certain embodiments, one or more AI / ML models are deployed locally on at least one computing system 102, 104, 106.

[0059] In some embodiments, multiple AI / ML models 132 may be used. Each AI / ML model 132 is an algorithm (or model) executed on data, and the AI / ML model itself may be a deep learning neural network (DLNN) consisting of artificial "neurons" trained on training data, for example. In some embodiments, the AI / ML model 132 has multiple layers that perform various functions such as statistical modeling (e.g., Hidden Markov Models (HMMs)) and may utilize deep learning techniques (e.g., Long Short-Term Memory (LSTM) deep learning, coding of preceding hidden states, etc.) to perform desired functions.

[0060] In some embodiments, the hyper-automation system 100 may provide four main functional groups: (1) discovery, (2) automation construction, (3) management, and (4) engagement. For example, automation performed on a user computing system, server, etc., may in some embodiments be performed by an RPA robot, AOP, or AI agent and may provide any of the functions described herein. For example, an RPA robot may include a manned robot, an unmanned robot, and / or a test robot. A manned robot assists with tasks in collaboration with the user (e.g., via UiPath Assistant®). An unmanned robot operates independently of the user and may run in the background without the user's knowledge. A test robot executes test cases against an application or RPA workflow. In some embodiments, a test robot may run in parallel on multiple computing systems.

[0061] The discovery function can discover various automation opportunities in business processes and provide automated recommendations for them. Such a function may be implemented by one or more servers, such as server 130. In some embodiments, the discovery function may include an automation hub, process mining, task mining, and / or task capture. The automation hub (e.g., UiPath Automation Hub®) may provide a mechanism for automation deployment management with visibility and control. For example, automation ideas may be crowdsourced from employees via a submission form. The feasibility and return on investment (ROI) calculations for automating these ideas may be provided, documentation for future automations may be collected, and collaboration may be provided to accelerate the process from automation discovery to build.

[0062] Process mining (e.g., via UiPath Automation Cloud® and / or UiPath AI Center®) refers to the process of collecting and analyzing data from applications (e.g., ERP applications, CRM applications, email applications, call center applications, etc.) to identify what end-to-end processes exist within an organization, how to effectively automate them, and what the impact of automation will be. This data may be collected, for example, by listeners from user computing systems 102, 104, 106 and processed by servers such as server 130. In some embodiments, one or more AI / ML models 132 may be used for this purpose. This information may be exported to an automation hub to expedite implementation and avoid manual information transfer. The goal of process mining may be to increase business value by automating processes within an organization. Examples of process mining goals include, but are not limited to, increased profits, improved customer satisfaction, regulatory and / or contractual compliance, and improved employee efficiency.

[0063] Task mining (e.g., via UiPath Automation Cloud® and / or UiPath AI Center®) identifies and aggregates workflows (e.g., employee workflows), then applies AI to reveal patterns and variations in daily tasks, and evaluates the ease of automation and potential savings (e.g., time and / or cost savings) for such tasks. One or more AI / ML models 132 may be used to reveal recurring task patterns in the data. Recurring tasks suitable for automation can then be identified. In some embodiments, this information is initially provided by a listener and can be analyzed on a server of the core hyperautomation system 120, e.g., server 130. Findings from task mining (e.g., XAML process data) can be exported to a process document or a designer application such as UiPath Studio® to more quickly create and deploy automations. Task mining in some embodiments may include taking screenshots with user actions (e.g., mouse click locations, keyboard inputs, application windows and graphical elements the user interacted with, timestamps of interactions, etc.), collecting statistical data (e.g., execution time, number of actions, number of text inputs, etc.), editing and annotating screenshots, and specifying the types of actions to be recorded.

[0064] Task capture (e.g., via UiPath Automation Cloud® and / or UiPath AI Center®) automatically documents human processes as users work on them, or provides a framework for unmanned processes. Such documentation may include desired tasks to be automated in the form of process definition documents (PDDs), skeletal workflows, action captures for each part of the process, recordings of user actions and automatic generation of comprehensive workflow diagrams including details of each step, Microsoft Word® documents, XAML files, etc. In some embodiments, build-ready workflows can be directly exported to designer applications such as UiPath Studio®. Task capture can simplify the requirements gathering process for both subject matter experts describing processes and CoE members providing production-quality automation.

[0065] The construction of automation can be achieved through designer applications (e.g., UiPath Studio®, UiPath StudioX®, or UiPath Studio Web®). For example, a developer of RPA development equipment 150 can use the designer application 154 of computing system 152 to build and test agent-based automation, RPA, AOP, and / or hybrid automation for various applications and environments such as web, mobile, SAP®, and virtualized desktops. The developer can also build AOP. For example, the developer can create automations that are executed by RPA robots, AI agents, AOP, or combinations thereof. API integration can be provided for various applications, technologies, and platforms. Predefined activities, drag-and-drop modeling, and workflow recorders can facilitate automation with minimal coding. Document understanding capabilities can be provided through drag-and-drop AI skills for data extraction and interpretation, which invoke one or more AI / ML models 132. Such automations can handle virtually any document type and format, including tables, checkboxes, signatures, and handwritten text. Once the data has been validated or exceptions have been handled, this information can be used to retrain each AI / ML model, improving its accuracy over time.

[0066] The designer application 152 may be designed to call one or more trained AI / ML models 132 on the server 130 and / or generated AI models 172 in a cloud environment via the network 120 (e.g., a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, or any combination thereof) to support the automated development process. In some embodiments, one or more AI / ML models may be packaged in the designer application 152 or stored locally on the computing system 150.

[0067] In some embodiments, the designer application 152 and one or more AI / ML models 132 may be configured to use an object repository stored in a database 140. See, for example, U.S. Patent No. 11,748,069, which is incorporated herein by reference in its entirety. Generally, an object repository is a storage mechanism used by automation for images, text, semantic data, taxonomic associations, ontological associations, UI objects, etc. For example, an object repository may include a library of UI objects that can be used to develop RPA workflows via the designer application 152. In some embodiments, the object repository may be used to add UI descriptors to activities in the workflow of the designer application 152 for UI automation. In some embodiments, one or more AI / ML models 132 may generate new UI descriptors and add them to the object repository in the database 140.

[0068] Once automation is complete in designer application 152, the results are exposed on server 130 and can be pushed to computing systems 102, 104, 106, etc. For example, as new UI descriptors are created and / or existing UI descriptors are modified, a global repository of a shareable and collaborative UI object library can be built for all automations. With respect to the object repository, classification systems and ontology can be used. A classification system is a hierarchical structure of subcategories. Ontology is a formal representation of a knowledge domain, including concepts, characteristics, and the relationships between them. In ontology, the relationships between categories are not necessarily hierarchical, and ontological relationships can span multiple screens of an application.

[0069] Integrated services can, for example, enable developers to seamlessly combine UI automation and API automation. Automation, such as any type of automation described herein, may require an API or be built to span both API applications and non-API applications or systems. A repository (e.g., UiPath Object Repository®) or marketplace (e.g., UiPath Marketplace®) for pre-built automation templates and solutions may be provided, enabling developers to automate a wide variety of processes more quickly. Thus, when building automation, the hyperautomation system 100 may provide a user interface, development environment, API integration, pre-built and / or custom-built AI / ML models, development templates, an integrated development environment (IDE), and advanced AI capabilities. In some embodiments, the hyperautomation system 100 may enable the development, deployment, management, configuration, monitoring, debugging, and maintenance of RPA robots and AI agents, which may provide automation for the hyperautomation system 100.

[0070] In some embodiments, components of the hyperautomation system 100, such as a designer application and / or an external rule engine, provide support for managing and enforcing governance policies to control the various functions provided by the hyperautomation system 100. Governance is the organization's ability to establish policies to prevent users from developing automations (e.g., RPA robots and / or AI agents) that could harm the organization, such as violating the EU General Data Protection Regulation (GDPR), the US Health Insurance Portability and Accountability Act (HIPAA), or the terms of use of third-party applications. On the other hand, developers may create automations that violate privacy laws, terms of use, etc., when implementing automation. Therefore, in some embodiments, access control and governance restrictions are implemented at the robot and / or robot design application level. This can provide an additional level of security and compliance to the automation process development pipeline by preventing developers from relying on unauthorized software libraries, which may introduce security risks or function in a way that violates policies, regulations, privacy laws and / or privacy policies. For example, see U.S. Patent No. 11,733,668, which is incorporated herein by reference in its entirety.

[0071] The management functions can provide management, deployment, and optimization of automation across the entire organization. In some embodiments, the management functions may include orchestration, test management, AI capabilities, and / or insights. The management functions of the hyperautomation system 100 may also function as an integration point with third-party solutions and applications for automation applications and / or RPA robots. The management capabilities of the hyperautomation system 100 may include, but are not limited to, facilitating the provisioning, deployment, configuration, queuing, monitoring, logging, and interoperability of RPA robots and / or AI agents.

[0072] Conductor applications such as UiPath Orchestrator® (which, in some embodiments, may be offered as part of UiPath Automation Cloud®, or as a cloud-native single container suite on-premises, in a VM, in a private or public cloud, in a Linux® VM, or via UiPath Automation Suite®) provide orchestration capabilities for deploying RPA robots and / or AI agents, enabling them to be deployed, monitored, optimized, scaled, and secured. A test suite (e.g., UiPath Test Suite®) may provide test management for monitoring the quality of deployed automations. The test suite can facilitate test planning and execution, requirements fulfillment, and defect traceability. The test suite may include comprehensive test reports.

[0073] Analytics software (e.g., UiPath Insights®) can track, measure, and manage the performance of deployed automations. Analytics software can align automation behavior with specific key performance indicators (KPIs) and strategic outcomes of the organization. Analytics software can present results in a dashboard format for easier understanding by human users.

[0074] A data service (e.g., UiPath Data Service®) can store data in a single, scalable, and secure location, for example, in a database 140, with a drag-and-drop storage interface. Some embodiments can provide low-code or no-code data modeling and storage for automation, while ensuring seamless access, enterprise-grade security, and scalability. AI functionality can be provided by an AI center (e.g., UiPath AI Center®), which facilitates the integration of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options can make such functionality available even to non-data scientists. Deployed automations (e.g., RPA robots) can call AI / ML models, such as AI / ML model 132, from the AI ​​center. The performance of AI / ML models can be monitored and trained and improved using human-validated data, such as that provided by a data review center 160. Human reviewers can provide labeled data to the core hyper-automation system 120 via a review application 152 on a computing system 154. For example, a human reviewer may verify the accuracy of predictions made by AI / ML model 132 and / or generative AI model 172, and provide corrections if necessary. The human reviewer may also provide dynamic direct user input to the AI ​​agent (e.g., within the scope of human-in-the-loop operation), and the responses and corrections provided by the human reviewer may be used to train the LLM used by the AI ​​agent to be more accurate. In other words, this dynamic input may be stored as training data for retraining AI / ML model 132 and / or generative AI model 172, for example, in a database such as database 140. The AI ​​center may then schedule and execute training jobs to train a new version of the AI / ML model using this training data. Both positive and negative examples may be stored and used to retrain AI / ML model 132 and / or generative AI model 172.

[0075] Engagement features bring humans and automation together as a team, enabling seamless collaboration in desired processes. Low-code applications can, in some embodiments, be built to connect browser tabs and legacy software, and even those lacking APIs (e.g., via UiPath Apps®). For example, applications can be quickly created through a web browser using a rich drag-and-drop control library. Applications can connect to a single automation or multiple automations.

[0076] An action center (e.g., UiPath Action Center®) provides a simple and efficient mechanism for handing over processes from automation to humans and vice versa. Humans can provide approvals or escalations, handle exceptions, etc. Automation can then execute the automated functions of a given workflow.

[0077] A local assistant may be provided as a launchpad for users to initiate automations (e.g., UiPath Autopilot®). Such an assistant may also provide semantic cut-and-paste functionality (e.g., UiPath Clipboard AI®). See, for example, U.S. Patent No. 12,124,806 and U.S. Patent Application Publications 2023 / 0107316, 2023 / 0415338, and 2024 / 0220581. This functionality may be provided, for example, in a tray provided by the operating system, allowing the user to interact with RPA robots and RPA robot-driven applications on their computing system. The interface may list automations approved for a given user and allow the user to run them. These may include readily available automations from an automation marketplace, automations from an internal automation store within an automation hub, etc. Once an automation is executed, it may run as a local instance in parallel with other processes on the computing system, so that the user can continue to use the computing system while the automation is performing its actions. In certain embodiments, the assistant is integrated with a task capture function, allowing the user to document processes that are soon to be automated from its assistant launchpad.

[0078] In some embodiments, the hyper-automation system 100 may provide end-to-end measurement and governance of automation programs of any scale. As described above, analytics may be used to understand the performance of automation (e.g., via UiPath Insights®). Data modeling and analytics using any combination of available business metrics and operational insights may be used for various automation processes. Custom-designed and pre-built dashboards enable data visualization across desired metrics, discovery of new analytical insights, tracking of performance metrics, discovery of ROI for automation, implementation of telemetry monitoring of user computing systems, detection of errors and anomalies, and debugging of automation. An automation management console (e.g., UiPath Automation Ops®) may be provided to manage automation throughout the entire automation lifecycle. Organizations can govern how automations are built, what users can do with them, and which automations users can access.

[0079] In some embodiments, the hyperautomation system 100 provides an iterative platform. Processes are discovered, automation is built, tested, and deployed, its performance is measured, the use of automation is easily made available to users, feedback is obtained, AI / ML models are trained and retrained, and the process can be repeated. This results in a more robust and effective set of automations.

[0080] In some embodiments, a generative AI model is used as described above. For example, an AI agent utilizes a generative AI model. A generative AI model can generate various types of content, such as text, images, audio, and synthetic data. Possible types of generative AI models include, but are not limited to, LLMs, generative adversarial networks (GANs), diffusion models, flow-based models, variational autoencoders (VAEs), and transformers. For example, in the case of LLMs, NLP models such as word2vec, BERT, GPT-3, and ChatGPT may be used in some embodiments to facilitate semantic understanding and provide more accurate and human-like responses. These models may be part of an AI / ML model 132 hosted on server 130. For example, a generative AI model can be trained on a large text information corpus to perform semantic understanding, understanding the properties of textual elements present on a screen, automatic code generation, etc. An AI agent may use such a generative AI model. In certain embodiments, generative AI models 172 provided by existing cloud ML service providers such as OpenAI®, Google®, Amazon®, Microsoft®, IBM®, Nvidia®, and Meta® may be used and trained to provide such functionality. In generative AI embodiments where the generative AI model is remotely hosted, the server 130 may be configured to integrate with a third-party API, thereby allowing the server 130 to send requests containing the necessary input information to the generative AI model 172 and receive responses in return (e.g., semantic matches of fields between application versions, classification of application types on a screen, responses to natural language queries from the user, etc.). Such embodiments may provide a more advanced and sophisticated user experience and may provide access to state-of-the-art NLP and other ML capabilities offered by these companies.

[0081] One aspect of generative AI models in some embodiments is the use of transfer learning. In transfer learning, a pre-trained generative AI model, such as a Language Language Model (LLM), is fine-tuned for a specific task or domain. This allows the LLM to leverage knowledge already learned during initial training and adapt it to its specific application. For example, in the case of an LLM, the pre-training phase typically involves training the LLM on a large text corpus consisting of billions of words. During this phase, the LLM learns the relationships between words and phrases, enabling it to generate consistent, human-like responses to text-based input. The output of this pre-training phase is an LLM with a high level of understanding of fundamental patterns in natural language.

[0082] In the fine-tuning phase, a pre-trained LLM is adapted to a specific task or domain by training the LLM on a small dataset specific to that task. For example, in some embodiments, the LLM may be trained to analyze specific types or combinations of data sources to improve its accuracy regarding their content. This data may include, but is not limited to, prompt tuning or instruction tuning, which the model is specifically trained to better understand and follow certain types of instructions or prompts. This improves the model's ability to perform a particular task when given appropriate instructions. Such information may be provided as part of the training data, and the LLM may learn to focus on these areas and more accurately identify the data elements contained therein. Through fine-tuning, the LLM can learn the nuances of its task or domain, such as specific vocabulary and syntax used in that domain, without requiring as much data as would be needed to train the LLM from scratch. By leveraging the knowledge learned in the pre-training phase, a fine-tuned LLM can achieve state-of-the-art performance for a particular task with relatively little training data.

[0083] LLM can utilize vector databases. Vector databases index, store, and make accessible structured or unstructured data (e.g., text, images, time-series data, etc.) along with their vector embeddings. Data such as text can be tokenized, with single characters, words, or sequences of words being parsed from text into tokens. These tokens are then "embedded" in vector embeddings, which are numerical representations of the data. Vector databases enable LLM to quickly and extensively discover and retrieve similar objects in a production environment, which is impossible with manual processes.

[0084] AI and ML enable the numerical representation of unstructured data within vector embeddings without losing their semantic meaning. A vector embedding is a long sequence of numbers, each describing a feature of the data object represented by that embedding. Similar objects are grouped closely together in the vector space. In other words, the more similar objects are, the closer their vector embeddings are to each other. Similar objects can be found using vector lookup, similarity lookup, or semantic lookup. The distance between vector embeddings can be calculated using a variety of techniques, including, but not limited to, squared Euclidean distance or L2-squared distance, Manhattan distance or L1 distance, cosine similarity, dot product, Hamming distance, etc. It may be beneficial to choose the same metrics used to train the AI / ML model.

[0085] Vector indexing can be used to organize vector embeddings so that data can be retrieved efficiently. Calculating the distance between a vector embedding and all other vector embeddings in a vector database using the k-nearest neighbors (kNN) algorithm can be computationally expensive, as the required computation increases linearly (O(n)) with respect to the number of dimensions and data points when the number of data points is large. Finding similar objects using an approximate nearest neighbors (ANN) approach is more efficient. Since the distances between vector embeddings are pre-calculated and similar vectors are organized and stored close to each other (e.g., in clusters or graphs), similar objects can be found more quickly. This process is called "vector indexing." ANN algorithms that may be used in some embodiments include, but are not limited to, cluster-based indexing, proximity-graph-based indexing, tree-based indexing, hash-based indexing, and compression-based indexing.

[0086] Figure 2 shows a part of the combined function 200 of an AI agent 210 and an RPA robot 220 according to one embodiment of the present invention. The AI ​​agent 210 is configured to process natural language instructions and achieve the expected goals therefrom, execute through dynamic decision-making or dynamic flow control with self-correction capabilities, store information in long-term memory to evaluate its own execution performance, and learn from human-in-the-loop and its own performance during execution. The RPA robot 220 can be used by the AI ​​agent 210 to respond to triggers (e.g., from conductor applications such as UiPath Orchestrator®), respond based on context (i.e., the RPA robot 220 can retrieve information from the context and perform deterministic steps such as updating a document based on the retrieved information; or the agent 210 can use the retrieved context to update a dynamic plan and perform the next steps to complete the goal according to the instructions), leverage AI models (e.g., CV models, document processing models, speech-to-text models, OCR models, etc.), leverage RPA tools (e.g., tools available within the RPA ecosystem, such as completed automations, workflows within automations, integrated service connector calls for third-party and first-party services, RPA designer application activities, LLM calls, etc.), and perform actions that the RPA robot can take based on input from the AI ​​agent (i.e., use the RPA robot as a tool). AI agent 210 can also update its own memory, update its plan to achieve goals according to instructions, perform self-assessment and learning from its actions, self-correct when it encounters obstacles, and take action to escalate to a human if assistance is needed.

[0087] As described above, in some embodiments, agent-based automation can achieve a variety of technical effects, benefits, and advantages. Agent-based automation improves memory usage by reducing the amount of storage required for data and improves processor efficiency by reducing the number of calls and actions. Agent-based automation also potentially offers the ability to process gigabytes, terabytes, petabytes, or more of data that would be impossible for human processes, whether mental or manual. It can also enable the use of fewer triggers and models through dynamic decision-making. In an exemplary scenario, while RPA alone might require 100 actions, using agent-based automation can significantly reduce this (e.g., to 15 actions). Context grounding can also be used to tie the AI ​​agent to the desired context for agent-based automation. This "constrains" the LLM to the appropriate context.

[0088] As used herein, “context grounding” refers to a methodology that improves models such as LLMs by integrating enterprise-specific information with pre-trained knowledge, enabling accurate responses to specialized or recent queries. In some embodiments, context grounding uses external data to augment LLM responses, obtaining responses that the LLM would not inherently know and answering queries within the provided context. For example, because proprietary industry jargon and complex document structures can be challenges in ensuring effective retrieval and semantic matching, context grounding addresses these challenges by precisely chunking documents so that relevant information derived from proprietary industry jargon and complex document structures can be passed to the LLM without noise. As an additional example, context grounding provides enhanced extraction and retrieval techniques adapted to diverse industries and applications (e.g., adapted to proprietary industry jargon and complex document structures), thereby improving LLM responses.

[0089] Figure 3 shows a pool 300 of AOP, AI agent, RPA robot, and application according to one embodiment of the present invention. The AOP pool 310 includes an AOP that implements a business process. As described above, the AOP may be implemented as a BPMN executed by an AOP execution engine such as Temporal®. The AOP may use an AI agent and / or an RPA robot to execute a portion of the business process.

[0090] The AI ​​agent pool 320 includes AI agents trained to perform various tasks such as investigating claims, requesting resolutions from human employees, and summarizing policies and technical specifications. The RPA robot pool 330 includes RPA robots that perform various automations such as UI automation, semantic matching automation, and form entry automation. The application pool 340 includes applications that the AI ​​agents and / or RPA robots can interact with. For example, applications may include CRM applications, billing applications, payroll applications, banking applications, web applications, legacy system applications, word processing applications, spreadsheet applications, email applications, etc. The AI ​​agents, RPA robots, and applications may reside on a single computing system, or on multiple or numerous computing systems. The AOP is typically located in the cloud or on another server side, and in some embodiments may be on the same computing system as the conductor application 350.

[0091] AOP can trigger or invoke AI agents and RPA robots via the conductor application 350. AI agents and RPA robots can also trigger or invoke each other via the conductor application. For example, to invoke an RPA robot, an AI agent may make a "Start Job" call in the conductor application 350. Note that the RPA robot is deployed as an automation controlled by the conductor application 350. AI agents and RPA robots can also trigger or invoke specific applications. For example, through information obtained from human-in-the-loop operations, an AI agent can dynamically learn which RPA robots, other AI agents, and / or applications to trigger or invoke to accomplish a task. For example, an AI agent may learn to trigger an RPA robot via the conductor application 350 to fill out and submit a web form. The AI ​​agent may also learn to open Microsoft Excel® and enter form information into the appropriate tab, open and update a payroll application, etc. It may also be learned to call or trigger an email resolution AI agent via the conductor application 350 to contact a human customer service representative at the bank if a problem arises. The technical effects, benefits, and advantages may be similar to those described above with respect to Figures 1 and 2 in some embodiments.

[0092] AI agents and RPA robots may belong to a tenant in order to find each other. A designer application may call a conductor to obtain a list of available RPAs. In some embodiments, there are three ways to obtain the capabilities of automation: (1) by providing a description of what the automation does when a user creates a workflow in a designer application; (2) by generating a summary of what a given workflow does using an AI agent and ML technology; and (3) by a developer describing what the automation does in a designer application. A conductor application may also have a list of which applications are available for a given AI agent and RPA robot. In other words, the descriptions of available AI agents, RPA robots, and / or applications are derived or assigned by the AI ​​agent, ML technology, or user.

[0093] Figures 4A and 4B show an example of an agent service interface 400 according to one embodiment of the present invention. Referring to Figure 4A, the agent answers questions regarding a policy document provided within context grounding. The agent command pane 410 contains a natural language description entered by the user about what the AI ​​agent is intended to do. The user prompt 420 allows the developer to enter content for the user prompt in the content field 422 as needed. The tool dropdown 430 allows the developer to select the tools the AI ​​agent will use, such as using an API for the application or calling an RPA robot to perform RPA.

[0094] The context dropdown 440 allows developers to configure context grounding for the AI ​​agent. The context configuration pane 442 allows developers to provide descriptions and ECS indexes via the description field 444 and the Elastic Common Schema (ECS) index field 446 for specific policy documents containing information about contracts, regulations, and what to do, in this example. Developers can also add additional contexts 450 to further complement the context grounding. Human escalation options may be configured via the dropdown 460.

[0095] The query field 470 allows the user to provide a query that the AI ​​agent will respond to. When the user clicks the execute button 480, the AI ​​agent executes the query. Moving to Figure 4B, the results of the AI ​​agent execution are then shown in the execute pane 490 as the AI ​​agent retrieves and outputs them.

[0096] Figure 5 shows an example of an AOP development interface 500 according to one embodiment of the present invention. The AOP development interface 500 includes AOP components 510, an AI agent 520, and an RPA 530, which can be selected by the user when developing a business process. These are selected and dragged onto a canvas 540, where the user can manually develop the AOP. In this example, the credit check is implemented by retrieving customer data from a database and calling an AI agent to analyze that customer data to determine the customer type (e.g., very likely to pay, likely to be late on payments, frequently unemployed, etc.). The type is then provided to an RPA robot, which conducts the credit check taking that information into consideration. Alternatively, the AOP developer can enter a description of the business process in field 550 and click the generate button 560. This text is provided to the LLM, which understands the requested business process and attempts to automatically create an AOP workflow. The AOP developer can then edit the AOP workflow as needed.

[0097] Figure 6 shows an example of an RPA development interface 600 according to one embodiment of the present invention. The RPA development interface 600 includes RPA components 610 that a user can select when developing an RPA workflow. These can be selected and dragged onto the canvas 620. Alternatively, an RPA developer can enter a description of the RPA in field 630 and click the generate button 640. This text is provided to the LLM, which understands the requested business process and attempts to automatically create an RPA workflow. The developer can then edit the RPA workflow as needed. Note that the functions shown and described with respect to Figures 4A, 4B, 5, and 6 may be provided in a single designer application in some embodiments.

[0098] Figure 7 shows an end-to-end AI agent, RPA robot, and AOP development and deployment system 700 according to one embodiment of the present invention. A designer application 710 enables developers to design AOP, AI agents, and RPA workflows. Once these are tested and validated, they are packaged and published to the automation database 720.

[0099] The conductor application 730 manages these automations, as well as the deployment of AOPs, AI agents, and RPA robots. When a human user or software process 732 requests the execution of an AOP, the conductor application 730 sends a start job command to the AOP engine 740, which then selects and starts the appropriate automation from AOP 742. While executing AOP 742, it may encounter a step implemented by an AI agent 750 or an RPA robot 760. In this case, the AOP engine 740 interrupts the AOP workflow execution and sends a request to the conductor application 730 to send a start job request to the appropriate AI agent 750 or RPA robot 760 to execute the step in question.

[0100] When an AI agent is requested, the conductor application 730 sends a start job request to the appropriate AI agent 750. This request may include natural language text or other information provided to the conductor application 730 from the AOP engine 740. The AI ​​agent 750 then performs the step by executing LLM 752 to assist in task execution. The AI ​​agent 750 then sends information related to the task (e.g., requested information, an indication that the step was completed, an indication that the step failed, etc.) to the conductor 730, which then provides this information to the AOP engine 740. The AOP engine 740 then resumes its operation.

[0101] When an RPA robot is requested, the conductor application 730 sends a start job request to the appropriate RPA robot 760. The RPA robot 760 then executes the requested RPA 762. The RPA robot 760 then sends information related to the task (e.g., the requested information, an indication that the step was completed, an indication that the step failed, etc.) to the conductor 730, which provides this information to the AOP engine 740. This allows the AOP engine 740 to resume its operation.

[0102] In some cases, human action may be required from AOP742, AI agent 750, or RPA762. In this case, AOP engine 740, AI agent 750, or RPA robot 760 contacts a human 770 for the human-in-the-loop portion of the automation. After the human completes the task, AOP engine 740, AI agent 750, or RPA robot 760 resumes the automated portion of the automation.

[0103] Figure 8 is an architectural diagram showing an agent-based automation and RPA system 800 according to one embodiment of the present invention. In some embodiments, the agent-based automation and RPA system 800 is part of the hyperautomation system 100 in Figure 1. The agent-based automation and RPA system 800 includes a designer 810 that enables developers to design automations for AI agents and RPA robots (e.g., workflows, natural language instructions for AI agents, context grounding, tool configurations, etc.). The designer 810 can provide solutions for application integration, as well as solutions for automating third-party applications, managed information technology (IT) tasks, and business IT processes. The designer 810 can facilitate the development of automation projects, which are graphical representations of business processes. In short, the designer 810 facilitates the development and deployment of automations for RPA robots and AI agents. In some embodiments, the designer 810 may be an application running on a user's desktop, an application running remotely in a VM, a web application, etc.

[0104] Automation projects enable the automation of rule-based processes by giving developers control over the execution order and relationships of custom steps developed within the workflow, i.e., the "activities" described above. One commercial example of an embodiment of Designer 810 is UiPath Studio®. Each activity may include actions such as clicking a button, reading a file, or writing to a log panel. In some embodiments, workflows may be nested or embedded.

[0105] Workflow types may include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences are particularly suitable for linear processes, allowing a flow from one activity to another without complicating the workflow. Flowcharts are particularly suitable for more complex business logic, allowing for the integration of decision-making and the connection of activities in a more diverse way through multiple branching logic operators. FSMs are particularly suitable for large-scale workflows. FSMs use a finite number of states in their execution, which are triggered by conditions (i.e., transitions) or activities. Global exception handlers are particularly suitable for determining workflow behavior and debugging processes when execution errors are encountered.

[0106] Once workflows and / or other configurations for AI agents are developed in Designer 810, the execution of business processes is orchestrated by Conductor 820. Conductor 820 orchestrates one or more robots 830, one or more AI agents 850, and / or one or more AOPs 870 that execute the workflows developed in Designer 810. One commercial example of an embodiment of Conductor 820 is UiPath Orchestrator®. Conductor 820 facilitates the creation, monitoring, and management of resources in the environment. Conductor 820 can function as an integration point with third-party solutions and applications. As described above, in some embodiments, Conductor 820 may be part of the core hyperautomation system 120 in Figure 1.

[0107] It should be noted that the RPA robot 830 can operate independently for deterministic processes. The AI ​​agent 850 and AOP 870 can also operate independently (for example, for non-deterministic processes). Alternatively, the RPA robot 830 or other AI agents 850 can be used as tools to achieve part of their agent-based automation. The AI ​​agent 850 can drive a composite automation that utilizes both the RPA robot 830 and the AI ​​agent 850, or vice versa, and the AOP 870 may include such composite automations.

[0108] The conductor 820 manages a fleet of robots 8530 and AI agents 850, and can connect and run RPA robots 830 and AI agents 850 from a central point (e.g., one requested by an AOP engine implementing AOP). The types of RPA robots 830 that can be managed may include, but are not limited to, manned robots, unmanned robots, development robots (similar to unmanned robots but used for development and testing purposes), and non-production robots (similar to manned robots but used for development and testing purposes). Manned robots are triggered by user events and operate in parallel with humans on the same computing system. Manned robots may be used with the conductor 820 for centralized process deployment and log recording media. Manned robots may assist human users in performing various tasks and may be triggered by user events. In some embodiments, this type of robot cannot start processes from the conductor 820 and / or run under a lock screen. In certain embodiments, manned robots can only be started from the robot tray or command prompt. In some embodiments, the manned robot should be operated under human supervision.

[0109] Unmanned robots can run autonomously within a virtual environment and automate many processes. Unmanned robots may be responsible for remote execution, monitoring, scheduling, and providing assistance with work queues. In some embodiments, debugging for all types of robots may be performed on the designer 810. Both manned and unmanned robots can automate a wide range of systems and applications, including but not limited to mainframes, web applications, VMs, enterprise applications (e.g., those provided by SAP®, Salesforce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0110] Conductor 820 may have a variety of capabilities, including but not limited to provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interoperability. Provisioning may include creating and maintaining connections between robots 830, AI agents 850, and / or AOP 870 and Conductor 820 (e.g., web applications). Deployment may include ensuring that the correct package versions for execution are delivered to assigned robots 830, AI agents 850, and / or AOPs. Configuration may include maintaining and delivering RPA robot and AI agent environments and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot and AI agent identification data and maintaining user privileges. Logging may include storing and indexing logs in a database (e.g., a Structured Query Language (SQL) database or a "not only" SQL (NoSQL) database) and / or other storage mechanisms (e.g., ElasticSearch®, which provides the ability to store and rapidly query large datasets). Conductor 820 can provide interoperability by acting as a communication hub for third-party solutions and / or applications.

[0111] Robot 830 is an execution agent that implements workflows built in Designer 810. One commercial example of some embodiments of Robot 830 is UiPath Robots®. In some embodiments, RPA Robot 830 installs the Microsoft Windows® Service Control Manager (SCM) management service by default. As a result, such RPA Robot 830 can open interactive Windows® sessions under the local system account and has Windows® service privileges.

[0112] In some embodiments, the RPA robot 830 may be installed in user mode. For such robots 830, this means they have the same privileges as the user on which the RPA robot 830 is installed. This functionality may also be available for high-density (HD) robots. HD robots ensure that each machine is fully utilized to its maximum potential. In some embodiments, any type of RPA robot 830 may be configured in an HD environment.

[0113] In some embodiments, the RPA robot 830 is divided into multiple components, each dedicated to a specific automation task. These robot components may include, but are not limited to, an SCM-managed robot service, a user-mode robot service, an executor, an agent, and a command line. The SCM-managed robot service manages and monitors Windows® sessions and acts as a proxy between the conductor 820 and the execution host (i.e., the computing system on which the robot 830 runs). These services are trusted and manage the credentials of the RPA robot 830. A console application is launched by the SCM under the local system.

[0114] In some embodiments, the user-mode robot service manages and monitors Windows® sessions and acts as a proxy between the conductor 820 and the execution host. The user-mode robot service can trust and manage the credentials of the RPA robot 830. If the SCM management robot service is not installed, the Windows® application may be launched automatically.

[0115] The executor can execute a given job (i.e., a workflow) under a Windows® session. The executor can recognize the dots per inch (DPI) setting for each monitor. The agent may be a Windows® Presentation Foundation (WPF) application that displays available jobs in a system tray window. Note that these agents are different from AI Agent 850. The agent is a client of the service and can request the start or stop of a job or change its settings. The command line is also a client of the service. The command line is a console application that can request the start of a job and wait for its output.

[0116] By dividing the components of the robot 830 as described above, developers, support users, and computing systems can more easily execute, identify, and track what each component is doing. For example, special actions can be configured for each component, such as setting different firewall rules for the executor and services. In some embodiments, the executor may always be aware of the DPI setting for each monitor. As a result, workflows can be executed at any DPI, regardless of the configuration of the computing system in which they were created. In some embodiments, projects from the designer 810 may also be independent of the browser zoom level. In some embodiments, DPI can be disabled for applications that are not DPI-aware or that are intentionally marked as not DPI-aware.

[0117] In this embodiment, the agent-based automation and RPA system 800 is part of a hyperautomation system, such as the hyperautomation system 100 in Figure 1. Developers can use the designer 810 to build and test RPA, AOP, and AI agents that utilize AI / ML models deployed within the core hyperautomation system 840 (for example, as part of its AI center). Such RPA robots can send inputs for execution of AI / ML models and receive outputs from them via the core hyperautomation system 840.

[0118] One or more of the RPA robots 830 may be listeners as described above. These listeners may provide the core hyperautomation system 840 with information about what the user is doing when using the computing system. This information can then be used by the core hyperautomation system for process mining, task mining, task capture, etc.

[0119] To enable users to launch RPA local robots, an assistant / chatbot (not shown) may be provided on the user's computing system. For example, the assistant / chatbot may be located in the system tray. The chatbot may have a user interface so that the user can see the text within the chatbot. Alternatively, the chatbot may lack a user interface and run in the background, listening to the user's speech using the computing system's microphone.

[0120] In some embodiments, data labeling may be performed by a user of the computing system on which the RPA robot or AI agent is running, or on another computing system on which the robot or AI agent provides information. For example, if a robot invokes an AI / ML model to perform CV on an image for a VM user, but the AI / ML model does not correctly identify a button on the screen, the user may draw a rectangle around the misidentified or unidentified component and, if applicable, provide text indicating the correct identification. This information is provided to the core hyperautomation system 540 and can then be used later to train a new version of the AI / ML model.

[0121] Figure 9 is an architecture diagram showing a deployed RPA system 900 according to one embodiment of the present invention. In some embodiments, the RPA system 900 may be part of the agent-based automation and RPA system 800 in Figure 8 and / or the hyperautomation system 100 in Figure 1. Note that the architecture of the deployed RPA system 900 may not be used in some embodiments. The deployed RPA system 900 may be a cloud-based system, an on-premise system, a desktop-based system providing enterprise-level, user-level, or device-level automation solutions, etc., and may be used to automate different computing processes.

[0122] It should be noted that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 910 includes an executor 912, an execution agent 914, and a designer 916. However, in some embodiments, the designer 916 may not run on the same computing system as the executor 912 and the execution agent 914. The executor 912 is a running process. Multiple business projects may run simultaneously. The execution agent 914 (e.g., Windows® service) is a single point of contact for all executors 912 in this embodiment. All messages in this embodiment are recorded in the conductor 940, which further processes them via the database server 950, the AI / ML server 960, the indexer server 970, or any combination thereof. As described above with respect to Figure 8, the executor 912 may be a robot component.

[0123] In some embodiments, an RPA robot represents an association between a machine name and a username. This robot can manage multiple executors simultaneously. On computing systems that support the simultaneous execution of multiple interactive sessions (e.g., Windows® Server 2012), multiple robots may run concurrently, each using a unique username within a different Windows® session. These are referred to as HD robots.

[0124] The execution agent 914 is also responsible for transmitting the robot's status (for example, periodically sending "heartbeat" messages to indicate that the robot is still functioning) and downloading the necessary versions of packages to be executed. In some embodiments, communication between the execution agent 914 and the conductor 940 is always initiated by the execution agent 914. In notification scenarios, the execution agent 914 may open a WebSocket channel, which is then used by the conductor 940 to send commands to the robot (e.g., start, stop, etc.).

[0125] To reduce the complexity of Figure 9, it is not shown here, but please note that the AI ​​agent can also interact with the conductor 940, as described above with respect to, for example, Figures 1 and 8. The conductor 940 can orchestrate the operation of the AI ​​agent. The conductor 940 can also facilitate interaction between the AI ​​agent and the AI / ML model via the AI / ML server 960, which can store and / or easily access the generated AI model.

[0126] The listener 930 monitors and records data relating to user interactions with the manned computing system in which the listener 930 resides and / or the operation of the unmanned computing system. The listener 930 may be an RPA robot, part of an operating system, a downloadable application for the computing system, or any other software and / or hardware that does not depart from the scope of the present invention. In fact, in some embodiments, the logic of the listener is partially or completely implemented by physical hardware.

[0127] The server side includes a presentation layer (web application 942, Open Data Protocol (oData) Representative State Transfer (REST) ​​Application Programming Interface (API) endpoint 944, and notification and monitoring 946), a service layer (API implementation / business logic 948), and a persistence layer (database server 950, AI / ML server 960, and indexer server 970). Conductor 940 includes the web application 942, the oData REST API endpoint 944, notification and monitoring 946, and API implementation / business logic 948. In some embodiments, most actions performed by the user on the interface of Conductor 940 (e.g., via a browser 920) are performed by calling various APIs. Such actions may include, but are not limited to, starting jobs on a robot, adding / deleting data to / from a queue, scheduling unattended execution of jobs, etc., without departing from the scope of the present invention. The web application 942 is the visual layer of the server platform. In this embodiment, the web application 942 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other form can be used without departing from the scope of the present invention. In this embodiment, the user interacts with web pages from the web application 942 via the browser 920 and performs various actions to control the conductor 940. For example, the user can create robot groups, assign packages to robots, analyze logs per robot and / or per process, start and stop robots, and so on.

[0128] In addition to the web application 942, the conductor 940 also includes a service layer that exposes an oData REST API endpoint 944. However, other endpoints may also be included without departing from the scope of the present invention. In this embodiment, the REST API is utilized by both the web application 942 and the execution agent 914. In this embodiment, the execution agent 914 is the supervisor of one or more robots on the client computer.

[0129] The REST API in this embodiment includes configuration, logging, monitoring, and queuing functions. The configuration endpoint may, in some embodiments, be used to define and configure application users, permissions, robots, assets, releases, and environments. The logging REST endpoint may be used to log different information, such as errors, explicit messages sent by robots, and other environment-specific information. The deployment REST endpoint may be used by robots to query the package version to be executed when a start job command is used in conductor 940. The queuing REST endpoint may be responsible for queue and queue item management, including adding data to the queue, retrieving transactions from the queue, and setting the status of transactions.

[0130] The monitoring REST endpoint may monitor the web application 942 and the execution agent 914. The notification and monitoring API 946 may be a REST endpoint used for registering the execution agent 914, delivering configuration settings to the execution agent 914, and sending and receiving notifications from the server and the execution agent 914. In some embodiments, the notification and monitoring API 946 may also use WebSocket communication.

[0131] In some embodiments, APIs within the service layer may be accessed, for example, through the configuration of appropriate API access paths depending on whether the conductor 940 and the overall hyperautomation system are deployed on-premises or cloud-based. The API for the conductor 940 may provide custom methods for querying statistics about various entities registered with the conductor 940. In some embodiments, each logical resource may be an oData entity. In such entities, components such as robots, processes, and queues may have properties, relationships, and behaviors. In some embodiments, the conductor 940 API may be utilized by a web application 942 and / or execution agent 914 in two ways: (1) by obtaining API access information from the conductor 940, or (2) by registering an external application and using the oAuth flow.

[0132] The persistence layer in this embodiment includes three servers: a database server 950 (e.g., an SQL server), an AI / ML server 960 (e.g., a server providing AI / ML model provisioning services such as AI Center functions), and an indexer server 970. In this embodiment, the database server 950 stores the configuration of robots and AI agents, groups of robots and AI agents, AOP, associated processes, users, roles, schedules, etc. This information is managed via a web application 942 in some embodiments. The database server 950 may manage queues and queue items. In some embodiments, the database server 950 may store messages recorded by robots and AI agents (in addition to or instead of the indexer server 970). The database server 950 may also store process mining, task mining, and / or task capture-related data received, for example, from a listener 930 installed on the client side. Although the arrow between listener 930 and database 950 is not shown, it should be understood that in some embodiments, listener 930 can communicate with database 950 and vice versa. This data may be stored in the form of PDDs, images, XAML files, etc. Note that structured data and / or unstructured data may be stored. Listener 930 may be configured to intercept user actions, processes, tasks, and performance metrics on the computing system in which listener 930 resides. For example, listener 930 may record user actions on the computing system (e.g., clicks, input characters, location, applications, active elements, time, etc.) and convert them into a format suitable for providing and storing them in the database server 950.

[0133] The AI / ML Server 960 facilitates the integration of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options make such functionality accessible even to non-data scientists. Deployed automations (e.g., RPA robots and / or AI agents) can invoke AI / ML models from the AI / ML Server 960. AI / ML model performance can be monitored and trained and improved using human-validated data. The AI / ML Server 960 can schedule and execute training jobs to train new versions of AI / ML models. The AI / ML model server can also store and / or access generated AI models.

[0134] The AI / ML server 960 can store data relating to AI / ML models and ML packages for configuring various ML skills for users during development. As used herein, "ML skill" refers to a pre-built and trained ML model for a process that can be used, for example, by automation. The AI / ML server 960 can also store data relating to document understanding technologies and frameworks, algorithms and software packages for various AI / ML capabilities, including but not limited to intent analysis, NLP, speech analysis, and various types of AI / ML models.

[0135] An indexer server 970, which is optional in some embodiments, stores and indexes information recorded by the robot. In certain embodiments, the indexer server 970 can be disabled via configuration settings. In some embodiments, the indexer server 970 uses ElasticSearch®, an open-source full-text search engine. Messages recorded by the robot (e.g., those recorded using activities such as log messages or write lines) are sent to the indexer server 970 via a logging REST endpoint, where they can be indexed for future use.

[0136] Figure 10 is an architectural diagram showing the relationships between a designer 1010, activities 1020, 1030, 1040, 1050, a driver 1060, an API 1070, and an AI / ML model 1080 according to one embodiment of the present invention. As described above, the developer uses the designer 1010 to develop workflows and automations to be executed by RPA robots, AI agents, and AOP engines. The developer can design and configure RPA robot workflows 1012, agent-based automations 1014 for AI agents (e.g., providing natural language description, context grounding, tools, etc. for AI agents), and AOP 1016. See, for example, Figures 4A, 4B, 5, and 6. Various types of activities may be displayed to the developer in some embodiments. The designer 1010 may be local to the user's computing system or remote to it (e.g., accessed via a VM or in the form of a local web browser interacting with a remote web server). A workflow for an RPA robot may include a user-defined activity 1020, an API-driven activity 1030, an AI / ML activity 1040, and / or a UI automation activity 1050. The user-defined activity 1020 and the API-driven activity 1040 interact with the application via their respective APIs. In some embodiments, the user-defined activity 1020 and / or the AI / ML activity 1040 may call one or more AI / ML models 1080, which may reside locally in the computing system on which the robot operates, or remotely from there.

[0137] In some embodiments, it is possible to identify non-textual visual elements within an image, which is referred to herein as CV. However, it should be noted that in some embodiments, CV includes OCR. CV may be performed, at least in part, by the AI / ML model 1080. CV activities relating to such elements may include, but are not limited to, text extraction from segmented label data using OCR, fuzzy text matching, clipping of segmented label data using ML, and comparison of extracted text in the label data with ground truth data. In some embodiments, the number of activities that can be implemented in user-defined activity 1020 may be in the hundreds or thousands. However, any number and / or types of activities may be used without departing from the scope of the present invention.

[0138] The UI automation activity 1050 is a subset of special low-level activities written in low-level code to facilitate interaction with the screen. The UI automation activity 1050 facilitates these interactions via a driver 1060 that enables the robot to interact with desired software. For example, the driver 1060 may include an operating system (OS) driver 1062, a browser driver 1064, a VM driver 1066, an enterprise application driver 1068, etc. In some embodiments, one or more AI / ML models 1080 may be used by the UI automation activity 1050 to perform interactions with a computing system. In certain embodiments, the AI / ML model 1080 may complement or completely replace the driver 1060. In fact, in certain embodiments, the driver 1060 is not included.

[0139] Driver 1060 can interact with the OS at a low level via OS driver 1062 to explore hooks, monitor keys, and so on. Driver 1060 can facilitate integration with Chrome®, IE®, Citrix®, SAP®, and others. For example, a "click" activity can perform the same role in these different applications via driver 1060.

[0140] Figure 11 is an architectural diagram showing a computing system 1100 configured to perform embodiments described herein. In some embodiments, the computing system 1100 may be one or more computing systems illustrated and / or described herein. In certain embodiments, the computing system 1100 may be part of a hyperautomation system as shown in Figures 1 and 8. The computing system 1100 includes a bus 1105 or other communication mechanism for communicating information and a processor 1110 coupled to the bus 1105 for processing information. The processor 1110 may be any type of general-purpose or purpose-specific processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor 1110 may also have multiple processing cores, at least some of which may be configured to perform specific functions. In some embodiments, multi-parallel processing may be used. In certain embodiments, at least one of the processors 1110 may be a neuromorphic circuit including processing elements that mimic biological neurons. In some embodiments, neuromorphic circuits may not require typical components of a von Neumann computing architecture.

[0141] The computing system 1100 further includes memory 1115 for storing information and instructions to be executed by the processor 1110. Memory 1115 may consist of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage devices such as magnetic disks or optical disks, or any other type of non-temporary computer-readable medium or a combination thereof. The non-temporary computer-readable medium may be any available medium accessible by the processor 1110, and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both. The computing system 1100 includes communication devices 1120, such as transceivers, for providing access to a communication network via wireless and / or wired connections. In some embodiments, the communication devices 1120 may include one or more antennas, including single, array, phased, switched, beamforming, beam steering, combinations thereof, and / or any other antenna configuration, without departing from the scope of the invention.

[0142] The processor 1110 is further coupled to the display device 1125 via the bus 1105. Any suitable display device and haptic I / O can be used without departing from the scope of the present invention. A keyboard 1130 and a cursor control device 1135, such as a computer mouse or touchpad, are also coupled to the bus 1105 to enable the user to interface with the computing system 1100. However, in certain embodiments, there is no physical keyboard and mouse, and the user can only interact with the device via the display device 1125 and / or touchpad (not shown). Any type and combination of input devices can be used as a design choice. In certain embodiments, there is no physical input device and / or display device. For example, the user can remotely interact with the computing system 1100 via another computing system in communication, or the computing system 1100 can operate autonomously.

[0143] Memory 1115 stores software modules that provide functionality when executed by processor 1110. These modules include an operating system 1140 for the computing system 1100. The modules further include an agent learning module 1145 configured to execute all or part of process 1400 in Figure 14, or any other process or subprocess or derivative thereof as described herein. The computing system 1100 may include one or more additional functionality modules 1150 that provide additional functionality.

[0144] Those skilled in the art will understand that “computing system” can be implemented as a server, embedded computing system, personal computer, console, personal digital assistant (PDA), mobile phone, tablet computing device, smartwatch, quantum computing system, or any other suitable computing device or group of devices without departing from the scope of the present invention. The fact that the above functions are presented as being performed by a “system” is not intended to limit the scope of the present invention in any sense, but rather to provide an example of a number of embodiments of the present invention. In fact, the methods, systems, and devices disclosed herein can be implemented in localized and distributed forms that are consistent with computing technologies, including cloud computing systems. The computing system may be part of, or accessible from, a LAN, mobile communication network, satellite communication network, internet, public or private cloud, hybrid cloud, server farm, or any combination thereof. Any localized or distributed architecture can be used without departing from the scope of the present invention.

[0145] It should be noted that some of the system features described herein are presented as modules to particularly emphasize their implementation independence. For example, modules may be implemented as hardware circuits including custom very large-scale integrated circuits (VLSI) or commercially available semiconductors such as gate arrays, logic chips, transistors, or other discrete components. Modules may also be implemented in programmable hardware devices such as field-programmable gate arrays, programmable array logic, programmable logic devices, and graphics processing units.

[0146] Modules can also be implemented, at least partially, as software executed by various types of processors. For example, an identified executable code unit may include one or more physical or logical blocks of computer instructions that can be organized as objects, procedures, or functions. However, the executable code of an identified module does not need to be physically located in one place, and may include multiple instructions stored in different locations, which, when logically combined, constitute the module and achieve the purpose described in the module. Furthermore, modules may be stored on computer-readable media, such as hard disk drives, flash devices, RAM, tapes, and / or any other such non-temporary computer-readable media used to store data without departing from the scope of the present invention.

[0147] In fact, a module of executable code may be a single instruction, a number of instructions, span multiple different code segments, be distributed across different programs, and be distributed across multiple memory devices. Similarly, operational data may be identified and illustrated within a module as herein, but may be implemented in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single dataset, distributed across different locations including entire different storage devices, and at least a portion of it may exist only as electronic signals on a system or network.

[0148] Without departing from the scope of the present invention, various types of AI / ML models can be trained and deployed. For example, Figure 12A shows an example of a neural network 1200 according to one embodiment of the present invention. The neural network 1200 includes multiple hidden layers. Both DLNNs and shallow learning neural networks (SLNNs) typically have multiple layers, although SLNNs may have only one or two layers, and are typically fewer than DLNNs. Typically, a neural network architecture includes an input layer, multiple hidden layers, and an output layer, and the neural network 1200 is no exception.

[0149] DLNNs often have many layers (e.g., 10, 50, 200, etc.), and subsequent layers typically reuse features from preceding layers to compute more complex and general functions. SLNNs, on the other hand, tend to have fewer layers and are trained relatively quickly because specialized features are pre-generated from raw data samples. However, feature extraction is labor-intensive. In contrast, DLNNs typically do not require specialized features but tend to take longer to train and have more layers.

[0150] In both approaches, layers are trained simultaneously on the training set, and overfitting is typically checked on a separate cross-validation set. Both techniques can yield excellent results, and there is considerable enthusiasm for both approaches. The optimal size, shape, and number of individual layers will vary depending on the problem each neural network addresses.

[0151] Returning to Figure 12A, inputs 1, 2, 3, ..., N, etc., are provided as the input layer and supplied as inputs to J neurons in Hidden Layer 1. Various other inputs are also possible, including, but are not limited to, computing system state information, published automations, business rules, information on which RPA workflows and / or tasks are relevant, initial definitions of automations, process automation documents, etc. In this example, all of these inputs are supplied to each neuron, but various architectures may be used individually or in combination, including but not limited to feedforward networks, radial basis networks, deep feedforward networks, deep convolutional inverse graphics networks, convolutional neural networks, recurrent neural networks, artificial neural networks, long short-term memory networks, gated recurrent unit networks, generative adversarial networks, liquid state machines, autoencoders, variational autoencoders, denoising autoencoders, sparse autoencoders, extreme learning machines, echo state networks, Markov chains, Hopfield networks, Boltzmann machines, restricted Boltzmann machines, deep residual networks, Kohonen networks, deep belief networks, deep convolutional networks, support vector machines, neural Turing machines, or any other suitable type or combination of neural networks that do not depart from the scope of the present invention.

[0152] Hidden layer 2 receives input from hidden layer 1, hidden layer 3 receives input from hidden layer 2, and so on for all hidden layers until the last hidden layer provides its output as input to the output layer. Although multiple proposals are shown as outputs here, in some embodiments only a single output proposal is provided. In certain embodiments, proposals are ranked based on confidence scores. In this embodiment, the outputs are outputs 1, 2, 3, ..., N.

[0153] It should be noted that the number of neurons I, J, K, and L are not necessarily equal. Therefore, any desired number of layers can be used for a given layer of the neural network 1200 without departing from the scope of the present invention. In fact, in certain embodiments, the types of neurons within a given layer do not all have to be the same.

[0154] The neural network 1200 is trained to assign confidence scores to appropriate outputs. To reduce inaccurate predictions, in some embodiments, only results with confidence scores that meet or exceed a confidence threshold may be provided. For example, if the confidence threshold is 80%, outputs with confidence scores exceeding this value may be used, and the rest may be ignored.

[0155] Neural networks are typically probabilistic constructs that have confidence scores. These scores can be learned based on how well the AI / ML model correctly identified similar inputs during training. Common types of confidence scores include decimals between 0 and 1 (which can also be interpreted as confidence percentages), numerical values ​​between negative infinity and positive infinity, and a set of representations (e.g., "low," "medium," "high"). Various post-processing calibration techniques such as temperature scaling, batch normalization, weight decay, and negative log-likelihood (NLL) may also be used to attempt to obtain more accurate confidence scores.

[0156] In a neural network, "neurons" are typically implemented algorithmically as mathematical functions based on the functions of biological neurons. A neuron receives weighted inputs and has a summation function and activation function that govern whether or not it passes the output to the next layer. This activation function can be a nonlinear threshold activity function, i.e., modified linear unit (ReLU) nonlinearity, where nothing happens if the value is below a threshold, and a linear response occurs above the threshold. The summation and ReLU functions are used in deep learning because real neurons can have nearly identical activity functions. Information can be subtracted, added, etc., through linear transformations. Essentially, a neuron functions as a gating function that passes the output to the next layer, governed by its underlying mathematical function. In some embodiments, different functions may be used for at least some of the neurons.

[0157] An example of neuron 1210 is shown in Figure 12B. The inputs x1, x2, ..., xn from the preceding layer are assigned weights w1, w2, ..., wn, respectively. Therefore, the collective input from preceding neuron 1 is weighted w1x1. These weighted inputs are used for the biased summation function of the neurons, which is expressed as follows:

number

[0158] This sum is compared to the activation function f(x) to determine whether or not a neuron "fires". For example, f(x) can be given by the following:

number

[0159] Therefore, the output y of neuron 1210 can be given by the following:

number

[0160] In this case, neuron 1210 is a single-layer perceptron. However, any suitable neuron type or combination of multiple neuron types can be used without departing from the scope of the present invention. It should also be noted that in some embodiments, the range of weight values ​​and / or the range of activation function output values ​​may differ.

[0161] A goal, or "reward function," is often used. The reward function guides the exploration of the state space, exploring intermediate transitions and steps along with both short-term and long-term rewards, in an attempt to achieve the goal (e.g., finding the most accurate answer to a user query based on relevant metrics). During training, various labeled data passes through the neural network. Successful classifications strengthen the weight of the input to the neuron, while unsuccessful classifications weaken it. A cost function such as mean squared error (MSE) or gradient descent is used, allowing for stronger penalties for significantly wrong predictions than for slightly wrong predictions. If the performance of the AI / ML model does not improve after a predetermined number of training iterations, data scientists may modify the reward function, provide corrections for wrong predictions, etc.

[0162] Backpropagation is a technique for optimizing synaptic weights in feedforward neural networks. Backpropagation is used to "open the hood" of the neural network's hidden layers, see how much each node is responsible for the loss, and then update the weights to minimize the loss by assigning lower weights to nodes with higher error rates and vice versa. In other words, backpropagation allows data scientists to iteratively adjust the weights to minimize the difference between the actual output and the desired output.

[0163] Backpropagation algorithms are mathematically based on optimization theory. In supervised learning, training data with known outputs is passed through a neural network, and the error is calculated from the known target output using a cost function, which gives the error for backpropagation. The error is calculated from the output, and this error is converted into a correction to the network weights to minimize the error.

[0164] In supervised learning, an example of backpropagation is shown below. The column vector input x is processed through N nonlinear activity functions fi across each layer i=1,...,N of the network. The output at a given layer is first multiplied by a synaptic matrix, and then a bias vector is added. The network output o is obtained as follows.

number

[0165] In some embodiments, o is compared to the target output t, and the resulting error is one that is to be minimized.

number

[0166] To minimize the error by changing the synaptic weights Wi of each layer, optimization of the form of the gradient descent procedure may be used. The gradient descent procedure requires calculating the output o given an input x corresponding to a known target output t, and generating the error o--t. This overall error is then propagated backward, and the local error for weight updating is obtained by a calculation similar to, but not entirely identical to, that used for forward propagation. In particular, the backpropagation step is usually p j (n j )=f j '(n j This requires an activity function of the form ), where n j This is the network activity in layer j (i.e., n j =W j o j-1 +bj ), and o j =f j (n j ), and the apostrophe ' denotes the derivative of the activation function f.

[0167] Weight update can be calculated according to a predetermined formula.

Mathematical Expression

[0168] Here, o denotes the Hadamard product (i.e., the element-wise product of two vectors), T denotes matrix transposition, o j is f j (W j o j-1 +b j ), and o0=x. Here, the learning rate η is selected in consideration of machine learning considerations. In the following, this is associated with the neural Hebbian learning mechanism used in neural implementation. It should be noted that synapses W and b can be combined into one large synapse matrix. In this case, it is assumed that 1 is added to the input vector, and an additional column representing the b synapse is included in W.

[0169] The AI / ML model can be trained over multiple epochs until a sufficient accuracy level is reached (for example, 97% or more using an F2 or F4 threshold for detection, and about 2000 epochs). This accuracy level may, in some embodiments, be determined using an F1 score, an F2 score, an F4 score, or any other suitable technique that does not depart from the scope of the present invention. After being trained on training data, the AI / ML model can be tested on a set of evaluation data that the AI / ML model has never encountered before. This helps prevent the situation where the AI / ML model is "overfitted" and functions well on training data but does not function well on other data.

[0170] In some embodiments, it may be unclear what level of accuracy an AI / ML model can achieve. Therefore, if the accuracy of the AI / ML model begins to decline when analyzing evaluation data (i.e., the model performs well on training data but begins to decline on evaluation data), the AI / ML model may undergo further training for more epochs on the training data (and / or new training data). In some embodiments, an AI / ML model is deployed only if its accuracy reaches a predetermined level, or only if the accuracy of the trained AI / ML model is better than that of an existing deployed AI / ML model. In certain embodiments, a set of trained AI / ML models may be used to accomplish a task. For example, one AI / ML model may be trained for image recognition, another for text recognition, and yet another for recognizing semantic and / or ontological associations.

[0171] It should be noted that, in addition to or instead of neural networks, transformer networks such as SentenceTransformers™ may be used in some embodiments. This is a state-of-the-art Python™ framework for sentence, text, and image embedding. Such transformer networks learn associations between words and phrases that have both high and low scores. This trains the AI / ML model to determine which inputs are close to and which are not. Rather than using only word / phrase pairs, transformer networks may also use field lengths and field types.

[0172] As described above, in some embodiments, NLP models such as word2vec, BERT, GPT-3, ChatGPT, and other LLMs may be used to facilitate semantic comprehension and provide more accurate and human-like responses. Other techniques, such as clustering algorithms, may also be used to find similarities between groups of elements. Clustering algorithms may include, but are not limited to, density-based algorithms, distribution-based algorithms, centroid-based algorithms, hierarchical-based algorithms, K-means clustering algorithms, DBSCAN clustering algorithms, Gaussian mixture model (GMM) algorithms, and BIRCH algorithms. Such techniques may also be useful for categorization.

[0173] Figure 13 is an architecture diagram showing a reference architecture 1300 for a generative AI model according to one embodiment of the present invention. This architecture consists of multiple layers: an API plugin, a prompt library, vector data source ingestion, access processing control, a model training pipeline, an evaluation layer for evaluating hallucination / telemetry / evaluation, a BYOM embedding layer, and an LLM orchestration layer. There are also search plugins, access control plugins, and API plugins for integration into enterprise systems.

[0174] This embodiment has three main flows.

[0175] Data Ingestion and Training Flow: Data is read from multiple data stores, preprocessed, chunked, and trained through an embedded model (e.g., Search Augmentation Generator (RAG)) and training pipeline (i.e., fine-tuning). A vector database stores the chunked document embeddings, enabling better semantic-based and similarity-based data retrieval.

[0176] Prompt enhancement using data retrieval: When a user query reaches the API layer, a prompt is selected, and then, before the prompt is passed to the LLM layer, a data retrieval is performed through a vector database or API plugin to obtain appropriate contextual data.

[0177] LLM Inference: Here, there is a choice between using a general-purpose or self-hosted infrastructure model. A finely tuned model tailored to a specific task or use case may be used. The response is evaluated for accuracy, including hallucination, and other metrics.

[0178] It should be noted that in some embodiments, generative AI models having multiple "heads" may be used. A head refers to the output layer of the generative AI model. A generative AI model, such as generative AI model 172 in Figure 1, typically has a series of layers, and each head often shares the first few layers of the model before branching off into its own unique layers.

[0179] Figure 14 is a flowchart showing a process 1400 for training an AI / ML model according to one embodiment of the present invention. In some embodiments, as described above, the AI / ML model may be a generative AI model. In the case of a neural network, its architecture typically includes multiple neuron layers, including an input layer, an output layer, and hidden layers. See, for example, Figures 12A and 12B. The intermediate hidden layers process the input data and generate an intermediate representation of the input used to generate the output. These hidden layers may include various types of neurons, such as convolutional neurons, recursive neurons, and / or transformer neurons. A generative AI model may also have various layers.

[0180] In some embodiments, the training process begins in 1410 with providing training data, whether labeled or unlabeled. For generative AI models, which are commonly trained, the training process may be omitted unless a finely tuned model is desired, as will be explained in more detail later. Subsequently, the AI / ML model is trained over multiple epochs in 1420, and the results are reviewed in 1430. While various types of AI / ML models can be used, LLMs and other generative AI models are typically trained (fine-tuned) using a process called "supervised learning," which was also discussed above. Supervised learning involves providing the model with a large dataset, which the model uses to learn the relationship between inputs and outputs. During the training process, the model adjusts the weights and biases of the neurons in the neural network to minimize the difference between the predicted output and the actual output in the training dataset.

[0181] One aspect of the model in some embodiments is the use of transfer learning. For example, transfer learning can utilize a pre-trained model such as ChatGPT and fine-tune it for a specific task or domain in step 1420. This allows the model to leverage knowledge already learned in the pre-training phase and adapt to a specific application throughout the training phase in step 1420.

[0182] The pre-training phase involves training the model on an initial training dataset, which may be more general. In this phase, the model learns the relationships within the data. The fine-tuning phase (performed in some embodiments in addition to or instead of the initial training phase in step 1420, for example, when the pre-trained model is used as the initial foundation for the final model) adapts the pre-trained model to a specific task or domain by training it on a smaller dataset specific to that task. For example, in some embodiments, the model may focus on a particular type of data source. This may allow it to more accurately identify data elements within that data source than a generative AI model that is only pre-trained. Fine-tuning allows the model to learn the nuances of its source, such as specific vocabulary and syntax, specific graphical properties, specific data formats, etc., without requiring as much data as would be needed to train the model from scratch. By leveraging the knowledge learned in the pre-training phase, the fine-tuning model can achieve state-of-the-art performance for a particular task with relatively little additional training data.

[0183] In some embodiments, if the AI / ML model does not meet the desired confidence threshold at 1440, the training data is augmented and / or the reward function is modified so that the AI / ML model can better achieve its objective at 1450, and the process returns to step 1420. If the AI / ML model meets the confidence threshold at 1440, the AI / ML model is tested on evaluation data at 1460 to verify that the AI / ML model has generalized well and is not overfitted to the training data. The evaluation data includes information that the AI / ML model has not previously processed. If the confidence threshold is met on the evaluation data at 1470, the AI / ML model is deployed at 1480. Otherwise, the process returns to step 1450, and the AI / ML model is further trained.

[0184] A process includes multiple activities performed by one or more computers to carry out various tasks. Embodiments described herein provide a method for adjusting user input to a process by automatically modifying the process with an AI agent, for example, by adding one or more steps that request user input to the process, and / or removing one or more steps that request user input from the process. According to one or more embodiments, such adjustment of user input in a process may be performed by one or more AI agents, for example, during process design based on process design data, during a specific execution instance of the process based on process execution data, and / or after one or more execution instances based on process history execution data. Advantageously, by automatically adjusting user input to a process, user input is requested only when necessary, thereby reducing unnecessary user input requests from the user, shortening process execution time, improving memory usage of the AI ​​agent by reducing the amount of storage space required for data, and improving processor efficiency for AI agent execution by reducing the number of calls and actions, thereby shortening execution time.

[0185] Figure 15 shows a method 1500 in which an AI agent is used to automatically modify a process to adjust user input to the process, according to one or more embodiments. The steps and / or substeps of method 1500 may be performed by one or more computing devices, such as the computing system 1100 in Figure 11.

[0186] In step 1502 of Figure 15, process data associated with the process is received. The process includes one or more activities for performing a task. The process is a computer process that is automatically executed by one or more computing systems, such as computing system 1100 in Figure 11. In one embodiment, the process is an RPA process that automatically performs a task using one or more RPA robots running on one or more computing systems. However, it should be understood that the process can be any suitable process that can be modeled as a workflow. The process may invoke or otherwise utilize one or more tools, such as RPA robots, AI agents, AOPs, or applications. Process data may be obtained, for example, during the design of the process, during a particular execution instance of the process, or after one or more execution instances. Process data may include any data associated with the process.

[0187] In one embodiment, process data may include process design data acquired, for example, during process design (i.e., before process execution). Process design data includes any data related to process design. In one embodiment, process design data may include a workflow visualization of the process design. This workflow visualization may be defined by a designer application that designs the process. The designer application may be, for example, designer application 154 in Figure 1, agent service interface 400 in Figures 4A and 4B, designer application 710 in Figure 7, designer 810 in Figure 8, or designer 1010 in Figure 10. In one example, the workflow visualization may be a directed graph that visually maps the activities, decisions, sequences, etc., of the process. An exemplary directed graph of a process is shown in Figure 16.

[0188] Figure 16 shows an exemplary directed graph 1600 of a process for processing and paying invoices according to one or more embodiments. The directed graph 1600 includes activities 1602–1614, which represent a predefined sequence of steps within the process. As shown in Figure 16, each activity 1602–1614 is represented as a node, and each transition between activities 1602–1614 is represented as an edge connecting the nodes. The transitions between activities represent the execution of the process from a source activity to a destination activity. The directed graph 1600 begins with the invoice receipt activity 1602 and proceeds to the received invoice confirmation activity 1604. If the received invoice confirmation activity 1604 determines that there is missing information in the received invoice, the directed graph 1600 proceeds to the data request activity 1606 and the contract terms confirmation activity 1608, and then proceeds to the final invoice confirmation activity 1610. If the received invoice confirmation activity 1604 determines that there is no missing information in the received invoice, the directed graph 1600 proceeds directly to the final invoice confirmation activity 1610. Subsequently, the directed graph 1600 proceeds to the invoice approval activity 1612 and the invoice payment activity 1614. The execution of the process is recorded in the form of an event log.

[0189] Returning to step 1502 in Figure 15, in other examples, process design data may include requirements data (e.g., the initial PDD (Process Definition Document) or SDD (Software Design Document) of the process) that defines the requirements of the process, process task mining data, data from designer applications for designing, changing, or modifying the process, user conversation logs discussing the process design, etc.

[0190] In one embodiment, process data may include, for example, process runtime data acquired during a specific execution instance of a process. Process runtime data includes any data related to that specific execution instance of the process. For example, process runtime data may include data on the activity of the currently running process, activities performed prior to that specific execution instance, the currently running activity, or the elapsed execution time of that specific execution instance of the overall process, the execution trace of that specific execution instance of the process, and the event log of that specific execution instance of the process.

[0191] In one embodiment, process data may include, for example, process history execution data obtained after one or more execution instances of the process. Process history execution data may include historical data over a predetermined period (e.g., the past six months). Process history execution data may include any data related to one or more execution instances of the process. In one example, process history execution data may include event logs of one or more preceding execution instances of the process. Event logs record events that occur during the execution of the process. An event refers to the execution of an activity at a predetermined point in time for a given case. The event may be represented as a tuple containing an activity, a case identifier, and a timestamp. An exemplary event log is shown in Figure 17.

[0192] Figure 17 shows an exemplary event log 1700 of process execution according to one or more embodiments. The event log 1700 is an event log that records the execution of a process represented by the directed graph 1600 in Figure 16, which will be shown and explained with reference to Figure 16. The event log 1700 records events that occurred during two execution instances of the process, corresponding to case ID 1 and case ID 2. As shown in Figure 17, the event log 1700 is configured as a table having a row 1702 corresponding to each event and a column 1704 that identifies the attributes of the event in the cell at the intersection of row 1702 and column 1704. Specifically, each row 1702 is associated with an event representing the execution of activities 1602-1614 (identified in column 1704-B), the execution timestamp of activities 1602-1614 (identified in column 1704-C), and a case ID (identified in column 1704-A) that identifies the execution instance of activities 1602-1614 that was performed. It should be understood that the event log 1700 may be in any other appropriate format and may include additional columns 1704 that identify other attributes of the event.

[0193] Returning to step 1502 in Figure 15, in other examples, process history execution data may include a process model representing process executions in one or more execution instances, data on the success or failure of the process, tools used during process execution (e.g., RPA robots, AI agents, AOP, applications, etc.), exceptions or errors that occurred during process execution, human intervention / input received during process execution, or any other appropriate data related to process execution.

[0194] Process data may be acquired or collected by, for example, an AI agent (e.g., the AI ​​agent used in step 1504 of Figure 15), and may be integrated with monitoring tools for receiving real-time data, or with messaging platforms for receiving messages or alerts. Process data may be received by, for example, loading the process data from a storage device or memory of a computer system (e.g., memory 1115 of computing system 1100 in Figure 11), or by receiving the process data from a remote computer system (e.g., computing system 1100 in Figure 11).

[0195] In step 1504 of Figure 15, the process is automatically modified based on process data using an AI agent to adjust user input to the process. User input may include any input from the user, such as user dialogue, feedback, corrections, solutions, and validation.

[0196] In one embodiment, for example, during process design, the process is automatically modified by an AI agent based on process design data. In another embodiment, for example, during a specific execution instance of the process, the process is automatically and dynamically modified by an AI agent based on process execution data associated with that specific execution instance. In yet another embodiment, for example, after one or more execution instances of the process, the process is automatically modified by an AI agent based on process history execution data (and optionally process design data). Process history execution data can be preprocessed by analysis, clustering, anomaly detection, etc. For example, process history execution data can be preprocessed to determine performance characteristics (e.g., execution time, resource usage, error rate, trajectory frequency), identify bottlenecks, or identify trends or patterns. In one example, process history execution data can be preprocessed to identify process segments that are likely to have bottlenecks, in accordance with U.S. Patent No. 11,314,561, which is incorporated herein by reference in its entirety.

[0197] In one embodiment, the AI ​​agents that automatically modify the process during process design, during a specific execution instance of the process, and after one or more execution instances of the process are different AI agents. In this embodiment, the AI ​​agents may operate in a multi-agent cooperative configuration and be able to learn from each other. In another embodiment, the AI ​​agent that automatically modifies the process during process design, during a specific execution instance of the process, and after one or more execution instances of the process is the same AI agent.

[0198] AI agents are implemented using AI / ML models and make probabilistic decisions independently, autonomously, or semi-autonomously in a non-deterministic manner. In one or more examples, an AI agent may be AI agent 210 in Figure 2, an AI agent in AI agent pool 320 in Figure 3, AI agent 520 in Figure 5, AI agent 750 in Figure 7, or AI agent 850 in Figure 8. In one embodiment, an AI agent is implemented using a pre-trained language model. For example, an AI agent may be implemented using an LLM such as word2vec, BERT, GPT-3, or ChatGPT. However, an AI agent may also be implemented using a small-scale language model with relatively fewer parameters than an LLM. Any other suitable type of language model may also be used to implement an AI agent, e.g., recurrent neural network (RNN) based models, transformer-based models, etc. A language model may learn to automatically modify the process to adjust to user input in the process, for example, by performing fine-tuning, prompt engineering, in-context learning, reinforcement learning, etc., using examples.

[0199] An AI agent is designed, for example, using a designer application. For example, a user may interact with the designer application to define a sequence of activity in a process, provide a natural language description of what the AI ​​agent is intended to do, define contextual information (e.g., via user prompts and / or system prompts), and select tools that the AI ​​agent can use (e.g., APIs, RPA robots, etc.). The AI ​​agent receives process data as input. For example, the AI ​​agent may receive process data as context for contextual grounding via one or more prompts (e.g., via the instruction pane 410, user prompt 420, or additional context 450 in Figure 5). The AI ​​agent may also receive instructions to automatically modify the process to adjust user input to the process, or instructions to provide other contextual information. A process can be modified by adding one or more activities that require user input from the user to the process, removing one or more activities that require user input from the user to the process, or modifying one or more activities that require user input within the process. The AI ​​agent generates the modified process as output. The modified process can be represented as a workflow visualization of the modified process (e.g., a directed graph), but may be represented in any other appropriate form. An exemplary directed graph of the modified process is shown in Figure 18.

[0200] Figure 18 shows an exemplary directed graph 1800 of a modified process for processing and paying invoices, according to one or more embodiments. Directed graph 1800 represents the modified process of the process represented by directed graph 1600 in Figure 16, and is shown and explained with reference to Figure 16. At design time, the AI ​​agent receives directed graph 1600 of Figure 16 as input, which represents a visual representation of the process design. The AI ​​agent generates directed graph 1800 of the modified process design as output, which adds a user approval request activity 1802 to directed graph 1600. Thus, in directed graph 1800, for example, when an invoice is approved in invoice approval activity 1612, if the invoice amount exceeds a predefined threshold amount, directed graph 1800 proceeds to user approval request activity 1802. Once user approval is received, directed graph 1800 proceeds to invoice payment activity 1614.

[0201] For example, one or more activities that request user input include activities for data validation. In this example, activities that request user input for data validation may be removed based on the number of instances of user input that validate the data. For example, an AI agent may compare the proportion of user input instances that validate the data to a threshold and remove the activity that requests user input if the threshold is met.

[0202] In another example, one or more activities that require user input include an activity for invoice approval. In this example, an activity that requires user input for invoice approval may be deleted based on the number of approval instances for a given amount or value of the invoice. For example, an AI agent may compare the percentage of user input instances that approve an invoice for a given amount to a threshold, and if the threshold is met, delete the activity or change the invoice amount for which approval is required in that activity.

[0203] In one embodiment, the process is automatically modified by the AI ​​agent without further user intervention. In another embodiment, the AI ​​agent sends a notification to the user about the change to the process and modifies the process in response to receiving the user's approval of the change.

[0204] In step 1506 of Figure 15, the modified process is output. The modified process can be output, for example, by displaying it to the user on a display device (e.g., display device 1125 of the computing system 1100 in Figure 11), by storing it in the memory or storage device of the computer system (e.g., memory 1115 of the computing system 1100 in Figure 11), or by transmitting it to a remote computer system (e.g., the computing system 1100 in Figure 11).

[0205] The steps disclosed herein, including the steps and substeps of Figures 14-15, may be executed by a computer program encoding instructions for a processor to perform at least a portion of the steps disclosed herein, according to embodiments of the present invention. The computer program may be implemented on a non-temporary computer-readable medium. The computer-readable medium may include, but is not limited to, a hard disk drive, a flash device, RAM, tape, and / or any other such medium or combination of mediums used to store data. The computer program may include encoded instructions for controlling a processor of a computing system (e.g., processor 1110 of computing system 1100 in Figure 11) to implement all or part of the process steps disclosed herein, and these may also be stored on a computer-readable medium.

[0206] The computer program may be implemented in hardware, software, or a hybrid implementation. The computer program may consist of modules configured to communicate with one another in an operable manner and designed to pass information or instructions to a display. The computer program may be configured to run on a general-purpose computer, an ASIC, or any other suitable device.

[0207] It will be readily apparent that the components of the various embodiments of the present invention, as generally described and illustrated in the drawings, can be arranged and designed in a wide variety of different configurations. Therefore, the detailed description of the embodiments of the present invention shown in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention.

[0208] The features, structures, or characteristics of the invention described throughout this specification can be combined in any suitable way in one or more embodiments. For example, any reference throughout this specification to “a particular embodiment,” “some embodiments,” or similar phrases means that the particular features, structures, or characteristics described in relation to that embodiment are included in at least one embodiment of the invention. Therefore, the occurrence of “a particular embodiment,” “some embodiments,” “other embodiments,” or similar phrases throughout this specification does not necessarily refer to the same group of embodiments, and the described features, structures, or characteristics can be combined in any suitable way in one or more embodiments.

[0209] It should be noted that references to features, advantages, or similar terms throughout this specification do not imply that all features and advantages that can be realized by the present invention should or actually be present in any one embodiment of the invention. Rather, terms referring to features and advantages should be understood to mean that a particular feature, advantage, or characteristic described in relation to an embodiment is included in at least one embodiment of the invention. Accordingly, discussions of features, advantages, and similar terms throughout this specification may, but not necessarily, refer to the same embodiment.

[0210] Furthermore, the described features, advantages, and characteristics of the invention can be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the invention can be practiced even if one or more of the specific features or advantages of a particular embodiment are absent. In other cases, additional features and advantages that are not present in all embodiments of the invention may be recognized in a particular embodiment.

[0211] Those skilled in the art will readily understand that the invention discussed above can be implemented in a different sequence of steps and / or with hardware elements of a different configuration than those disclosed. Therefore, although the invention is described based on these preferred embodiments, certain modifications, variations, and alternative configurations will be apparent to those skilled in the art, while remaining within the spirit and scope of the invention. Thus, to define the boundaries of the invention, one should refer to the appended claims.

Claims

1. A computer implementation method, Receiving process data associated with a process, Using an AI (artificial intelligence) agent, the process is automatically modified based on the process data in order to adjust the user input of the process. A computer implementation method that includes outputting the modified process.

2. A computer implementation method according to claim 1, Receiving process data associated with the aforementioned process means This includes receiving process design data associated with the aforementioned process, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A computer implementation method that includes automatically modifying the process based on the process design data during the design of the process.

3. A computer implementation method according to claim 1, Receiving process data associated with the aforementioned process means This includes receiving process runtime data associated with the aforementioned process, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A computer implementation method that includes automatically modifying the process during a specific execution instance of the process based on process runtime data associated with the specific execution instance.

4. A computer implementation method according to claim 1, Receiving process data associated with the aforementioned process means This includes receiving process history execution data associated with the aforementioned process, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A computer implementation method comprising automatically modifying the process based on the process history execution data after one or more execution instances of the process.

5. A computer implementation method according to claim 4, Automatically modifying the process based on the process history execution data after one or more execution instances of the process is: Preprocessing the aforementioned process history execution data, A computer implementation method comprising: automatically modifying the process based on pre-processed process history execution data.

6. A computer implementation method according to claim 1, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A computer implementation method comprising adding one or more activities to the process that require user input from a user.

7. A computer implementation method according to claim 1, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A computer implementation method comprising removing one or more activities that require user input from a user from the process.

8. A computer implementation method according to claim 1, A computer implementation method in which the AI ​​agent receives the process data as context via one or more prompts.

9. A computer implementation method according to claim 1, The AI ​​agent receives commands via one or more prompts, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A computer implementation method comprising further automatically modifying the process using the AI ​​agent based on the aforementioned instructions.

10. It is a system, At least one processor, A memory for storing computer instructions, wherein when the computer instructions are executed by the at least one processor, the system... Receiving process data associated with a process, Using an AI (artificial intelligence) agent, the process is automatically modified based on the process data in order to adjust the user input of the process. A system including the memory and which causes the modified process to output the modified process and to perform an operation including the operation.

11. The system according to claim 10, Receiving process data associated with the aforementioned process means This includes receiving process design data associated with the aforementioned process, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A system that includes automatically modifying the process based on the process design data during the design of the process.

12. The system according to claim 10, Receiving process data associated with the aforementioned process means This includes receiving process runtime data associated with the aforementioned process, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A system that includes automatically modifying the process during a particular execution instance of the process based on process runtime data associated with the particular execution instance.

13. The system according to claim 10, Receiving process data associated with the aforementioned process means This includes receiving process history execution data associated with the aforementioned process, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A system that includes automatically modifying the process based on the process history execution data after one or more execution instances of the process.

14. The system according to claim 13, Automatically modifying the process based on the process history execution data after one or more execution instances of the process is: Preprocessing the aforementioned process history execution data, A system that includes automatically modifying the process based on pre-processed process history execution data.

15. A non-temporary computer-readable medium for storing computer program instructions, When executed on at least one processor, the at least one processor: Receiving process data associated with a process, Using an AI (artificial intelligence) agent, the process is automatically modified based on the process data in order to adjust the user input of the process. A non-temporary computer-readable medium that causes an operation to be performed, including outputting the modified process.

16. A non-temporary computer-readable medium according to claim 15, Receiving process data associated with the aforementioned process means This includes receiving process design data associated with the aforementioned process, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A non-temporary computer-readable medium that includes automatically modifying the process based on the process design data during the design of the process.

17. A non-temporary computer-readable medium according to claim 15, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A non-temporary computer-readable medium that includes adding one or more activities to the process that require user input from the user.

18. A non-temporary computer-readable medium according to claim 15, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A non-temporary computer-readable medium, which includes removing one or more activities from the process that require user input from the user.

19. A non-temporary computer-readable medium according to claim 15, The AI ​​agent receives the process data as context via one or more prompts in a non-temporary, computer-readable medium.

20. A non-temporary computer-readable medium according to claim 15, The AI ​​agent receives commands via one or more prompts, Using an AI (artificial intelligence) agent to adjust user input for the process, automatically modifying the process based on the process data is: A non-temporary computer-readable medium, which includes further automatically modifying the process using the AI ​​agent based on the aforementioned instructions.

Images