Performing Robotic Process Automation Robotic Maintenance Using Cognitive AI Layer

Abstract

Embodiments of the present disclosure relate to using a cognitive artificial intelligence layer to perform robotic process automation robot maintenance. Disclosed is the use of a cognitive artificial intelligence (AI) layer (e.g., a generative AI model and one or more other AI / ML models) to perform robotic process automation (RPA) robot maintenance. The RPA robot can call a computer vision (CV) model and / or an optical character recognition (OCR) model to identify graphic elements and text, respectively, in a user interface (UI) when performing activities in an RPA workflow. However, the CV model or the OCR model may not find the graphic element or text targeted by the activity of the RPA workflow, or the RPA robot may encounter an exception and fail when performing the activity. In this case, a cognitive AI layer including a generative AI model can be used to find the target graphic element or text, or to resolve the robot failure.

Description

Technical Field

[0001] The present invention relates generally to artificial intelligence (AI), and more particularly to using a cognitive AI layer (e.g., a generative AI model and one or more other AI / ML models) to perform robotic process automation (RPA) robot maintenance. Background Art

[0002] Computer vision (CV) models and / or optical character recognition (OCR) models can be used to identify graphical elements and / or text present in a user interface (UI). Such models will typically identify candidate graphical elements and / or text and provide confidence scores for the candidates. However, these techniques may not always accurately identify graphical elements in a UI. Accordingly, improved and / or alternative methods may be beneficial. Summary of the invention

[0003] Certain embodiments of the present invention may provide solutions to problems and needs in the field that have not yet been fully identified, understood, or solved by current AI technology. For example, some embodiments of the present invention involve using a cognitive AI layer (e.g., a generative AI model and one or more other AI / ML models) to perform RPA robot maintenance.

[0004] In one embodiment, one or more non-transitory computer-readable media store a computer program for performing RPA robot maintenance using a cognitive AI layer. The computer program is configured to cause at least one processor to perform automation through an RPA robot, which performs logic associated with an RPA workflow. The computer program is also configured to cause at least one processor to: detect failures in activities of an RPA workflow being executed by the RPA robot, and provide information as input to a cognitive AI layer including a generative AI model, which is trained to provide outputs for automated maintenance. The computer program is also configured to cause at least one processor to: receive outputs from the cognitive AI layer, and perform one or more actions through the RPA robot in response to outputs from the cognitive AI layer to facilitate automated maintenance. An activity is a step in an RPA workflow that is performed by an RPA robot in the order of the RPA workflow or some other logical flow.

[0005] In another embodiment, one or more computing systems include: a memory storing computer program instructions for performing RPA robot maintenance using a cognitive AI layer; and at least one processor configured to execute the computer program instructions. The computer program instructions are configured to cause at least one processor to: detect failures in activities of an RPA workflow being executed by an RPA robot, and provide information as input to a cognitive AI layer including a generative AI model and one or more other models, the one or more other models being configured to understand the intent of the automation associated with the RPA workflow, the execution of the automation so far, and what humans would do and action plans for maintenance when the automation fails before completion. The cognitive AI layer is trained to provide output for automated maintenance. The computer program instructions are also configured to cause at least one processor to: receive output from the cognitive AI layer, and provide output to the RPA robot. An activity is a step in an RPA workflow that is performed by an RPA robot in accordance with or in some other logical flow in the order of the RPA workflow.

[0006] In another embodiment, a computer-implemented method for performing RPA robot maintenance using a cognitive AI layer includes: performing automation by the RPA robot, the automation executing logic associated with the RPA workflow, and detecting failures of activities of the RPA workflow by the RPA robot. The computer-implemented method also includes providing information as input to the cognitive AI layer including a generative AI model by the RPA robot. The cognitive AI layer is trained to provide output for automated maintenance. The computer-implemented method also includes: receiving output from the cognitive AI layer by the RPA robot, and performing one or more actions by the RPA robot in response to the output from the cognitive AI layer to facilitate automated maintenance. Activities are steps in the RPA workflow that are performed by the RPA robot in the order of the RPA workflow or some other logical flow. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order that the advantages of certain embodiments of the present invention will be readily understood, the invention briefly described above will be described more particularly by reference to specific embodiments shown in the accompanying drawings. Although it should be understood that these drawings depict only typical embodiments of the invention and are not to be considered limiting of its scope, the present invention will be described and explained more particularly and in more detail through the use of the accompanying drawings, in which:

[0008] Figure 1 is an architectural diagram showing a hyper-automation system according to an embodiment of the present invention.

[0009] Figure 2 is an architecture diagram showing an RPA system according to an embodiment of the present invention.

[0010] Figure 3 is an architectural diagram showing a deployed RPA system according to an embodiment of the present invention.

[0011] Figure 4 is an architectural diagram illustrating the relationship between designers, activities, and drivers according to an embodiment of the present invention.

[0012] Figure 5 is an architectural diagram illustrating a computing system configured to perform RPA robot maintenance using generative AI according to an embodiment of the present invention.

[0013] Fig. 6A An example of a neural network that has been trained to perform RPA robot maintenance according to an embodiment of the present invention is shown.

[0014] Figure 6B An example of a neuron according to an embodiment of the present invention is shown.

[0015] Figure 7 is a flow chart illustrating a process for training (multiple) AI / ML models according to an embodiment of the present invention.

[0016] Figure 8 An AI / ML model of a cognitive AI layer according to an embodiment of the present invention is shown.

[0017] Fig. 9 is a flow chart illustrating a process for performing RPA robot maintenance using generative AI according to an embodiment of the present invention.

[0018] Unless otherwise indicated, like reference numerals refer to corresponding features throughout the drawings. DETAILED DESCRIPTION

[0019] Some embodiments involve using a cognitive AI layer (e.g., a generative AI model and one or more other AI / ML models) to perform RPA robot maintenance. The RPA robot can call a CV model and / or an OCR model to recognize graphic elements and text in the UI, respectively, when performing activities in an RPA workflow. In some embodiments, the OCR model can be part of the CV model.

[0020] However, the CV model or OCR model may not find the graphic element or text targeted by the activity of the RPA workflow, or the RPA robot may encounter an exception when performing the activity and fail in other ways. In this case, a cognitive AI layer including a generative AI model can be used to find the target graphic element or text to resolve the robot failure, or to find an alternative route to solve the original problem. Consider an example in which a user wants to upload a document in an application. In the initial design of the application, the user needs to click the upload button. However, in the new version of the application, the upload button no longer exists. Instead, there is a drag and drop area. In some embodiments, a cognitive AI layer (e.g., advanced CV, OCR, generative AI, and potential additional models) can determine how to perform operations in a modified interface. In certain embodiments, such a cognitive AI layer can be applied to repair test cases during the testing process.

[0021] The cognitive AI layer model can provide formatted data from the UI processing to the generative AI model. However, the data provided to the generative AI model may additionally or alternatively include: (1) the original automation blueprint (code), which represents the intention of automation (e.g., process definition documents, automation code, design and debugging time snapshots, etc.); (2) the current execution log, which represents the trajectory of the execution so far in text form; (3) the current execution record, which intuitively displays the execution so far; (4) the cognitive AI layer model's interpretation of the code, logs, and records (e.g., as text output, JavaScript Object Notation (JSON), Extensible markup Language (XML), etc.); (5) RPA automation language (such as Automation language), including activities and building blocks (e.g., activities and building blocks in JSON / XML format); (6) screen ontology (such as (a) a screen ontology that represents how screen elements and screens are constructed and interact with each other in text, programming language code, JSON, XML, etc. (e.g., from an object repository); (7) a current screen technology representation, such as a Document Object Model (DOM) of a web page; (8) textual boundaries or rules that prevent the RPA robot from performing certain actions, accessing certain information, etc.; or (9) any combination thereof. All of the above represent input data for the generative AI model and for other AI / AI models of the potential cognitive AI layer. In some embodiments, the data can be in hypertext markup language (HTML), XML, Extensible Application Markup Language (XAML), or another structured format.

[0022] Generative AI models and / or other models of the cognitive AI layer can process the data and provide indications of where unrecognized graphical elements are located, apply semantic understanding to match the (multiple) elements targeted by the activity and overcome anomalies, etc. For example, consider a situation where an RPA workflow is developed for the current version of the target application, but a new version of the application is released after the production automation is created for the RPA workflow, which moves graphical elements to different locations in the UI or different application screens, changes the text labels of fields in the application to different but semantically similar text, changes the appearance of buttons that still perform the same function intended by the activity, etc. The RPA robot can call the generative AI model and / or other cognitive AI layer models to find elements that have not been found within a certain period of time, find elements that no longer match their original versions, etc. The RPA robot can then use this information to perform the activity. In some embodiments, the RPA robot or another software process can automatically update the RPA workflow and generate a new version of the automation that will correct the problem in the future.

[0023] To perform repairs, the cognitive AI layer should understand: (1) the intent of the RPA process; (2) how the RPA process has executed so far; and (3) what a human would do when the RPA process fails before completion and what the best course of action is in terms of repairs. However, in some embodiments, generative AI and CV alone may not be sufficient to create such a cognitive AI layer. To address this issue, some embodiments employ one or more additional AI / ML models in the cognitive AI layer to facilitate this understanding. In some embodiments, recommendations can be provided on what to change for future versions of the automation to avoid experiencing the same problem. To recover during live (runtime) execution, code blocks can be automatically generated, and / or automation can be created that understands the intent and cause of the failure. In other words, in some embodiments, the RPA robot can attempt to solve the problem on its own instead of providing recommendations.

[0024] In some embodiments, RPA workflow execution is paused until a human user provides manual input. This does not necessarily mean that an error or failure has occurred in the automated execution. Instead, the RPA robot requires human confirmation before making a potential repair.

[0025] In some embodiments, the RPA robot can roll back previously taken steps to eliminate the failure. For example, the RPA robot can: (1) suggest code changes to bypass the failure but keep the underlying logic of the automation; (2) provide application change suggestions to bypass the problem but keep the underlying business logic; (3) provide business rule suggestions to improve the execution of the workflow in terms of performance and / or return on investment (ROI), which business rule suggestions can essentially be process improvement suggestions based on execution data; (4) provide test cases that should be added to detect failure situations faster in the future; or (5) any combination thereof.

[0026] In some embodiments, a certain minimum confidence score may be required to accept a repair recommendation from the cognitive AI layer (e.g., 95%, 99%, etc.). Alternatively, a recommendation from the cognitive AI layer model may be tried regardless of the confidence score and attempted to see if it can repair the activity. In some embodiments, different types of generative AI models and / or other AI / ML models may be used simultaneously or consecutively to attempt repairs.

[0027] In some embodiments, an "AI Task" may be performed. As used herein, an AI Task is a "no-code" block in an RPA workflow where an RPA robot is configured to use a cognitive AI layer to solve a task without a specific set of design-time workflow steps. The cognitive AI layer may be used to detect the best course of action at runtime based on context, the application being used, execution data, LLM, etc.

[0028] In some embodiments, the concept of "decision making" can be used to perform end-to-end business process automation. Consider a situation where there is an action-centric task. Such a task can be assigned to a human for manual input. However, the task may expire before the human provides input. The RPA robot can utilize the cognitive AI layer to associate past decisions with the task and make a decision for the human after the time limit for human input expires.

[0029] Figure 1 is an architectural diagram showing a hyper-automation system 100 according to an embodiment of the present invention. "Hyper-automation" as used herein refers to an automation system that combines components of process automation, integrated tools, and technologies that enhance the ability to automate work. For example, in some embodiments, robotic process automation (RPA) can be used at the core of a hyper-automation system, and in certain embodiments, automation capabilities can be extended using AI / machine learning (ML), process mining, analysis, and / or other advanced tools. For example, as the hyper-automation system learns processes, trains AI / ML models, and employs analysis, more and more knowledge work can be automated, and computing systems in an organization (e.g., computing systems used by individuals and computing systems that execute autonomously) can all participate in the hyper-automation process. The hyper-automation system of some embodiments allows users and organizations to efficiently and effectively discover, understand, and extend automation.

[0030] The hyper-automated system 100 includes user computing systems such as a desktop computer 102, a tablet computer 104, and a smart phone 106. However, any desired user computing system may be used without departing from the scope of the present invention, including but not limited to smart watches, laptop computers, servers, Internet-of-Things (IoT) devices, etc. Figure 1 Three user computing systems are shown in FIG. 1 , but any suitable number of user computing systems may be used without departing from the scope of the invention. For example, in some embodiments, dozens, hundreds, thousands, or millions of user computing systems may be used. The user computing systems may be actively used by the user, or may be automatically executed without much or any user input.

[0031] Each user computing system 102, 104, 106 has a corresponding (multiple) automation process 110, 112, 114 executed thereon. In some embodiments, the automation process is stored remotely (e.g., stored on a server 130 or in a database 140 and accessed via a network 120) and loaded by an RPA robot to implement automation. The automation may exist as a script (e.g., XML, XAML, etc.), or compiled into machine-readable code (e.g., as a digital link library).

[0032] The automated processes 110, 112, 114 may include, but are not limited to, an RPA robot, a portion of an operating system, a downloadable application for a corresponding computing system, any other suitable software and / or hardware, or any combination of these, without departing from the scope of the present invention. In some embodiments, one or more processes 110, 112, 114 may be a listener. The listener may be an RPA robot, a portion of an operating system, a downloadable application for a corresponding computing system, or any other software and / or hardware, without departing from the scope of the present invention. In fact, in some embodiments, the logic of the listener(s) is partially or completely implemented via physical hardware.

[0033] The listener monitors and records data about the user's interaction with the corresponding computing system and / or the operation of the unattended computing system, and sends the data to the core hyper-automation system 120 via a network (e.g., a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, and any combination thereof, etc.). The data may include, but is not limited to, which buttons were clicked, where the mouse was moved, the text entered in the field, one window was minimized and another window was opened, the application associated with the window, etc. In some embodiments, the data from the listener may be sent periodically as part of a heartbeat message. In some embodiments, the data may be sent to the core hyper-automation system 120 once a predetermined amount of data has been collected, after a predetermined time period has passed, or in both cases. One or more servers (such as server 130) receive the data from the listener and store it in a database (such as database 140).

[0034] The automation process can execute the logic developed during design time in the workflow. In the case of RPA, a workflow can include a set of steps defined herein as "activities" that are executed in a sequential or other logical flow. Each activity can include an action, such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows can be nested or embedded.

[0035] In some embodiments, the long-running workflow of RPA is the main project that supports service orchestration, manual intervention, and long-running transactions in an unattended environment. For example, see U.S. Patent No. 10,860,905, which is incorporated herein by reference in its entirety. Manual intervention comes into play when certain processes require manual input to handle exceptions, approvals, or verifications before proceeding to the next step of the activity. In this case, the process execution is suspended, freeing up the RPA robot until the manual task is completed.

[0036] Long-running workflows can support workflow fragmentation via persistent activities, and can be combined with calling processes and non-user interactive activities to orchestrate manual tasks with RPA robot tasks. In some embodiments, multiple or many computing systems may participate in the execution of the logic of long-running workflows. Long-running workflows can be executed in sessions to facilitate high-speed execution. In some embodiments, long-running workflows can orchestrate background processes, which may include activities that execute application programming interface (API) calls and are executed in long-running workflow sessions. In some embodiments, these activities can be called by calling process activities. Processes with user interactive activities executed in user sessions can be called by starting jobs from commander activities (the commander will be described in detail later). In some embodiments, users can interact by requiring tasks to complete forms in the commander. It may include activities that cause the RPA robot to wait for the form task to be completed and then resume the long-running workflow.

[0037] One or more of the automated processes 110, 112, 114 communicate with the core hyper-automation system 120. In some embodiments, the core hyper-automation system 120 may execute a command application on one or more servers, such as server 130. Although one server 130 is shown for illustrative purposes, multiple or many servers close to each other or in a distributed architecture may be employed without departing from the scope of the present invention. For example, one or more servers may be provided for commander functions, AI / ML model services, authentication, governance, and / or any other suitable functions without departing from the scope of the present invention. In some embodiments, the core hyper-automation system 120 may be incorporated into a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, or the like, or may be part of a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, or the like. In some embodiments, the core hyper-automation system 120 may host multiple software-based servers on one or more computing systems, such as server 130. In some embodiments, one or more servers, such as server 130, of the core hyper-automation system 120 may be implemented via one or more virtual machines (VMs).

[0038] In some embodiments, one or more of the automated processes 110, 112, 114 may call one or more AI / ML models 132 deployed on or accessible by the core hyper-automation system 120 and trained to perform various tasks. For example, the AI / ML models 132 may include models trained to find various application versions, perform CV, perform OCR, generate user interface (UI) descriptors, provide suggestions for the next activity or sequence of activities in the RPA workflow, etc. The AI / ML models may be trained using labeled data, including but not limited to elements from data sources (e.g., web pages, forms, scanned documents, application interfaces, screens, etc.), previously created RPA workflows, screenshots of various application screens of various versions and their corresponding UI elements, UI object libraries, etc. The AI / ML models 132 may be trained to achieve a desired confidence threshold while not overfitting to a given training data set.

[0039] The AI / ML model 132 may be trained for any suitable purpose without departing from the scope of the present invention, which will be discussed in more detail below. In some embodiments, two or more AI / ML models 132 may be connected (e.g., in series, in parallel, or a combination thereof) so that they jointly provide (multiple) collaborative outputs. The AI / ML model 132 may perform or assist in CV, OCR, document processing and / or understanding, semantic learning and / or analysis, analytical prediction, process discovery, task mining, testing, automatic RPA workflow generation, sequence extraction, cluster detection, audio to text translation, any combination thereof, etc. However, any desired number and / or type of AI / ML model may be used without departing from the scope of the present invention. For example, using multiple AI / ML models may allow the system to develop a global picture of what is happening on a given computing system. For example, one AI / ML model may perform OCR, another AI / ML model may detect buttons, another AI / ML model may compare sequences, and so on. The style may be determined by the AI / ML model alone, or by multiple AI / ML models together. In certain embodiments, one or more AI / ML models are deployed locally on at least one of computing systems 102 , 104 , 106 .

[0040] In some embodiments, multiple AI / ML models 132 may be used. Each AI / ML model 132 is an algorithm (or model) executed on data, and the AI / ML model itself may be a deep learning neural network (DLNN) of trained artificial "neurons" trained on, for example, training data. In some embodiments, the AI / ML model 132 may have multiple layers that perform various functions, such as statistical modeling (e.g., hidden Markov models (HMMs)), and utilize deep learning techniques (e.g., long short term memory (LSTM) deep learning, encoding of previous hidden states, etc.) to perform the desired function.

[0041] In some embodiments, the hyperautomation system 100 can provide four main groups of functions: (1) discovery; (2) building automation; (3) management; and (4) engagement. In some embodiments, automation (e.g., executed on a user computing system, server, etc.) can be performed by a software robot (such as an RPA robot). For example, attended robots, unattended robots, and / or testing robots can be used. Attended robots work with users to assist them in completing tasks (e.g., via UiPath Assistant). TM). Unattended robots work independently of users and can execute in the background without the user's knowledge. Test robots are unattended robots that execute test cases against applications or RPA workflows. In some embodiments, test robots can execute in parallel on multiple computing systems.

[0042] The discovery function can discover different opportunities for business process automation and provide automatic recommendations for them. Such functions can be implemented by one or more servers (such as server 130). In some embodiments, the discovery function can include providing an automation hub, process mining, task mining, and / or task capture. The automation hub (e.g., UiPath Automation Hub) TM ) can provide a mechanism for managing automation rollouts with visibility and control. For example, automation ideas can be solicited from employees via a submission form. Feasibility and ROI calculations for automating these ideas can be provided, documentation for future automation can be collected, and collaboration can be provided to move faster from automation discovery to full development.

[0043] Process mining (e.g., via UiPath Automation Cloud TM and / or UiPath AI Hub TM ) refers to the collection and analysis of data from applications (e.g., enterprise resource planning (ERP) applications, customer relationship management (CRM) applications, email applications, call center applications, etc.) to identify what end-to-end processes exist in the organization and how to effectively automate them, as well as indicate what the impact of automation will be. For example, the data can be collected by a listener from user computing systems 102, 104, 106 and processed by a server (such as server 130). In some embodiments, one or more AI / ML models 132 can be used for this purpose. This information can be exported to an automation hub to accelerate implementation and avoid manual information migration. The goal of process mining may be to increase business value by automating processes within an organization. Some examples of process mining goals include, but are not limited to, increasing profits, improving customer satisfaction, regulatory and / or contractual compliance, improving employee efficiency, and the like.

[0044] Task mining (e.g., via UiPath Automation Cloud TM and / or UiPath AI Hub TM) identifies and aggregates workflows (e.g., employee workflows), and then applies AI to reveal patterns and variations in daily tasks, thereby scoring such tasks for automation and potential savings (e.g., time and / or cost savings). One or more AI / ML models 132 may be employed to reveal recurring task patterns in the data. Repetitive tasks suitable for automation may then be identified. In some embodiments, this information may initially be provided by a listener and analyzed on a server (e.g., server 130) of the core hyperautomation system 120. Results from task mining (e.g., XAML process data) may be exported to a process document or designer application (UiPath Studio TM ) to more quickly create and deploy automation. In some embodiments, task mining can include taking screenshots with user actions (e.g., mouse click locations, keyboard input, application windows and graphical elements that the user is interacting with, timestamps of interactions, etc.), collecting statistics (e.g., execution time, number of actions, text entries, etc.), editing and annotating screenshots, specifying action types to be recorded, etc.

[0045] Task capture (e.g., via UiPath Automation Cloud TM and / or UiPath AI Hub TM ) automatically records attended processes as users work, or provides a framework for unattended processes. Such documents can include process definition documents (PDDs), skeleton workflows (which capture the actions for each part of the process, record user actions, and automatically generate a comprehensive workflow diagram that includes details about each step), Microsoft In some embodiments, a build-ready workflow can be exported directly to a designer application, such as UiPath Studio. TM Task capture can streamline the requirements gathering process, whether for subject matter experts explaining the process or for Center of Excellence (CoE) members providing production-grade automation.

[0046] Building automation can be done via a designer application (e.g., UiPath Studio TM 、UIPath Studio X TM or UiPath Web TM For example, an RPA developer at the RPA development facility 150 may use an RPA designer application 154 of a computing system 152 to build and test RPA applications for various applications and environments (such as web, mobile, and virtualized desktops). API integration can be provided for a variety of applications, technologies, and platforms. Predefined activities, drag-and-drop modeling, and workflow recorders can make automation easier with minimal decoding. Document understanding capabilities can be provided via drag-and-drop AI skills for data extraction and interpretation that call one or more AI / ML models 132. Such automation can handle virtually any document type and format, including tables, checkboxes, signatures, and handwriting. When validating data or handling exceptions, the information can be used to retrain the corresponding AI / ML model, thereby improving its accuracy over time.

[0047] The RPA designer application 152 can be designed to call one or more trained AI / ML models 132 on the server 130 and / or a generative AI model 172 in a cloud environment via a network 120 (e.g., a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, any combination thereof, etc.) to assist in the RPA automation development process. In some embodiments, one or more AI / ML models can be packaged with the RPA designer application 152 or otherwise stored locally on the computing system 150.

[0048] In some embodiments, the RPA designer application 152 and the one or more AI / ML models 132 can be configured to use an object repository stored in the database 140. The object repository can include a UI object library, and the UI objects can be used to develop RPA workflows via the RPA designer application 152. The object repository can be used to add UI descriptors to activities in the workflow of the RPA designer application 152 for UI automation. In some embodiments, the one or more AI / ML models 132 can generate new UI descriptors and add them to the object repository in the database 140. Once the automation is completed in the designer application 152, they can be published on the server 130 and pushed out to the computing systems 102, 104, 106, etc.

[0049] For example, integration services can allow developers to seamlessly combine UI automation with API automation. Automation can be built for applications and systems that require APIs or traverse APIs and non-APIs. A repository of templates and solutions for pre-built RPA and AI can be provided (e.g., UiPath Object Repository). TM ) or a marketplace (e.g., UiPath Marketplace TM) to allow developers to automate a variety of processes more quickly. Therefore, when building automation, the hyper-automation system 100 can provide a user interface, development environment, API integration, pre-built and / or custom-built AI / ML models, development templates, integrated development environment (IDE), and advanced AI capabilities. In some embodiments, the hyper-automation system 100 enables the development, deployment, management, configuration, monitoring, debugging, and maintenance of RPA robots, which can provide automation for the hyper-automation system 100.

[0050] In some embodiments, components of the hyperautomation system 100, such as (multiple) designer applications and / or external rule engines, provide support for managing and enforcing governance policies for controlling various functions provided by the hyperautomation system 100. Governance is the ability of an organization to establish policies to prevent users from developing automations (e.g., RPA robots) that can take actions that may harm the organization (such as violating the EU General Data Protection Regulation (GDPR), the US Health Insurance Portability and Accountability Act (HIPAA), third-party application terms of service, etc.). Because developers can create automations that violate privacy laws, terms of service, etc. while executing their automations, some embodiments implement access control and governance restrictions at the robot and / or robot designer application level. In some embodiments, this can provide a higher level of security and compliance for the automation process development pipeline by preventing developers from relying on unapproved software libraries that may introduce security risks or work in a manner that violates policies, regulations, privacy laws, and / or privacy policies. For example, see U.S. Patent Application Publication No. 2022 / 0011732, which is incorporated herein by reference in its entirety.

[0051] The management functions may provide automated management, deployment, and optimization across organizations. In some embodiments, the management functions may include orchestration, test management, AI functions, and / or insights. The management functions of the hyper-automation system 100 may also serve as an integration point with third-party solutions and applications for automated applications and / or RPA robots. The management capabilities of the hyper-automation system 100 may include, but are not limited to, facilitating the provisioning, deployment, configuration, queuing, monitoring, logging, and interconnection of RPA robots.

[0052] Commander applications (such as UiPath Orchestrator TM (which in some embodiments can be used as UiPath Automation CloudTM or on-premises, in a VM, in a private or public cloud, or on Linux TM Available in VM, or via UiPath Automation Suite TM Delivered as a cloud-native single-container suite)) provides orchestration capabilities to deploy, monitor, optimize, scale, and secure RPA robot deployments. Test suites (e.g., UiPath Test Suite TM ) can provide test management to monitor the quality of the deployed automation. Test suites can facilitate test planning and execution, meeting requirements, and defect traceability. Test suites can include comprehensive test reports.

[0053] Analytics software (e.g., UiPath Insight TM ) can track, measure, and manage the performance of deployed automation. Analytics software can align automation operations with specific key performance indicators (KPIs) and strategic outcomes for the organization. Analytics software can present results in a dashboard format for better understanding by human users.

[0054] For example, data services (e.g., UiPath Data Services TM ) can be stored in the database 140 and brought to a single, scalable, secure location using a drag-and-drop storage interface. Some embodiments can provide low-code or no-code data modeling and storage for automation while ensuring seamless access to data, enterprise-grade security, and scalability. AI capabilities can be powered by an AI hub (e.g., UiPath AI Hub TM) is provided, which facilitates the incorporation of AI / AI models into automation. Pre-built AI / ML models, model templates, and various deployment options can make such functionality accessible even to those who are not data scientists. Deployed automation (e.g., RPA robots) can call AI / ML models, such as AI / ML model 132, from the AI ​​center. The performance of the AI / ML model can be monitored and trained and improved using manually verified data (such as data provided by the data review center 160). The reviewer can provide labeled data to the core hyper-automation system 120 via the review application 152 on the computing system 154. For example, the reviewer can verify that the predictions made by the AI / ML model 132 and / or the generative AI model 172 are accurate or otherwise provide corrections. The dynamic input can then be saved as training data for retraining the AI / ML model 132 and / or the generative AI model 172, and can be stored in a database (such as database 140). The AI ​​center can then schedule and execute training jobs to train a new version of the AI / ML model using the training data. Both positive and negative samples may be stored and used to retrain the AI / ML model 132 and / or the generative AI model 172 .

[0055] The engagement function brings humans and automation together as a team to collaborate seamlessly on the required processes. Low-code applications can be built (e.g., via UiPath App TM ) to connect browser tabs and traditional software, even if the API is missing in some embodiments. For example, applications can be quickly created using a web browser through a rich library of drag-and-drop controls. Applications can be connected to a single automation or multiple automations.

[0056] Action Center (for example, UiPath Action Center TM ) provides a straightforward and efficient mechanism to hand over processes from automation to humans and vice versa. Humans can provide approvals or escalations, handle exceptions, etc. Automation can then perform the automation functions for a given workflow.

[0057] A local assistant can be provided as a launchpad for users to start automation (e.g., UiPath Assistant TM). For example, this functionality may be provided in a tray provided by the operating system and may allow users to interact with RPA robots and RPA robot-driven applications on their computing systems. The interface may list automations approved by a given user and allow the user to execute them. These may include off-the-shelf automations from an automation marketplace, internal automation shops in an automation hub, and the like. When automations execute, they may execute as local instances in parallel with other processes on the computing system, so the user can use the computing system while the automation performs its actions. In some embodiments, the assistant is integrated with a task capture function so that users can record their upcoming automated processes from the assistant launchpad.

[0058] Chatbots (e.g., UiPath Chatbot TM , social messaging apps, and / or voice commands) can enable users to perform automation. This can simplify access to the information, tools, and resources that users need to interact with customers or perform other activities. Conversations between people may be as easily automated as other processes. Triggered RPA robots in this way can potentially use plain language commands to perform actions such as checking order status, posting data in a CRM, etc.

[0059] In some embodiments, the hyperautomation system 100 can provide end-to-end measurement and governance of automation programs of any size. As described above, analytics can be employed to understand the performance of automation (e.g., via UiPath Insights TM ). Data modeling and analysis using any combination of available business metrics and operational insights can be used for a variety of automation processes. Custom designed and pre-built dashboards allow for visualization of data across desired metrics, discovery of new analytical insights, tracking of performance metrics, discovery of ROI for automation, performance monitoring on user computing systems, detection of errors and anomalies, and debugging of automation. An automation management console (e.g., UiPath Automation Operations) can be provided TM ) to manage automations throughout the automation lifecycle. Organizations can govern how automations are built, what users can do with them, and which automations users can access.

[0060] In some embodiments, the hyperautomation system 100 provides an iterative platform. Processes can be discovered, automation can be built, tested, and deployed, performance can be measured, automation usage can be easily provided to users, feedback can be obtained, AI / ML models can be trained and retrained, and the process itself can be repeated. This promotes a more robust and effective automation suite.

[0061] In some embodiments, a generative AI model is used. Generative AI can generate various types of content, such as text, images, audio, and synthetic data. Various types of generative AI models can be used, including but not limited to large language models (LLM), generative adversarial networks (GAN), variational autoencoders (VAE), transformers, etc. These models can be part of the AI / ML model 132 hosted on the server 130. For example, a generative AI model can be trained on a large corpus of text information to perform semantic understanding to understand the nature of the content presented on the screen from the text, etc. In some embodiments, a generative AI model provided by an existing cloud ML service provider (such as In the generative AI embodiments where the generative AI model(s) 172 are remotely hosted, the server 130 can be configured to integrate with a third-party API, which allows the server 130 to send a request including the necessary input information to the generative AI model(s) 172 and receive a response back (e.g., semantic matching of fields between application versions, classification of application types on the screen, etc.). These embodiments can provide a more advanced and sophisticated user experience, as well as access to the most advanced natural language processing (NLP) and other ML capabilities provided by these companies.

[0062] In some embodiments, one aspect of generative AI models is the use of transfer learning. In transfer learning, a pre-trained generative AI model (such as an LLM) is fine-tuned on a specific task or domain. This allows the LLM to leverage knowledge already learned during its initial training and apply that knowledge to a specific application. In the case of an LLM, the pre-training phase involves training the LLM on a large text corpus, typically consisting of billions of words. In this phase, the LLM learns the relationships between words and phrases, which enables the LLM to generate coherent and human-like responses to text-based inputs. The output of this pre-training phase is an LLM that has a high level of understanding of the underlying patterns in natural language.

[0063] In the fine-tuning phase, the pre-trained LLM is adapted to a specific task or domain by training the LLM on a smaller dataset specific to the task. For example, in some embodiments, the LLM can be trained to analyze a certain type or multiple types of data sources to improve the accuracy of its content about those data sources. Such information can be provided as part of the training data, and the LLM can learn to focus on these areas and more accurately identify the data elements therein. Fine-tuning allows the LLM to learn the nuances of the task or domain (such as the specific vocabulary and grammar used in the field) without requiring as much data as would be necessary to train the LLM from scratch. By leveraging the knowledge learned in the pre-training phase, the fine-tuned LLM can achieve state-of-the-art performance on a specific task using a relatively small amount of training data.

[0064] Figure 2 2 is a diagram showing an architecture of an RPA system 200 according to an embodiment of the present invention. Figure 1 The RPA system 200 includes a designer 210 that allows developers to design and implement workflows. The designer 210 can provide solutions for application integration and automation of third-party applications, administrative information technology (IT) tasks, and business IT processes. The designer 210 can facilitate the development of automation projects, which are graphical representations of business processes. In short, the designer 210 facilitates the development and deployment of workflows and robots. In some embodiments, the designer 210 can be an application executed on a user's desktop, an application executed remotely in a VM, a web application, etc.

[0065] Automation projects enable automation of rule-based processes by giving developers control over the order of execution and the relationships between a set of custom steps developed in a workflow (defined herein as "activities" as described above). A commercial example of an embodiment of the designer 210 is UiPath Studio TM Each activity may include an action, such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0066] Some types of workflows may include, but are not limited to, sequences, flow charts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suitable for enabling a linear flow from one activity to another without cluttering the workflow. Flow charts may be particularly suitable for implementing more complex business logic for decision integration and activity connection in a more diverse manner through multiple branching logic operators. FSMs may be particularly suitable for large workflows. FSMs may use a limited number of states triggered by conditions (i.e., transitions) or activities in their execution. Global exception handlers may be particularly suitable for determining workflow behavior and debugging processes when execution errors are encountered.

[0067] Once a workflow is developed in the designer 210, the execution of the business process is orchestrated by the director 220, which orchestrates one or more robots 230 that execute the workflow developed in the designer 210. One commercial example of an embodiment of the director 220 is the UiPath Orchestrator. TM The director 220 facilitates the management of the creation, monitoring, and deployment of resources in the environment. The director 220 may serve as an integration point with third-party solutions and applications. As described above, in some embodiments, the director 220 may be Figure 1 Part of the core hyper-automation system 120.

[0068] The commander 220 can manage a team of robots 230 that connect and execute robots 230 from a central point. The types of robots 230 that can be managed include, but are not limited to, attended robots 232, unattended robots 234, development robots (similar to unattended robots 234, but for development and testing purposes), and non-production robots (similar to attended robots 232, but for development and testing purposes). The attended robots 232 are triggered by user events and operate with humans on the same computing system. The attended robots 232 can be used with the commander 220 for centralized process deployment and recording media. The attended robots 232 can help human users complete various tasks and can be triggered by user events. In some embodiments, processes cannot be started from the commander 220 on this type of robot, and / or they cannot be executed under a locked screen. In some embodiments, the attended robots 232 can only be started from the robot tray or from a command prompt. In some embodiments, the attended robots 232 should be executed under human supervision.

[0069] Unattended robots 234 execute unattended in a virtual environment and can automate many processes. Unattended robots 234 can be responsible for remote execution, monitoring, scheduling, and providing support for work queues. In some embodiments, debugging for all robot types can be performed in the designer 210. Both attended and unattended robots can automate various systems and applications, including but not limited to mainframes, web applications, virtual machines, enterprise applications (e.g., etc.) and computing system applications (e.g., desktop and notebook applications, mobile device applications, wearable computer applications, etc.).

[0070] The commander 220 may have various capabilities, including, but not limited to, provisioning, deploying, configuring, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between the robots 230 and the commander 220 (e.g., a network application). Deployment may include ensuring that grouped versions are properly delivered to assigned robots 230 for execution. Configuration may include maintenance and delivery of robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs into a database (e.g., a structured query language (SQL) database or a "non-relational" SQL (NoSQL) database) and / or another storage mechanism (e.g., It provides the ability to store and quickly query large data sets.) Director 220 can provide interconnectivity by acting as a central point of communication for third-party solutions and / or applications.

[0071] Robot 230 is an execution agent that implements the workflow built into designer 210. One commercial example of some embodiments of robot(s) 230 is UiPath Robot TM In some embodiments, the robot 230 installs Microsoft Services managed by the Service Control Manager (SCM). As a result, such a robot 230 can open interactive session, and has Permissions for services.

[0072] In some embodiments, robots 230 can be installed in user mode. For such robots 230, this means that they have the same permissions as the user who has installed the given robot 230. This feature can also be available for High Density (HD) robots, which can ensure that each machine is fully utilized at its maximum potential. In some embodiments, any type of robot 230 can be configured in an HD environment.

[0073] In some embodiments, the robot 230 is split into several components, each dedicated to a specific automation task. In some embodiments, the robot components include but are not limited to SCM-managed robot services, user-mode robot services, executors, agents, and command lines. SCM-managed robot service management and monitoring The SCM starts the console application under the local system.

[0074] In some embodiments, user-mode robot services manage and monitor The SCM manages the session and acts as a proxy between the commander 220 and the execution host. The user mode robot service can be trusted and manages the credentials for the robot 230. If the SCM manages the robot service, then Applications can be started automatically.

[0075] The actuator can be Executors can be aware of the dots per inch (DPI) settings of each monitor. Agents can be the ones that display available jobs in a system tray window. Display Basics( Presentation Foundation, WPF) applications. Agents can be clients of services. Agents can request to start or stop jobs and change settings. Command lines are clients of services. Command lines are console applications that can request to start jobs and wait for their output.

[0076] Splitting the components of the robot 230 as described above helps developers, supports users, and the computing system to more easily execute, identify, and track what each component is executing. Special behaviors can be configured for each component in this way, such as establishing different firewall rules for actuators and services. In some embodiments, the actuator may always know the DPI setting of each monitor. As a result, workflows can be executed at any DPI, regardless of the configuration of the computing system on which they are created. In some embodiments, projects from the designer 210 can also be independent of the browser extension level. In some embodiments, DPI can be disabled for applications that are not DPI-unaware or are deliberately marked as not perceived.

[0077] In this embodiment, the RPA system 200 is part of a hyper-automation system. Developers can use the designer 210 to build and test RPA robots that utilize AI / ML models deployed in the core hyper-automation system 240 (e.g., as part of its AI center). Such RPA robots can send inputs for executing (multiple) AI / ML models via the core hyper-automation system 240 and receive outputs from them.

[0078] As described above, one or more robots 230 may be listeners. These listeners may provide information to the core hyper-automation system 240 about what users are doing when they use their computing systems. This information may then be used by the core hyper-automation system for process mining, task mining, task capture, etc.

[0079] An assistant / chatbot 250 may be provided on the user's computing system to allow the user to launch an RPA local robot. For example, the assistant may be located in the system tray. The chatbot may have a user interface so that the user can see the text in the chatbot. Alternatively, the chatbot may lack a user interface and execute in the background, using the computing system's microphone to listen to the user's voice.

[0080] In some embodiments, data labeling can be performed by a user of the computing system on which the robot is executing, or on another computing system to which the robot provides information. For example, if the robot calls an AI / ML model that performs CV on an image of a VM user, but the AI / ML model does not correctly identify a button on the screen, the user can draw a rectangle around the misidentified or unidentified component and potentially provide text with the correct identification. This information can be provided to the core hyperautomation system 240 and then used later to train a new version of the AI / ML model.

[0081] Figure 3is an architectural diagram showing an RPA system 300 deployed according to an embodiment of the present invention. In some embodiments, the RPA system 300 may be Figure 2 RPA system 200 and / or Figure 1 The deployed RPA system 300 may be a cloud-based system, a local deployment system, a desktop-based system, and the like, which provides enterprise-level, user-level, or device-level automation solutions for automation of different computing processes.

[0082] It should be noted that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 310 includes an executor 312, an agent 314, and a designer 316. However, in some embodiments, the designer 316 may not be executed on the same computing system as the executor 312 and the agent 314. The executor 312 is executing a process. Several business projects may be executed simultaneously, such as Figure 3 In this embodiment, the agent 314 (e.g., Service) is a single point of contact for all executors 312. In this embodiment, all messages are logged to the director 340, which further processes them via the database server 350, AI / ML server 360, indexer server 370, or any combination thereof. Figure 2 As discussed, the actuator 312 may be a robotic component.

[0083] In some embodiments, a robot represents an association between a machine name and a user name. A robot can manage multiple executors simultaneously. In a computing system that supports multiple interactive sessions executing simultaneously (e.g., Server 2012), multiple robots can be executed simultaneously, each with a unique user name in a separate This is called the HD robot mentioned above.

[0084] The agent 314 is also responsible for sending the status of the robot (e.g., periodically sending a "heartbeat" message indicating that the robot is still executing) and downloading the required version of the package to be executed. In some embodiments, communication between the agent 314 and the commander 340 is always initiated by the agent 314. In a notification scenario, the agent 314 can open a WebSocket channel that is later used by the commander 340 to send commands to the robot (e.g., start, stop, etc.).

[0085] Listener 330 monitors and records data about user interactions with an attended computing system and / or the operation of an unattended computing system in which listener 330 resides. Listener 330 may be an RPA robot, part of an operating system, a downloadable application for a corresponding computing system, or any other software and / or hardware without departing from the scope of the present invention. In fact, in some embodiments, the logic of the listener is partially or completely implemented via physical hardware.

[0086] On the server side, it includes a presentation layer (web application 342, Open Data Protocol (OData) Representative State Transfer (REST) ​​application programming interface (API) endpoint 344, and notification and monitoring 346), a service layer (API implementation / business logic 348), and a persistence layer (database server 350, AI / ML server 360, and indexer server 370). The commander 340 includes a web application 342, an OData REST API endpoint 344, notification and monitoring 346, and API implementation / business logic 348. In some embodiments, most actions performed by a user in the interface of the commander 340 (e.g., via a browser 320) are performed by calling various APIs. These actions may include, but are not limited to, starting a job on a robot, adding / removing data from a queue, scheduling a job to be executed unattended, etc., without departing from the scope of the present invention. The web application 342 is the visual layer of the server platform. In this embodiment, the web application 342 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the present invention. In this embodiment, a user interacts with a web page from a web application 342 via a browser 320 to perform various actions to control the commander 340. For example, a user may create robot groups, assign packages to robots, analyze logs for each robot and / or each process, start and stop robots, and the like.

[0087] In addition to the web application 342, the director 340 also includes a service layer that exposes an OData REST API endpoint 344. However, other endpoints may be included without departing from the scope of the present invention. The REST API is consumed by both the web application 342 and the agent 314. In this embodiment, the agent 314 is a supervisor of one or more robots on a client computer.

[0088] In this embodiment, the REST API covers configuration, logging, monitoring, and queuing functionality. In some embodiments, the configuration endpoint can be used to define and configure application users, permissions, robots, assets, versions, and environments. The logging REST endpoint can be used to log different information, such as errors, explicit messages sent by the robot, and other environment-specific information. The deployment REST endpoint can be used by the robot to query the grouped version that should be executed when the start job command is used in the commander 340. The queueing REST endpoint can be responsible for queue and queue item management, such as adding data to the queue, getting transactions from the queue, setting the status of transactions, etc.

[0089] The monitoring REST endpoint can monitor the web application 342 and the agent 314. The notification and monitoring API 346 can be a REST endpoint for registering the agent 314, passing configuration settings to the agent 314, and sending / receiving notifications from the server and the agent 314. In some embodiments, the notification and monitoring API 346 can also use WebSocket communication.

[0090] In some embodiments, the APIs in the service layer can be accessed by configuring the appropriate API access path, for example, based on whether the commander 340 and the entire hyperautomation system have a local deployment type or a cloud-based deployment type. The API for the commander 340 can provide custom methods for querying statistics about various entities registered in the commander 340. In some embodiments, each logical resource can be an OData entity. In such an entity, components such as robots, processes, queues, etc. can have attributes, relationships, and operations. In some embodiments, the commander 340 API can be consumed by the web application 342 and / or the agent 314 in two ways: by obtaining API access information from the commander 340, or by registering an external application to use the OAuth flow.

[0091] In this embodiment, the persistence layer includes three servers - a database server 350 (e.g., a SQL server), an AI / ML server 360 (e.g., a server providing AI / ML model services, such as an AI center function), and an indexer server 370. In this embodiment, the database server 350 stores configurations for robots, robot groups, related processes, users, roles, scheduling, etc. In some embodiments, this information is managed by a web application 342. The database server 350 can manage queues and queue items. In some embodiments, the database server 350 can store messages recorded by the robot (in addition to or instead of the indexer server 370). For example, the database server 350 can also store data related to process mining, task mining, and / or task capture received from a listener 330 installed on the client side. Although no arrow is shown between the listener 330 and the database 350, it should be understood that in some embodiments, the listener 330 is able to communicate with the database 350, and vice versa. The data can be stored in the form of a PDD, an image, a XAML file, etc. Listener 330 may be configured to intercept user actions, processes, tasks, and performance indicators on the corresponding computing system where listener 330 resides. For example, listener 330 may record user actions (e.g., clicks, typed characters, locations, applications, active elements, time, etc.) on its corresponding computing system and then convert these into a suitable format to be provided to and stored in database server 350.

[0092] The AI / ML server 360 facilitates the incorporation of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options can make these capabilities accessible even to those who are not data scientists. Deployed automation (e.g., RPA robots) can call AI / ML models from the AI / ML server 360. The performance of AI / ML models can be monitored and trained and improved using manually verified data. The AI / ML server 360 can schedule and execute training jobs to train new versions of AI / ML models.

[0093] The AI / ML server 360 may store data related to AI / ML models and ML groups for configuring various ML skills for users at the time of development. As used herein, for example, ML skills are pre-built and trained ML models for processes that can be used by automation. The AI / ML server 360 may also store data related to document understanding technologies and frameworks, algorithms, and software groups for various AI / ML capabilities (including but not limited to intent analysis, NLP, speech analysis, different types of AI / ML models, etc.).

[0094] Indexer server 370 (optional in some embodiments) stores and indexes information recorded by the robot. In some embodiments, indexer server 370 can be disabled through configuration settings. In some embodiments, indexer server 370 uses the open source project full-text search engine Messages logged by the robot (e.g., using activities like logging messages or writing lines) can be sent to the indexer server 370 via the logging REST endpoint(s), where they are indexed for future utilization.

[0095] Figure 4 4 is an architectural diagram showing the relationship 400 between the designer 410, activities 420, 430, 440, 450, driver 460, API 470 and AI / ML model 480 according to an embodiment of the present invention. As described above, the developer uses the designer 410 to develop a workflow executed by the robot. In some embodiments, various types of activities can be displayed to the developer. The designer 410 can be local to the user's computing system or remote from it (for example, accessed via a VM or a local web browser interacting with a remote web server). The workflow can include user-defined activities 420, API-driven activities 430, AI / ML activities 440 and / or UI automation activities 450. User-defined activities 420 and API-driven activities 440 interact with applications via their APIs. In some embodiments, user-defined activities 420 and / or AI / ML activities 440 can call one or more AI / ML models 480, which can be local to and / or remote from the computing system on which the robot operates.

[0096] Some embodiments are capable of identifying non-textual visual components (referred to herein as CV) in images. However, it should be noted that in some embodiments, CV incorporates OCR. CV may be performed at least in part by (multiple) AI / ML models 480. Some CV activities related to such components may include, but are not limited to, extracting text from segmented label data using OCR, fuzzy text matching, using ML to crop segmented label data, comparing text extracted from label data with real data, etc. In some embodiments, there may be hundreds or even thousands of activities that can be implemented in user-defined activities 420. However, any number and / or type of activities may be used without departing from the scope of the present invention.

[0097] UI automation activities 450 are a subset of special lower-level activities written in lower-level code and facilitating interaction with the screen. UI automation activities 450 facilitate these interactions via drivers 460 that allow the robot to interact with the desired software. For example, drivers 460 may include operating system (OS) drivers 462, browser drivers 464, VM ​​drivers 466, enterprise application drivers 468, etc. In some embodiments, UI automation activities 450 may use one or more AI / ML models 480 to perform interactions with the computing system. In some embodiments, AI / ML models 480 may augment drivers 460 or completely replace them. In fact, in some embodiments, drivers 460 are not included.

[0098] Driver 460 can interact with the OS at a low level via OS driver 462 to find hooks, monitor keys, etc. Driver 460 can facilitate communication with For example, the "click" activity performs the same role in these different applications via the driver 460.

[0099] Figure 5 is an architectural diagram illustrating a computing system 500 configured to perform RPA robot maintenance using generative AI according to an embodiment of the present invention. In some embodiments, computing system 500 may be one or more computing systems depicted and / or described herein. In some embodiments, computing system 500 may be part of a hyper-automated system, such as Figure 1 and Figure 2As shown. The computing system 500 includes a bus 505 or other communication mechanism for transmitting information, and (multiple) processors 510 coupled to the bus 505 to process information. (Multiple) processors 510 can be any type of general or special-purpose processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. (Multiple) processors 510 may also have multiple processing cores, and at least some of the cores may be configured to perform specific functions. In some embodiments, multi-parallel processing may be used. In some embodiments, at least one of the (multiple) processors 510 may be a neuromorphic circuit including a processing element that simulates a biological neuron. In some embodiments, the neuromorphic circuit may not require typical components of the von Neumann computing architecture.

[0100] The computing system 500 also includes a memory 515 for storing information and instructions to be executed by the (multiple) processors 510. The memory 515 can be composed of random access memory (RAM), read-only memory (ROM), flash memory, cache, static memory (such as a disk or optical disk), or any other type of non-transitory computer-readable medium or any combination thereof. Non-transitory computer-readable media can be any available media that can be accessed by the (multiple) processors 510, and can include volatile media, non-volatile media, or both. The medium can also be removable, non-removable, or both. The computing system 500 includes a communication device 520 (such as a transceiver) to provide access to a communication network via wireless and / or wired connections. In some embodiments, the communication device 520 may include one or more antennas, which are single, array, phased, switched, beamformed, beam-steering, combinations thereof, and / or any other antenna configurations without departing from the scope of the present invention.

[0101] Processor(s) 510 are also coupled to display 525 via bus 505. Any suitable display device and tactile I / O may be used without departing from the scope of the present invention. Keyboard 530 and cursor control device 535 (such as a computer mouse, touchpad, etc.) are also coupled to bus 505 to enable a user to interact with computing system 500. However, in some embodiments, there may be no physical keyboard and mouse, and the user may interact with the device only through display 525 and / or touchpad (not shown). Any type and combination of input devices may be used, depending on design choice. In some embodiments, there is no physical input device and / or display. For example, a user may interact with computing system 500 remotely via another computing system communicating therewith, or computing system 500 may operate autonomously.

[0102] The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. The modules include an operating system 540 for the computing system 500. The modules also include an automated RPA robot maintenance module 545, which is configured to perform all or part of the AI / ML processes described herein or derivatives thereof. The computing system 500 may include one or more additional function modules 550 that include additional functionality.

[0103] Those skilled in the art will appreciate that a "system" may be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device or device combination without departing from the scope of the present invention. Presenting the above functions as being performed by a "system" is not intended to limit the scope of the present invention in any way, but is intended to provide an example of many embodiments of the present invention. In fact, the methods, systems, and devices disclosed herein may be implemented in a localized and distributed form consistent with computing technology (including cloud computing systems). The computing system may be part of a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, a public or private cloud, a hybrid cloud, a server cluster, any combination thereof, or otherwise accessible. Any localized or distributed architecture may be used without departing from the scope of the present invention.

[0104] It should be noted that some of the system features described in this specification have been presented as modules in order to more particularly emphasize their implementation independence. For example, a module can be implemented as a hardware circuit, including a custom very large scale integration (VLSI) circuit or gate array, a finished semiconductor such as a logic chip, a transistor or other discrete component. A module can also be implemented in a programmable hardware device (such as a field programmable gate array, a programmable array logic, a programmable logic device, a graphics processing unit, etc.).

[0105] Modules may also be implemented at least in part in software for execution by various types of processors. The identified unit of executable code may, for example, include one or more physical blocks or logical blocks of computer instructions, which may, for example, be organized as objects, programs, or functions. However, the executable programs of the identified modules do not need to be physically located together, but may include multiple instructions stored in different locations, which, when logically combined together, constitute the modules and achieve the purpose for the modules. In addition, the modules may be stored on a computer-readable medium, which may, without departing from the scope of the present invention, be, for example, a hard drive, a flash memory device, a RAM, a magnetic tape, and / or any other such non-transient computer-readable medium for storing data.

[0106] In practice, a module of executable code may be a single instruction or many instructions, and may even be distributed across several different code segments, between different programs, and across several memory devices. Similarly, operational data may be identified and shown within a module herein, and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set, or may be distributed across different locations (including across different storage devices), and may exist at least in part as only electronic signals on a system or network.

[0107] Various types of AI / ML models can be trained and deployed without departing from the scope of the present invention. For example, Fig. 6A An example of a neural network 600 (e.g., a generative AI model) that has been trained to perform RPA robot maintenance according to an embodiment of the present invention is shown. The neural network 600 includes multiple hidden layers. Both DLNNs and shallow learning neural networks (SLNNs) tend to have multiple layers, although SLNNs may have only one or two layers in some cases, and are generally less than DLNNs. Typically, a neural network architecture includes an input layer, multiple intermediate layers, and an output layer, as is the case in the neural network 600.

[0108] DLNNs often have many layers (e.g., 10, 50, 200, etc.), and subsequent layers often reuse features from previous layers to compute more complex general functions. On the other hand, SLNNs tend to have only a few layers and are relatively fast to train because expert features are created in advance from raw data samples. However, feature extraction is laborious. On the other hand, DLNNs tend not to require expert features, but tend to take longer to train and have more layers.

[0109] For both methods, the layers are trained simultaneously on a training set, usually with a check for overfitting on an isolated cross-validation set. Both techniques can produce excellent results, and there is considerable enthusiasm for both. The optimal size, shape, and number of layers varies depending on the problem being solved by the neural network.

[0110] Back to Fig. 6A , text, screenshots, graphic elements, contextual information (e.g., information about the application to which the source belongs, the type of source, etc.) are provided as input layers and fed as inputs to the J neurons of the hidden layer 1. Although in this example all of these inputs are fed to each neuron, a variety of architectures that can be used alone or in combination are possible, including but not limited to feedforward networks, radial basis networks, deep feedforward networks, deep convolutional inverse graph networks, convolutional neural networks, recurrent neural networks, artificial neural networks, long / short-term memory networks, gated recurrent unit networks, generative adversarial networks, liquid machines, autoencoders, variational autoencoders, denoising autoencoders, sparse autoencoders, extreme learning machines, echo state networks, Markov chains, Hopfel networks, Boltzmann machines, restricted Boltzmann machines, deep residual networks, Kohonen networks, deep belief networks, deep convolutional networks, support vector machines, neural Turing machines, or any other suitable type or combination of neural networks without departing from the scope of the present invention.

[0111] Hidden layer 2 receives input from hidden layer 1, hidden layer 3 receives input from hidden layer 2, and so on for all hidden layers until the last hidden layer uses its output as input to the output layer. Although multiple suggestions are shown here as outputs, in some embodiments, only a single output suggestion is provided. In some embodiments, the suggestions are ranked based on confidence scores.

[0112] It should be noted that the number of neurons I, J, K, and L is not necessarily equal. Thus, any desired number of layers may be used for a given layer of neural network 600 without departing from the scope of the present invention. In fact, in some embodiments, the types of neurons in a given layer may not be exactly the same.

[0113] The neural network 600 is trained to assign (multiple) confidence scores to the appropriate outputs. In some embodiments, to reduce inaccurate predictions, only those results whose confidence scores meet or exceed a confidence threshold can be provided. For example, if the confidence threshold is 80%, outputs with confidence scores exceeding this value can be used, and the remaining outputs can be ignored.

[0114] It should be noted that neural networks are probabilistic constructs that typically have (multiple) confidence scores. This may be a score that the AI / ML model has learned based on how often it correctly identified similar inputs during training. Some common types of confidence scores include a decimal between 0 and 1 (which can also be interpreted as a confidence percentage), a number between negative ∞ and positive ∞, a set of expressions (e.g., "low", "medium", and "high"), etc. Various post-processing calibration techniques can also be employed to try to obtain more accurate confidence scores, such as temperature scaling, batch normalization, weight decay, negative log likelihood (NLL), etc.

[0115] A "neuron" in a neural network is algorithmically implemented as a mathematical function that is generally based on the function of a biological neuron. A neuron receives a weighted input and has a summation function and an activation function that governs whether the neuron passes the output to the next layer. The activation function can be a nonlinear threshold activity function, where nothing happens if the value is below the threshold, but the function responds linearly when it is above the threshold (i.e., a rectified linear unit (ReLU) nonlinearity). The summation function and the ReLU function are used for deep learning because real neurons can have roughly similar activity functions. Information can be subtracted, added, etc. via linear transformations. In essence, a neuron acts as a gating function that passes the output to the next layer governed by its underlying mathematical function. In some embodiments, different functions may be used for at least some neurons.

[0116] Figure 6B An example of a neuron 610 is shown in FIG. The inputs x1, x2, ..., x from the previous layer are assigned corresponding weights w1, w2, ..., w n . Therefore, the total input from the previous neuron 1 is w1x1. These weighted inputs are used in the bias-modified summation function of the neuron, such as:

[0117]

[0118] This sum is compared to the activation function f(x) to determine whether the neuron "fires". For example, f(x) can be given by:

[0119]

[0120] Therefore, the output y of neuron 610 can be given by:

[0121]

[0122] In this case, neuron 610 is a single layer perceptron. However, any suitable neuron type or combination of neuron types may be used without departing from the scope of the present invention. It should also be noted that in some embodiments, the range of weight values ​​and / or output values ​​of the activation function may be different without departing from the scope of the present invention.

[0123] An objective or "reward function" is often employed. The reward function explores intermediate transitions and steps with short-term rewards and long-term rewards to guide the search of the state space and attempt to achieve the goal (e.g., finding the most accurate answer to a user query based on relevant metrics). During training, various labeled data are fed through the neural network 600. Successful recognitions strengthen the weights of the neuron inputs, while unsuccessful recognitions weaken the weights. A cost function (such as mean square error (MSE) or gradient descent) can be used to penalize slightly wrong predictions much less than very wrong predictions. If the performance of the AI / ML model does not improve after a certain number of training iterations, the data scientist can modify the reward function, provide corrections for incorrect predictions, and so on.

[0124] Backpropagation is a technique for optimizing synaptic weights in feed-forward neural networks. Backpropagation can be used to "open" the hidden layers of a neural network to see how much loss each node is responsible for, and then update the weights in a way that minimizes the loss by giving lower weights to nodes with higher error rates (and vice versa). In other words, backpropagation allows data scientists to repeatedly adjust weights in order to minimize the difference between actual and desired outputs.

[0125] The back propagation algorithm is mathematically based on optimization theory. In supervised learning, training data with known outputs is passed through a neural network and the error is calculated from the known target output using a cost function, which provides the error for back propagation. The error is calculated at the output and transformed into a correction to the network weights that minimizes the error.

[0126] In the case of supervised learning, an example of backpropagation is provided below. The column vector input x passes through a series of N nonlinear activation functions f between each layer i=1,...,N of the network. i The process is performed where the output of a given layer is first multiplied by the synaptic matrix W i And add the bias vector b i .

[0127] The network output o is given by

[0128] o=f N (W N f N-1 (W N-1 f N-2 (...f1(W1x+b1)...)+b N-1 )+b N ) (4)

[0129] In some embodiments, o is compared to a target output t, thereby generating an error that is desired to be minimized.

[0130] Optimization in the form of a gradient descent procedure can be used to modify the synaptic weights W of each layer by i Minimize the error. The gradient descent procedure requires computing an output o given an input x corresponding to a known target output t, and producing an error ot. This global error is then backpropagated given local errors in the weight updates, which are computed similarly but not identically to those used in the forward pass. In particular, the backpropagation step typically requires a procedure of the form p j (n j ) = f′ j (n j ) activity function, where n j is the network activity at layer j (i.e., n j =W j o j-1 +b j ), where o j =f j (n j ), the prime symbol ′ represents the derivative of the activity function f.

[0131] The weight update can be calculated using the following formula:

[0132]

[0133] where ° denotes the Hadamard product (i.e., the element-wise product of two vectors), T represents the matrix transpose, and o j represents f j (W j o j-1 +b j ), where o0=x. Here, the learning rate η is chosen with respect to machine learning considerations. Below, η is related to the neural Hebbian learning mechanism used in the neural implementation. Note that the synapses W and b can be combined into a large synaptic matrix, where the input vector is assumed to have additional synapses, and additional columns representing b synapses are included in W.

[0134] The AI / ML model may be trained over multiple training epochs until it reaches a good level of accuracy (e.g., 97% or better using an F2 or F4 threshold for detection and approximately 2000 training epochs). In some embodiments, this level of accuracy may be determined using an F1 score, an F2 score, an F4 score, or any other suitable technique without departing from the scope of the invention. Once trained on the training data, the AI / ML model may be tested on an evaluation data set that the AI / ML model has not previously encountered. This helps ensure that the AI / ML model is not "overfitted," such that it performs well on the training data but poorly on other data.

[0135] In some embodiments, it may not be known what level of accuracy the AI / ML model may achieve. Accordingly, if the accuracy of the AI / ML model begins to decline when analyzing the evaluation data (i.e., the model performs well on the training data, but poorly on the evaluation data), the AI / ML model may undergo more training cycles on the training data (and / or new training data). In some embodiments, the AI / ML model is deployed only if the accuracy reaches a certain level or the accuracy of the trained AI / ML model is better than the existing deployed AI / ML models. In certain embodiments, a collection of trained AI / ML models may be used to complete a task. For example, one model may be trained to recognize images, another model may recognize text, yet another model may recognize semantic and / or ontological associations, and so on.

[0136] Some embodiments may use transformer networks (such as SentenceTransformers TM ), a Python library for state-of-the-art sentence, text, and image embeddings TM Framework. Such a transformer network learns associations between words and phrases that have both high and low scores. This trains the AI / ML model to determine what is close to the input and what is not. The transformer network can also use field lengths and field types instead of pairs of words / phrases.

[0137] As described above, in some embodiments, NLP technologies such as word2vec, BERT, GPT-3, ChatGPT, other LLMs, etc. can be used to promote semantic understanding and provide more accurate and human-like answers. Other technologies such as clustering algorithms can be used to find similarities between groups of elements. Clustering algorithms can include but are not limited to density-based algorithms, distribution-based algorithms, centroid-based algorithms, and hierarchy-based algorithms. K-means clustering algorithm, DBSCAN clustering algorithm, Gaussian mixture model (Gaussian mixture model, GMM) algorithm, balance iterative reducing and clustering using hierarchies (balance iterative reducing and clustering using hierarchies, BIRCH) algorithm, etc. These technologies can also assist in classification.

[0138] Figure 7 700 is a flow chart illustrating a process 700 for training (multiple) AI / ML models according to an embodiment of the present invention. In some embodiments, as described above, (multiple) AI / ML models may be generative AI models. The neural network architecture of an AI / ML model typically includes multiple layers of neurons, including an input layer, an output layer, and a hidden layer. For example, see Fig. 6A and Figure 6B The intermediate hidden layers process the input data and generate an intermediate representation of the input that is used to generate the output. These hidden layers may include various types of neurons, such as convolutional neurons, recurrent neurons, and / or transformer neurons.

[0139] At 710, the training process begins by providing text, screenshots, graphical elements, and contextual information, whether labeled or unlabeled. The AI / ML model is then trained over multiple training cycles at 720, and the results are reviewed at 730. While various types of AI / ML models can be used, LLM and other generative AI models are typically trained using a process known as "supervised learning" (also discussed above). Supervised learning involves providing the model with a large data set, which the model uses to learn the relationship between inputs and outputs. During the training process, the model adjusts the weights and biases of the neurons in the neural network to minimize the difference between the predicted outputs and the actual outputs in the training data set.

[0140] In some embodiments, one aspect of the model is the use of transfer learning. For example, transfer learning can utilize a pre-trained model (such as ChatGPT) that is fine-tuned on a specific task or domain in step 720. This allows the model to leverage knowledge already learned from the pre-training phase and adapt it to a specific application via the training phase of step 720.

[0141] The pre-training phase involves training the model on a potentially more general initial training data set. During this phase, the model learns relationships in the data. In the fine-tuning phase (e.g., in some embodiments, if the pre-trained model is used as the initial basis for the final model, the fine-tuning phase is performed during step 720 in addition to or in place of the initial training phase), the pre-trained model is adapted to a specific task or domain by training the model on a smaller data set specific to the task. For example, in some embodiments, the model may focus on a certain (certain) type of data source. This can help the model identify the data elements therein more accurately than a generative AI model pre-trained alone. Fine-tuning allows the model to learn the nuances of the source, such as specific vocabulary and grammar, certain graphic features, certain data formats, etc., without requiring as much data as would be necessary to train the model from scratch. By leveraging the knowledge learned in the pre-training phase, the fine-tuned model can achieve state-of-the-art performance on a specific task with relatively little additional training data.

[0142] If the AI / ML model fails to meet the desired confidence threshold at 740, the training data is supplemented and / or the reward function is modified at 750 to help the AI / ML model better achieve its goal, and the process returns to step 720. If the AI / ML model meets the confidence threshold at 740, the AI / ML model is tested on the evaluation data at 760 to ensure that the AI / ML model generalizes well and that the AI / ML model is not overfitted relative to the training data. The evaluation data includes information that the AI / ML model has not processed before. If the confidence threshold is met for the evaluation data at 770, the AI / ML model is deployed at 780. If not, the process returns to step 750 and the AI / ML model is further trained.

[0143] Figure 8 FIG. 8 shows an AI / ML model of the cognitive AI layer 800 according to an embodiment of the present invention. In some embodiments, the model of the cognitive AI layer 800 may be used Figure 7 The generative AI model 810 provides results to other cognitive AI layer models in a series configuration 840, a parallel configuration 842, or a combination of a series configuration and a parallel configuration 844. For example, the generative AI model can generate code, provide semantic associations between text on the screen, and so on. The CV model 820 and the OCR model also provide detected graphic elements and recognized text to the generative AI model 810 and other cognitive AI layer models in configurations 840, 842, or 844, respectively.

[0144] Other cognitive AI layer models in configurations 840, 842, or 844 use outputs from generative AI model 810, CV model 820, and / or OCR model 830 to facilitate understanding of the intent of the RPA process, how the RPA process has executed so far, and what a human would do when the RPA process fails before completion, and what the best course of action is in terms of repair. Outputs (i.e., results 850) from other cognitive AI layer models in configurations 840, 842, or 844 include recommendations on what to change for future versions of the automation to avoid encountering the same problem, automatically generated code blocks and / or automation that understand the intent and cause of the failure for recovery during runtime execution, instructions for causing the RPA robot to pause execution until a human user provides manual input, rollback of steps previously taken in the automation to eliminate the failure, and the like.

[0145] Fig. 9 is a flow chart illustrating a process for performing RPA robot maintenance using generative AI according to an embodiment of the present invention. The process begins at 910 with the RPA robot or a monitoring process (e.g., another application that monitors the performance of the RPA robot as it executes) detecting a failure in an RPA robot activity. For example, when the RPA robot calls a CV model or an OCR model, the CV model or the OCR model does not find a graphic element or text associated with the activity on the screen after a predetermined amount of time or after all screen information has been processed, thereby detecting that the action of the RPA robot associated with the activity causes the operating system or application to throw an error, determining that the execution of the activity produces an output that causes the next activity in the RPA workflow to not receive correct input and / or failure, and so on, such a failure can be determined.

[0146] The RPA robot or monitoring process then provides the formatted information as input to the cognitive AI layer at 920, which then uses the formatted information to generate output and corresponding (multiple) confidence scores. In some embodiments, the cognitive AI layer includes a CV model, an OCR model, a generative AI model, and one or more additional AI / ML models that facilitate understanding of what is happening. Then, at 930, the RPA robot receives the output from the cognitive AI layer. The output from the cognitive AI layer may include, but is not limited to, suggestions on what to change for future versions of the automation to avoid encountering the same problem, automatically generated code blocks and / or automation that understand the intent and cause of the failure for recovery during runtime execution, instructions for pausing the execution of the RPA robot until a human user provides manual input, rollback of steps previously taken in the automation to eliminate the failure, etc.

[0147] If the confidence score(s) associated with the output at 940 meets or exceeds an acceptance threshold (e.g., 90%, 95%, 99%, etc.), the RPA robot performs the action(s) associated with the cognitive AI layer output at 950. However, in some embodiments, the RPA robot may attempt the action(s) associated with the cognitive AI layer output regardless of the confidence score. However, if the confidence score(s) at 940 is below the threshold, the RPA robot or monitoring process may request human assistance at 960. For example, the RPA robot may ask a human user of the computing system on which the RPA robot is executing to click on the correct graphical element on the screen, provide correct text, etc. This information can then be used as part of the training data to retrain the AI / ML model of the cognitive AI layer in the future.

[0148] According to an embodiment of the present invention, Figure 7 and Fig. 9 The process steps performed in the embodiment may be performed by a processor (or processors) for causing the processor (or processors) to perform Figure 7 and Fig. 9 The computer program may be embodied on a non-transitory computer readable medium. The computer readable medium may be, but is not limited to, a hard drive, a flash memory device, a RAM, a magnetic tape, and / or any other such medium or combination of media for storing data. The computer program may include instructions for controlling the processor(s) of the computing system (e.g., Figure 5 The computing system 500 of the processor (s) 510) implements Figure 7 and Fig. 9 The coded instructions of all or part of the process steps described in the present invention can also be stored on a computer-readable medium.

[0149] The computer program may be implemented in hardware, software or a hybrid implementation. The computer program may consist of modules that are in effective communication with each other and are designed to pass information or instructions to display. The computer program may be configured to operate on a general-purpose computer, an ASIC or any other suitable device.

[0150] It will be readily understood that, as generally described and illustrated in the accompanying drawings herein, the components of the various embodiments of the present invention may be arranged and designed in a variety of different configurations. Therefore, the detailed description of the embodiments of the present invention, as represented in the accompanying drawings, is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention.

[0151] The features, structures or characteristics of the present invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, reference to "certain embodiments", "some embodiments" or similar language throughout this specification means that the special features, structures or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present invention. Therefore, the phrases "certain embodiments", "in some embodiments", "in other embodiments" or similar language that appear throughout this specification do not necessarily all refer to the same set of embodiments, and the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments.

[0152] It should be noted that references to features, advantages, or similar language throughout this specification do not mean that all features and advantages that may be realized by the present invention are or should be in any single embodiment of the present invention. Instead, language referring to features and advantages is understood to mean that a particular feature, advantage, or characteristic described in conjunction with an embodiment is included in at least one embodiment of the present invention. Therefore, throughout the specification, discussion of features and advantages and similar language may, but do not necessarily, refer to the same embodiment.

[0153] In addition, the features, advantages and characteristics of the present invention may be combined in one or more embodiments in any suitable manner. Those skilled in the relevant art will recognize that the present invention may be practiced without one or more specific features or advantages of a particular embodiment. In other cases, additional features and advantages that may not be present in all embodiments of the present invention may be recognized in certain embodiments.

[0154] Those skilled in the art will readily appreciate that the invention as discussed above may be practiced utilizing steps in a different order and / or utilizing hardware elements in a configuration different from that disclosed. Thus, although the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative configurations will be apparent while remaining within the spirit and scope of the invention. Therefore, in order to determine the boundaries of the invention, the appended claims should be referenced.

Claims

1. One or more non-transitory computer-readable media storing a computer program for performing robotic process automation (RPA) robot maintenance using a cognitive artificial intelligence (AI) layer, the computer program being configured to cause at least one processor to: Executing automation via RPA robots, said automation executing logic associated with the RPA workflow; Detecting a failure in an activity of the RPA workflow being executed by the RPA robot; providing information as input to a cognitive AI layer comprising a generative AI model, the cognitive AI layer being trained to provide output for said automating said maintenance; receiving the output from the cognitive AI layer; and performing one or more actions by the RPA robot in response to the output from the cognitive AI layer to facilitate the automated maintenance, wherein The activities are steps in the RPA workflow that are performed by the RPA robot in the sequence of the RPA workflow or some other logical flow.

2. One or more non-transitory computer-readable media according to claim 1, wherein the cognitive AI layer includes one or more other models, and the one or more other models are configured to: receive output from the generative AI model and understand the intention of the automation, the execution of the automation to date, and what a human would do and an action plan for the repair when the automation fails before completion.

3. One or more non-transitory computer readable media according to claim 2, wherein The cognitive AI layer includes a computer vision CV model, an optical character recognition OCR model, or both, and The CV model and / or the OCR model are configured to provide the detected graphic elements and text to the generative AI model, the one or more other models, or both, respectively.

4. One or more non-transitory computer-readable media according to claim 1, wherein the output received from the cognitive AI layer includes one or more activity goal suggestions for the RPA workflow, and the computer program is configured to cause the at least one processor to use the one or more activity goal suggestions to perform the maintenance of the RPA workflow.

5. The one or more non-transitory computer-readable media of claim 1 , wherein the output received from the cognitive AI layer comprises a suggested repair for a faulty activity of the RPA workflow, and the computer program is further configured to cause the at least one processor to: transmitting the suggested repair to the RPA designer application; receiving updated automation from the RPA designer application; and The updated automation is executed in place of the automation.

6. The one or more non-transitory computer-readable media of claim 1, wherein the computer program is further configured to cause the at least one processor to: Providing input to the generative AI model and / or the one or more other models of the cognitive AI layer, the input comprising: the original automation blueprint, representing the intent of said automation, The current execution log is a textual record of the execution of the automation so far. The current execution log, a visual representation of the automation's execution so far, The cognitive AI layer’s interpretation of code, logs, and records, RPA automation language, including the activities and building blocks of the RPA workflow, Current screen technology means that Boundaries or rules in text form that prevent the RPA robot from performing certain actions, accessing certain information, or both, or Any combination of the above.

7. One or more non-transitory computer-readable media according to claim 1, wherein the output from the cognitive AI layer includes one or more recommendations regarding changes to future versions of the automation to avoid the failure.

8. One or more non-transitory computer-readable media according to claim 1, wherein the output from the cognitive AI layer includes automatically generated code blocks or versions of the automation for recovery by the RPA robot during runtime execution of the automation.

9. One or more non-transitory computer-readable media according to claim 1, wherein the RPA robot is configured to: pause execution of the automation and wait for a human user to provide manual input before continuing with the maintenance based on the output from the cognitive AI layer.

10. The one or more non-transitory computer-readable media of claim 1, wherein the output from the cognitive AI layer comprises: recommending a code change to circumvent the fault, the recommended code change maintaining the underlying logic of the automation; Applying a change suggestion to circumvent an issue associated with the failure but maintaining the underlying business logic of the automation; Business rule recommendations to improve said execution of said RPA workflow in terms of performance and / or return on investment (ROI); test cases to detect said fault more quickly in the future; or Any combination of the above.

11. The one or more non-transitory computer-readable media of claim 1 , wherein the output from the cognitive AI layer comprises instructions for the RPA robot to roll back the automated steps associated with the failure, and the computer program is further configured to cause the at least one processor to: The steps of the automation associated with the failure are rolled back based on the output from the cognitive AI layer.

12. One or more computing systems comprising: A memory storing computer program instructions for performing robotic process automation (RPA) robot maintenance using a cognitive artificial intelligence (AI) layer; as well as at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to: Detect failures in the activities of the RPA workflow being executed by the RPA robot; providing information as input to a cognitive AI layer including a generative AI model and one or more other models configured to understand the intent of the automation associated with the RPA workflow, the execution of the automation to date, and what a human would do and a course of action for the repair when the automation fails before completion, the cognitive AI layer being trained to provide output for the repair of the automation; receiving the output from the cognitive AI layer; and providing the output to the RPA robot, wherein The activities are steps in the RPA workflow that are performed by the RPA robot in the sequence of the RPA workflow or some other logical flow.

13. One or more computing systems according to claim 12, wherein The cognitive AI layer includes a computer vision CV model, an optical character recognition OCR model, or both, and The CV model and / or the OCR model are configured to provide the detected graphic elements and text to the generative AI model, the one or more other models, or both, respectively.

14. One or more computing systems according to claim 12, wherein the output received from the cognitive AI layer includes one or more activity goal suggestions for the RPA workflow, and the computer program instructions are configured to cause the at least one processor to perform the maintenance of the RPA workflow using the one or more activity goal suggestions.

15. The one or more computing systems of claim 12, wherein the output received from the cognitive AI layer comprises a suggested repair for a faulty activity of the RPA workflow, and the computer program instructions are further configured to cause the at least one processor to: transmitting the suggested repair to the RPA designer application; receiving updated automation from the RPA designer application; and The updated automation is sent to the RPA robot to be executed in place of the automation.

16. The one or more computing systems of claim 12, wherein the computer program instructions are further configured to cause the at least one processor to: Providing input to the generative AI model and / or the one or more other models of the cognitive AI layer, the input comprising: the original automation blueprint, representing the intent of said automation, The current execution log is a textual record of the execution of the automation so far. The current execution log, a visual representation of the automation's execution so far, The cognitive AI layer’s interpretation of code, logs, and records, RPA automation language, including the activities and building blocks of the RPA workflow, Current screen technology means that Boundaries or rules in text form that prevent the RPA robot from performing certain actions, accessing certain information, or both, or Any combination of the above.

17. The one or more computing systems of claim 12, wherein the output from the cognitive AI layer includes one or more recommendations regarding changes to future versions of the automation to avoid the failure.

18. The one or more computing systems of claim 12, wherein the output from the cognitive AI layer comprises an automatically generated code block or version of the automation for recovery by the RPA robot during runtime execution of the automation.

19. The one or more computing systems of claim 12, wherein the output from the cognitive AI layer comprises: recommending a code change to circumvent the fault, the recommended code change maintaining the underlying logic of the automation; Applying a change suggestion to circumvent an issue associated with the failure but maintaining the underlying business logic of the automation; Business rule recommendations to improve the execution of the RPA workflow in terms of performance and / or return on investment (ROI); test cases to detect said fault more quickly in the future; or Any combination of the above.

20. The one or more computing systems of claim 12, wherein the output from the cognitive AI layer includes instructions for the RPA robot to roll back the automated steps associated with the failure.

21. A computer-implemented method for performing Robotic Process Automation (RPA) robot maintenance using a cognitive artificial intelligence (AI) layer, comprising: Automation is performed by an RPA robot, the automation executing logic associated with the RPA workflow; Detecting, by the RPA robot, a failure in an activity of the RPA workflow; providing information as input by the RPA robot to a cognitive AI layer including a generative AI model, the cognitive AI layer being trained to provide output for the automated maintenance; The RPA robot receives the output from the cognitive AI layer; as well as performing one or more actions by the RPA robot in response to the output from the cognitive AI layer to facilitate the automated maintenance, wherein The activities are steps in the RPA workflow that are performed by the RPA robot in the sequence of the RPA workflow or some other logical flow.

22. The computer-implemented method of claim 21 , wherein the cognitive AI layer comprises one or more other models configured to: receive the output from the generative AI model and understand the intent of the automation, the execution of the automation to date, and what a human would do and a course of action for the repair if the automation fails before completion.

23. The computer-implemented method of claim 22, wherein The cognitive AI layer includes a computer vision CV model, an optical character recognition OCR model, or both, and The CV model and / or the OCR model are configured to provide the detected graphic elements and text to the generative AI model, the one or more other models, or both, respectively.

24. A computer-implemented method according to claim 21, wherein the output received from the cognitive AI layer includes one or more activity goal suggestions for the RPA workflow, and the RPA robot is configured to use the one or more activity goal suggestions to perform the maintenance of the RPA workflow.

25. The computer-implemented method of claim 21, wherein the output received from the cognitive AI layer comprises a suggested repair for a faulty activity of the RPA workflow, and the method further comprises: The RPA robot transmits the suggested maintenance to the RPA designer application; receiving, by the RPA robot, updated automation from the RPA designer application; as well as The updated automation is performed by the RPA robot to replace the automation.

26. The computer-implemented method of claim 21, further comprising: The RPA robot provides input to the generative AI model and / or the one or more other models of the cognitive AI layer, wherein the input includes: the original automation blueprint, representing the intent of said automation, The current execution log is a textual record of the execution of the automation so far. The current execution log, a visual representation of the automation's execution so far, The cognitive AI layer’s interpretation of code, logs, and records, RPA automation language, including the activities and building blocks of the RPA workflow, Current screen technology means that Boundaries or rules in text form that prevent the RPA robot from performing certain actions, accessing certain information, or both, or Any combination of the above.

27. The computer-implemented method of claim 21, wherein the output from the cognitive AI layer includes one or more recommendations regarding changes to future versions of the automation to avoid the failure.

28. The computer-implemented method of claim 21, wherein the output from the cognitive AI layer comprises an automatically generated code block or version of the automation for recovery by the RPA robot during runtime execution of the automation.

29. The computer-implemented method of claim 21, further comprising: Suspending execution of the automation by the RPA robot and waiting for a human user to provide manual input before continuing with the repair based on the output from the cognitive AI layer.

30. The computer-implemented method of claim 21, wherein the output from the cognitive AI layer comprises: recommending a code change to circumvent the fault, the recommended code change maintaining the underlying logic of the automation; Applying a change suggestion to circumvent an issue associated with the failure but maintaining the underlying business logic of the automation; Business rule recommendations to improve said execution of said RPA workflow in terms of performance and / or return on investment (ROI); test cases to detect said fault more quickly in the future; or Any combination of the above.

31. The computer-implemented method of claim 21 , wherein the output from the cognitive AI layer includes instructions for the RPA robot to roll back automated steps associated with the failure, and the method further comprises: The automated steps associated with the failure are rolled back by the RPA robot based on the output from the cognitive AI layer.

Images