Anomaly Detection and Self-Healing for Robotic Process Automation with Artificial Intelligence / Machine Learning

Abstract

Anomaly detection and self-healing for robotic process automation (RPA) via artificial intelligence (AI) / machine learning (ML) is disclosed. RPA robots utilizing AI / ML models and computer vision (CV) can interpret and / or interact with most encountered graphical elements through normal, learned interactions. However, such RPA robots may occasionally encounter new, unhandled anomalies where they are unable to identify the graphical element and / or normal interactions do not work. Such anomalies can be handled by an anomaly handler. The RPA robot may have a self-healing capability that automatically attempts to find information to address the anomaly.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This is an international application claiming the benefit of and priority to U.S. Patent Application No. 17 / 070,250, filed October 14, 2020. The subject matter of this previously filed application is incorporated herein by reference in its entirety.

[0002] The present invention relates generally to user interface (UI) automation, and more specifically to anomaly detection and self-healing for robotic process automation (RPA) via artificial intelligence (AI) / machine learning (ML). [Background technology]

[0003] Currently, when an anomaly occurs during runtime (e.g., in a production environment), RPA analyzes the cause of the problem and RPA developers provide a programmatic solution based on the analysis. However, this process tends to be time-consuming and costly. Therefore, an improved approach could be beneficial. Summary of the Invention

[0004] Certain embodiments of the present invention may provide solutions to problems and needs in the field that have not yet been fully identified, appreciated, or solved by current UI automation technologies. For example, some embodiments of the present invention relate to anomaly detection and self-healing for RPA via AI / ML.

[0005] In an embodiment, a computer-implemented method includes performing, by an RPA robot, an RPA workflow that performs UI automation using an AI / ML model. The computer-implemented method also includes, by the RPA robot, using the AI / ML model to search for a target graphical element in a UI that is interacted with by an activity of the RPA workflow. If the target graphical element is not uniquely found or interacted with by the AI / ML model, the computer-implemented method further includes, by the RPA robot or the AI / ML model, automatically attempting to correct the anomaly.

[0006] In another embodiment, a non-transitory computer-readable medium stores a computer program. The computer program is configured to cause at least one processor to perform an RPA workflow that performs UI automation using an AI / ML model. Using the AI / ML model, the computer program is also configured to cause the at least one processor to search for a target graphical element in the UI that is interacted with by an activity of the RPA workflow. If the target graphical element is not uniquely found or interacted with by the AI / ML model, the computer program is further configured to cause the at least one processor to automatically attempt to correct the anomaly.

[0007] In yet another embodiment, a computing system includes a memory storing computer program instructions and at least one processor configured to execute the computer program instructions. The computer program instructions configure the at least one processor to execute, by an RPA robot, an RPA workflow that performs UI automation using an AI / ML model. Using the AI / ML model, the computer program instructions also configure the at least one processor to search, by the RPA robot, for a target graphical element in a UI that is interacted with by an activity of the RPA workflow. If the target graphical element is not uniquely found or interacted with by the AI / ML model, the computer program instructions further configure the at least one processor to automatically attempt, by the RPA robot or the AI / ML model, to correct the anomaly. The automatic attempt to correct the anomaly includes determining whether one or more characteristics distinguish the target graphical element from other similar graphical elements, attempting a self-healing process to complete missing data without user input, or both. [Brief explanation of the drawings]

[0008] So that the advantages of particular embodiments of this invention may be readily understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments which are illustrated in the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope, but the invention will be described and explained with additional specificity and detail through the use of the following accompanying drawings, in which:

[0009] [Figure 1] FIG. 1 is an architectural diagram illustrating a robotic process automation (RPA) system, according to an embodiment of the present invention.

[0010] [Figure 2] FIG. 1 is an architectural diagram illustrating a deployed RPA system according to an embodiment of the present invention.

[0011] [Figure 3] FIG. 2 is an architecture diagram illustrating the relationships between designers, activities, and drivers according to an embodiment of the present invention.

[0012] [Figure 4] FIG. 1 is an architectural diagram illustrating an RPA system according to an embodiment of the present invention.

[0013] [Figure 5] FIG. 1 is an architectural diagram illustrating a computing system configured to perform anomaly detection and self-healing for RPA via AI / ML, according to an embodiment of the present invention.

[0014] [Figure 6] FIG. 1 is an architectural diagram illustrating a system configured to perform anomaly detection and self-healing for RPA via AI / ML, according to an embodiment of the present invention.

[0015] [Figure 7] 1 is a flowchart illustrating a process for performing anomaly detection and self-healing of RPA via AI / ML, according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] Unless otherwise noted, like reference characters denote corresponding features consistently throughout the accompanying drawings.

[0017] Detailed Description of the Embodiments Some embodiments relate to anomaly detection and self-healing for RPA via AI / ML. RPA robots utilizing AI / ML models and computer vision (CV) may interpret and / or interact with most encountered graphical elements in a UI through normal, learned interactions. However, such RPA robots may occasionally encounter new, unhandled anomalies where they are unable to identify the graphical element and / or normal interactions do not work. Such anomalies, in some embodiments, may be handled by an anomaly handler in the RPA robot or AI / ML model.

[0018] When an anomaly occurs, such as when multiple submit buttons appear on a screen and the AI / ML model cannot determine on its own which button to click, the anomaly handler may attempt to determine if there is something that distinguishes these graphical elements from one another. This may include, but is not limited to, looking at the graphical elements that surround the target graphical element(s) within a certain radius, looking at the order of the graphical elements on the screen, attempting to determine if there is something that visually distinguishes the target graphical element from other similar graphical elements, etc. In some embodiments, a default action may be taken, such as automatically selecting the graphical element that appears closest to the top or bottom of the screen.

[0019] In some embodiments, the default action may be to search a UI object library containing UI objects or a UI object repository of UI object libraries for a UI descriptor that enables the RPA robot or AI / ML model to interact with the target graphical element. UI elements on a screen (e.g., text fields, buttons, labels, menus, check boxes, etc.) may be grouped by application, application version, application screen, and collection of UI elements, with each screen having multiple UI elements. As used herein, a “screen” is an image of an application UI or a portion of an application UI at a point in time (e.g., a screenshot). In the case of running UI automation, a screen may be the current UI of a computing system. An “application” or a version of a given application may be a combination of screens in this context or may be the running application at runtime. Each UI element, in some embodiments, may be described by one or more UI descriptors. UI elements, UI descriptors, applications, and application screens are UI objects. In some embodiments, UI elements and screens may be further distinguished into specific types of UI elements (e.g., buttons, checkboxes, text fields, etc.) and screens (e.g., top windows, modal windows, pop-up windows, etc.).

[0020] The UI object library, in some embodiments, may be stored in a UI object repository. As used herein, a UI object repository is a collection of UI object libraries. In some embodiments, the UI object repository may be a NuGet™ feed, a web service, or the like. The UI object repository of a UI object library in some embodiments may facilitate improved management, reuse, and reliability of UI descriptors in a project. In some embodiments, UI descriptors may be added to a UI library and published or republished for global reuse in the UI object repository.

[0021] To make the UI objects reusable, they can be extracted into a UI object library that can be referenced by RPA processes. For example, if a selector or other UI descriptor is modified due to a new version of an application, the UI object library can be recreated (or republished) to include the modified UI descriptor. RPA processes that use the UI object library can then invoke the modified UI descriptor version. References to new application programming interfaces (APIs) associated with the new library can be modified manually by a developer for the RPA process, via a software tool (e.g., a tool that updates UI descriptor references when selectors or other components change through the RPA process), or automatically in some embodiments.

[0022] In some embodiments, a package of UI objects may be attached as a dependency (e.g., a NuGet™ package). However, in certain embodiments, the UI objects may be obtained via one or more API calls to a web service, allowing the UI objects to be stored remotely and retrieved and used during automation.

[0023] UI elements added to the UI object library may represent selectors for use in an RPA workflow. To automate specific actions in the UI, an RPA robot may interact with various windows, buttons, drop-down lists, and / or other graphical elements. Traditionally, this is done using the expected screen positions of UI elements. However, this is unreliable.

[0024] Some embodiments use selectors that store attributes of UI elements and their parents in an XML fragment, overcoming problems associated with specifying fixed coordinates. In some embodiments where the UI is static, selectors may be generated automatically, but in some software programs, such as some web applications, layouts change and attribute nodes exist that have volatile values. These changes may not be easily predicted, which previously required some selectors to be generated manually. However, some embodiments may overcome this problem, for example, by using other types of UI descriptors.

[0025] A selector is a type of UI descriptor that can be used to find UI elements in some embodiments. A selector, in some embodiments, has the following structure: <node_1 / ><node_2 / > ...<node_N / >

[0026] The final node represents the target GUI element, and all previous nodes represent the parents of that element.<node_1> is usually called the root node and represents the top window of the application.

[0027] Each node may have one or more attributes that assist in correctly identifying the particular level of the selected application. Each node, in some embodiments, has the following format: <ui_system attr_name_1=’attr_value_1’...attr_name_N=’attr_value_N’ / >

[0028] All attributes may have values ​​assigned, and attributes with constant values ​​may be chosen because changing the value of an attribute every time the application launches may prevent the selector from correctly identifying the associated element.

[0029] UI object library descriptors can be added directly to RPA workflow activities, saving developers time that would otherwise be required to create custom selectors for the activities. An object library is defined herein as a collection of UI descriptors corresponding to one or more screens from a particular version of an application. A UI descriptor is a set of instructions for locating UI elements. In some embodiments, a UI descriptor is an encapsulated data / structure format that includes UI element selector(s), anchor selector(s), computer vision (CV) descriptor(s), integrated target descriptor(s), screen image capture (context), element image capture, other metadata (e.g., application and application version), or a combination thereof. The encapsulated data / structure format may be extensible with future updates to the platform and is not limited to the above definition. Any suitable UI descriptor for identifying UI elements on a screen may be used without departing from the scope of the present invention. An integrated target descriptor concatenates multiple types of UI descriptors. The integrated target descriptor may function like a finite state machine (FSM), where in a first context a first UI descriptor mechanism is applied, in a second context a second UI descriptor is applied, and so on.

[0030] Traditionally, making selectors reliable requires testing, understanding specific selector details, etc. This makes creating / improving selectors difficult or impossible for non-technical individuals. By providing a library of working selectors for common UI elements and tasks, some embodiments enable non-technical individuals to develop automation that works in production. A user can specify a screen of the version of an application to automate, and the RPA development application can fetch that screen. The user can then interact with the screen to define what he or she wants to do.

[0031] In some embodiments, as new UI descriptors are created and / or existing UI descriptors are modified, a global database of UI object libraries that is shareable, collaborative, and potentially open source may be built. In certain embodiments, taxonomies and ontologies may be used. Applications, versions, screens, UI elements, descriptors, etc. may be defined as taxonomies, which are hierarchical structures of subcategories.

[0032] However, many real-world concepts do not readily lend themselves to taxonomic organization. Rather, they may be closer to concepts in mathematical ontologies. In ontologies, relationships between categories are not necessarily hierarchical. For example, if clicking a button on a screen takes a user to another screen, the next screen is not in the hierarchy and therefore cannot be easily captured in that screen's taxonomy. In some embodiments, interactions between UI elements on the same or different screens may be accommodated, and information about how UI elements relate to each other on potentially different screens may be provided.

[0033] Consider the example where clicking an OK button brings up the Employees screen. With an ontology structure, the designer application may suggest that the user filter the employees on the next screen. The ontology information of the relationships between these screens via the OK button allows the designer application to do this. By defining a graphical structure that is not necessarily a tree, but rather related to what the application is actually doing, more complex and richer relationships can be captured.

[0034] UI descriptors may be extracted from activities in an RPA workflow and added to a structured schema that groups the UI descriptors by UI application, screen, and UI element. In some embodiments, the UI descriptors may be part of a single project for widespread reuse, part of a global repository for testing purposes, or part of a UI object library for global cross-project sharing. In some embodiments, the object library is an encapsulation of UI descriptors grouped by application, application version, and screen. UI descriptors may be defined and added to the object library, which, in some embodiments, may be installed as a dependency in other projects after publication. This allows the object library of UI descriptors to be reused. In some embodiments, the UI application is a target application with multiple versions, each with multiple screens.

[0035] The reuse of UI object repositories and UI descriptors can be beneficial for a variety of reasons. For example, if the position, appearance, and / or functionality of a UI element in an application changes, the UI descriptor can be modified, and the change can then be propagated to activities that use the UI descriptor. In this way, reusability can be provided at the specific level of graphical elements in the UI.

[0036] This can be particularly beneficial for accommodating new versions of an application. For a new version, software developers only need to update the UI descriptor and / or develop a few new selectors from scratch, significantly reducing development time. For example, for a web page, the hypertext markup language (HTML) may be inspected to obtain paths to UI elements for corresponding selectors. The selectors for corresponding UI elements may differ between different web browsers and / or different versions of the same web browser. This concept may also apply to visual desktop, server, smartphone, and tablet applications. A UI object library may include the actual selectors for an application, which, as described above, is just one type of UI descriptor. A UI object library may have multiple UI descriptors of different types for one or more applications. UI descriptors may refer to different technologies on which the application is built. For example, different UI descriptors may be created and used for standalone desktop, web, and mobile applications. For one technology, you may want to use selectors, while for another, you may want to use CV descriptors.

[0037] The UI descriptor, in some embodiments, may work with a unified target that encompasses all UI detection mechanisms by which image detection and definition are performed. The unified target may merge multiple techniques for identifying and automating UI elements into a single, cohesive approach. The unified target may prioritize selector-based and driver-based UI detection mechanisms and fall back to CV to find images if the first two mechanisms are unsuccessful. In some embodiments, the selector editor and UI explorer may be configured to support the unified target.

[0038] An example of the structure of an object library is shown below. SAP Version 1 Screen 1 Object 1 Object 2 ·… Screen 2… ·… Version 2… ·… Salesforce… ·…

[0039] Note that the above hierarchy is provided by way of example only. Any desired number of levels in the hierarchy and elements associated with each level may be used without departing from the scope of the present invention. In particular embodiments, a user may define the application map as he or she desires. Furthermore, some nodes in the tree may, in some embodiments, serve only an administrative role and not a functional role. For example, UI elements may be grouped together in a container that does not have a UI descriptor. The container, in some embodiments, may exist solely for grouping purposes.

[0040] To utilize a UI object library and / or UI object repository, if the target graphical element cannot be identified, the RPA robot or AI / ML model of some embodiments defaults to searching a UI object library or a repository of UI object libraries for UI descriptors that may be able to interact with the target graphical element. For example, the RPA robot or AI / ML model may look for UI descriptors with the same name, UI descriptors with the same or similar attributes, UI descriptors for the same target application but different versions, etc. If such UI descriptor(s) are found, they may be automatically tried by the RPA robot or AI / ML model. If the interaction with the target graphical element is successful, the RPA robot or AI / ML model may be modified to use the successful UI descriptor in the future in addition to or instead of previously tried UI descriptors. The UI object library or UI object repository may be local or remote to the RPA robot or AI / ML model that searches them.

[0041] If the anomaly handler cannot identify the target graphical element and / or if the default action fails, the RPA robot may gracefully handle the exception. For example, if the target graphical element is not necessary to complete the overall task with sufficient accuracy, the anomaly handler may cause the RPA robot to continue executing its workflow, prompt the user regarding whether to proceed, request labeled training data to use to address the anomaly in the future, etc. Such anomaly handling may make AI / ML models more robust to real-world variations and changes at runtime, which may occur when a new version of an application is released, when different browser settings are encountered, etc.

[0042] In some embodiments, self-healing capabilities may be provided such that the RPA robot and / or AI / ML model attempts to fill in missing data from anomalies without user input. The RPA developer may provide specific inputs to variables in the AI / ML model, pull data from a database, etc. In some embodiments, AI tools may be used to generate this code without human design time.

[0043] Self-healing, in some embodiments, may employ various techniques. In document understanding or other applications, for example, an AI / ML model may infer that it is not sufficient if human validation is frequently required. The AI / ML model may employ various techniques to attempt to improve its own performance as a means of "self-healing." For example, the AI / ML model may try different optical character recognition (OCR) engines, modify image characteristics (e.g., brightness, hue, contrast, conversion to grayscale, etc.), search for similar-looking images and check the applied technique(s) and / or results for those images, etc. To determine whether the AI / ML model's self-healing efforts are working, the AI / ML model may monitor whether human validation effort is reduced, execution speed and / or efficiency is improved (e.g., processes run faster and / or steps in an RPA workflow can be sped up or eliminated), or return on investment (ROI) is improved. If not, the AI / ML model may try different techniques and / or use different information. Based on this self-healing approach, AI / ML models can then learn how to impute missing, inaccurate, and / or incomplete data at runtime.

[0044] In some embodiments, reinforcement learning may be used. In reinforcement learning, there is an exploration phase and an exploitation phase. In the exploration phase, the system may randomly attempt to find a solution from a start state to an end state using a reward function that explores intermediate transitions and steps with both short-term and long-term rewards to guide the exploration of the state space. For example, if the anomaly is that a target graphical element is not found in the UI or an interaction with it has failed, the winning state may be the correct interaction with the target graphical element that an activity in the RPA workflow should achieve. If a solution is found, the RPA robot or AI / ML model may be modified to attempt this solution in addition to or instead of previous approaches.

[0045] Rather than conducting a completely random search, some embodiments determine what the policy should be via a policy network. This can dramatically reduce the number of simulations required to achieve a winning state, allowing AI / ML models trained via reinforcement learning to be developed and deployed more quickly, trained using fewer computing hardware resources, or both. Indeed, in some embodiments, reinforcement learning may be performed on a user's own computing system, such as a desktop computer, laptop computer, or smartphone, which is typically not traditionally possible.

[0046] In the context of interacting with a user interface, for example, a robot might randomly click on a screen to determine what is clickable. Reinforcement learning can then be used to drive the desired automation. For example, given a spreadsheet of email addresses and desired email text, reinforcement learning can be used to discover which clickable button will open the email application, discover where different text should be entered in the email, and discover which button to click to send the email. In this example, a policy network can first be used to determine whether it is more feasible or desirable to send the email. Once this determination is made, the policy network can then determine whether the email address and email body from the spreadsheet can be copied into the email before sending.

[0047] Certain embodiments may be employed in robotic process automation (RPA). FIG. 1 is an architectural diagram illustrating an RPA system 100 according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that enables developers to design and implement workflows. The designer 110 provides solutions for application integration and automates third-party applications, management information technology (IT) tasks, and business IT processes. The designer 110 can facilitate the development of automation projects, which are graphical representations of business processes. Simply put, the designer 110 facilitates the development and deployment of workflows and robots.

[0048] Automation projects enable rule-based process automation by giving developers control over the order of execution and relationships between custom sets of steps developed in workflows, defined herein as "activities." One commercial example of an embodiment of the designer 110 is UiPath Studio™. Each activity may include an action such as clicking a button, reading a file, or writing to a log panel. In some embodiments, workflows may be nested or embedded.

[0049] Workflow types may include, but are not limited to, sequences, flowcharts, FSMs, and / or global exception handlers. Sequences may be particularly well-suited for linear processes, allowing the flow of one activity from another without cluttering the workflow. Flowcharts may be particularly well-suited for more complex business logic, allowing for the integration of decisions and the connection of activities in more diverse ways through multiple branching logic operators. FSMs may be particularly well-suited for large workflows. FSMs may use a finite number of states during their execution that are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly well-suited for determining workflow behavior when an execution error is encountered or for debugging the process.

[0050] Once a workflow is developed in Designer 110, the execution of the business process is orchestrated by Conductor 120, which coordinates one or more Robots 130 that execute the workflow developed in Designer 110. One commercial example of an embodiment of Conductor 120 is UiPath Orchestrator™. Conductor 120 facilitates the management of the creation, monitoring, and deployment of resources in an environment. Conductor 120 may act as, or one of, an integration point with third-party solutions and applications.

[0051] The conductor 120 may manage all robots 130, connecting and executing them from a centralized point. Types of robots 130 that may be managed include, but are not limited to, attended robots 132, unattended robots 134, development robots (similar to unattended robots 134 but used for development and testing purposes), and non-production robots (similar to attended robots 132 but used for development and testing purposes). Attended robots 132 are triggered by user events and operate side by side with humans on the same computing system. Attended robots 132 may be used with the conductor 120 for centralized process deployment and logging media. Attended robots 132 may assist human users in accomplishing various tasks and may be triggered by user events. In some embodiments, processes cannot be initiated from the conductor 120 on this type of robot, and / or they cannot be run under a locked screen. In certain embodiments, the attended robot 132 can only be launched from the robot tray or from a command prompt. The attended robot 132 preferably operates under human supervision in some embodiments.

[0052] Unattended robots 134 operate unattended in virtual environments and can automate many processes. Unattended robots 134 can be responsible for providing remote execution, monitoring, scheduling, and work queue support. Debugging for all robot types can be performed in designer 110 in some embodiments. Both attended and unattended robots can automate a variety of systems and applications, including, but not limited to, mainframes, web applications, VMs, enterprise applications (e.g., those produced by SAP®, Salesforce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0053] The conductor 120 may have various capabilities, including, but not limited to, provisioning, deployment, versioning, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between robots 130 and the conductor 120 (e.g., web applications). Deployment may include ensuring the correct delivery of package versions to robots 130 assigned to perform. Versioning, in some embodiments, may include managing unique instances of some processes or configurations. Configuration may include maintaining and delivering robot environments and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot-specific data and maintaining user permissions. Logging may include storing and indexing logs in a database (e.g., an SQL database) and / or another storage mechanism (e.g., ElasticSearch®, which stores large data sets and provides the ability to quickly query them). The conductor 120 may provide interconnectivity by operating as a centralized point of communication for third-party solutions and / or applications.

[0054] Robots 130 are execution agents that execute workflows built by designer 110. One commercial example of some embodiments of robot(s) 130 is UiPath Robots™. In some embodiments, robots 130 install the Microsoft Windows Service Control Manager (SCM) management service by default. As a result, such robots 130 can open interactive Windows sessions under the local system account and may have Windows service rights.

[0055] In some embodiments, a robot 130 can be installed in user mode, meaning that for such a robot 130, the robot has the same rights as the user to whom it is installed. This feature can also be used for high-density (HD) robots, ensuring maximum utilization of each machine. In some embodiments, either type of robot 130 can be configured in an HD environment.

[0056] In some embodiments, the robot 130 is divided into multiple components, each specialized for a specific automation task. In some embodiments, the robot components include, but are not limited to, an SCM-managed robot service, a user-mode robot service, an executor, an agent, and a command line. The SCM-managed robot service manages and monitors Windows sessions and acts as a proxy between the conductor 120 and the execution host (i.e., the computing system on which the robot 130 executes). These services are responsible for managing credentials for the robot 130. A console application is launched by the SCM under Local System.

[0057] The user-mode robot service in some embodiments manages and monitors Windows sessions and acts as a proxy between the conductor 120 and the execution host. The user-mode robot service may be delegated and manage credentials for the robot 130. If the SCM management robot service is not installed, a Windows application may be launched automatically.

[0058] An Executor can execute a given job under a Windows session (i.e., execute a workflow). An Executor can be aware of dots per inch (DPI) settings per monitor. An Agent can be a Windows Presentation Foundation (WPF) application that displays available jobs in a system tray window. An Agent can be a client of a service. An Agent can ask to start or stop a job or change settings. A Command Line is a client of a service. A Command Line is a console application that can request the start of a job and wait for its output.

[0059] As described above, the separation of the robot 130 components helps developers, support users, and computing systems more easily implement, identify, and track what each component is doing. In this way, special behaviors can be configured for each component, such as setting different firewall rules for executors and services. Executors may always be aware of per-monitor DPI settings in some embodiments. As a result, workflows may execute at any DPI regardless of the configuration of the computing system on which the workflow was created. Also, in some embodiments, projects from the designer 110 may be made independent of the browser zoom level. For applications that are not DPI-aware or are intentionally marked as not-aware, some embodiments may disable DPI.

[0060] FIG. 2 is an architecture diagram illustrating a deployed RPA system 200 according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be or be part of the RPA system 100 of FIG. 1. It should be noted that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 210 includes an executor 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not be running on the computing system 210. The executor 212 executes processes. As shown in FIG. 2, multiple business projects may be running simultaneously. The agent 214 (e.g., a Windows service) is the single connection point for all executors 212 in this embodiment. All messages in this embodiment are logged to the conductor 230, which further processes them via the database server 240, the indexer server 250, or both. As described above with respect to FIG. 1, the executor 212 may be a robotic component.

[0061] In some embodiments, a Robot represents an association between a machine name and a username. A Robot may manage multiple executors simultaneously. In computing systems that support multiple interactive sessions running simultaneously (such as Windows Server 2012), multiple Robots may run simultaneously, each running in a separate Windows session using a unique username. This is referred to as an HD Robot above.

[0062] The agent 214 is also responsible for transmitting the robot's status (e.g., periodically sending "heartbeat" messages to indicate that the robot is still functioning) and downloading required versions of packages to be fulfilled. Communication between the agent 214 and the conductor 230 is, in some embodiments, always initiated by the agent 214. In notification scenarios, the agent 214 may open a WebSocket channel that is later used by the conductor 230 to send commands (e.g., start, stop, etc.) to the robot.

[0063] The server side includes a presentation layer (web application 232, Open Data Protocol (OData) Representational State Transfer (REST) ​​Application Programming Interface (API) endpoint 234, notification and monitoring 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240, indexer server 250). Conductor 230 includes web application 232, OData REST API endpoint 234, notification and monitoring 236, and API implementation / business logic 238. In some embodiments, most actions a user performs in the conductor 230 interface (e.g., via browser 220) are performed by calling various APIs. Such actions may include, but are not limited to, launching jobs on a robot, adding / removing data from a queue, scheduling jobs to run unattended, etc., without departing from the scope of the present invention. Web application 232 is the visual layer of the server platform. In this embodiment, web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the present invention. A user interacts with web pages from web application 232, in this embodiment via browser 220, to perform various operations to control conductor 230. For example, a user may create robot groups, assign packages to robots, analyze per-robot and / or per-process logs, start and stop robots, etc.

[0064] In addition to the web application 232, the conductor 230 also includes a services layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the present invention. The REST API is consumed by both the web application 232 and the agent 214, which in this embodiment is a supervisor of one or more robots on a client computer.

[0065] The REST API of this embodiment covers configuration, logging, monitoring, and queuing functionality. The configuration endpoint, in some embodiments, may be used to define and configure users, permissions, robots, assets, releases, and environments for an application. The logging REST endpoint may be used to log various information, such as errors, explicit messages sent by robots, and other environment-specific information. The deployment REST endpoint may be used by robots to query the version of the package that should be executed when a start job command is used in conductor 230. The queuing REST endpoint may be responsible for managing queues and queue items, such as adding data to a queue, retrieving transactions from a queue, and setting the status of transactions.

[0066] Monitoring REST endpoints may monitor the web application 232 and the agents 214. The notification and monitoring API 236 may be a REST endpoint used to register the agents 214, deliver configuration settings to the agents 214, and send and receive notifications from the server and the agents 214. The notification and monitoring API 236 may use WebSocket communication in some embodiments.

[0067] The persistence layer, in this embodiment, includes a pair of servers—a database server 240 (e.g., SQL Server) and an indexer server 250. The database server 240 in this embodiment stores configurations for robots, robot groups, associated processes, users, roles, schedules, etc. This information is managed, in some embodiments, via a web application 232. The database server 240 may also manage queues and queue items. In some embodiments, the database server 240 may also store messages logged by robots (in addition to or instead of the indexer server 250).

[0068] Optionally in some embodiments, indexer server 250 stores and indexes information logged by the robots. In particular embodiments, indexer server 250 may be disabled via a configuration setting. In some embodiments, indexer server 250 uses ElasticSearch®, a full-text search engine from an open source project. Messages logged by the robots (e.g., using activities such as log messages or line writes) may be sent via logging REST endpoint(s) to indexer server 250, where they are indexed for future use.

[0069] FIG. 3 is an architecture diagram illustrating the relationships 300 between a designer 310, activities 320, 330, a driver 340, and an AI / ML model 350, according to an embodiment of the present invention. As can be seen, a developer uses the designer 310 to develop a workflow to be performed by the robot. The workflow may include user-defined activities 320 and UI automation activities 330. The user-defined activities 320 and / or UI automation activities 330, in some embodiments, may be located locally and / or remotely relative to the computing system on which the robot is operating and may invoke one or more AI / ML models 350. In some embodiments, non-text visual components in an image may be identified, referred to herein as computer vision (CV). Some CV activities associated with such components may include, but are not limited to, click, type, get text, hover, detect presence or absence of element, update scope, highlight, etc. In some embodiments, clicking identifies an element and clicks on it, for example, using CV, optical character recognition (OCR), fuzzy text matching, and multi-anchor. Type may identify an element using the above and types within elements. Get text may locate specific text and scan it using OCR. Hover may identify an element and hover over it. Detect element presence may check to detect the presence or absence of an element on the screen using the techniques described above. In some embodiments, there may be hundreds or thousands of activities that can be implemented in designer 310. However, any number and / or types of activities may be utilized without departing from the scope of the present invention.

[0070] UI automation activities 330 are a subset of specialized low-level activities written in low-level code (e.g., CV activities) that facilitate interactions with a screen. UI automation activities 330 facilitate these interactions through drivers 340 and / or AI / ML models 350 that enable the robot to interact with desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc. One or more AI / ML models 350 may be used by UI automation activities 330 to determine the execution of interactions with the computing system. In some embodiments, AI / ML models 350 may augment or completely replace drivers 340. Indeed, in certain embodiments, drivers 340 are not included.

[0071] Drivers 340 may interact with the OS at a low level, such as by looking for hooks, monitoring keys, etc. They may facilitate integration with Chrome®, IE®, Citrix®, SAP®, etc. For example, a "click" activity plays the same role in these different applications via drivers 340.

[0072] FIG. 4 is an architecture diagram illustrating an RPA system 400, according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be or include the RPA systems 100 and / or 200 of FIGS. 1 and / or 2. The RPA system 400 includes multiple client computing systems 410 that execute robots. The computing systems 410 can communicate with a conductor computing system 420 via web applications running thereon. The conductor computing system 420 can, in turn, communicate with a database server 430 and an optional indexer server 440.

[0073] 1 and 3, it should be noted that although web applications are used in these embodiments, any suitable client / server software may be used without departing from the scope of the present invention. For example, a conductor may run a server-side application on a client computing system that communicates with a non-web-based client software application.

[0074] FIG. 5 is an architectural diagram illustrating a computing system 500 configured to perform RPA anomaly detection and self-healing through AI / ML, according to embodiments of the present invention. In some embodiments, computing system 500 may be one or more of the computing systems depicted and / or described herein. Computing system 500 includes a bus 505 or other communication mechanism for communicating information and processor(s) 510 coupled to bus 505 for processing information. Processor(s) 510 may be any type of general or application-specific processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. Processor(s) 510 may also have multiple processing cores, at least some of which may be configured to perform specific functions. In some embodiments, multiple parallel processing may be used. In certain embodiments, at least one processor(s) 510 may be a neuromorphic circuit including processing elements that mimic biological neurons. In some embodiments, neuromorphic circuits may not require typical components of a von Neumann computing architecture.

[0075] The computing system 500 further includes memory 515 for storing information and instructions executed by the processor(s) 510. The memory 515 may be comprised of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage such as a magnetic or optical disk, or other types of non-transitory computer-readable media, or any combination thereof. The non-transitory computer-readable media may be any available media that can be accessed by the processor(s) 510 and may include volatile media, non-volatile media, or both. Also, the media may be removable, non-removable, or both.

[0076] Additionally, the computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communication network via wireless and / or wired connections. In some embodiments, the communications device 520 may support any of the following radio technologies: Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiplexing (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile (GSM) communications, General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), cdma2000, Wideband CDMA (W-CDMA), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), High-Speed ​​Packet Access (HSPA), Long Term Evolution (LTE), LTE Advanced (LTE-A), LTE-Advanced (LTE-B), LTE-Advanced (LTE-C), LTE-Advanced (LTE-B), LTE-Advanced (LTE-C), LTE-Advanced (LTE-C), LTE-Advanced (LTE-C), LTE-Advanced (LTE-C), LTE-Advanced (LTE-C), LTE-Advanced (LTE-C), LTE-Advanced (LTE-C), LTE-Advanced (LTE-C), LTE-Advanced (LTE-A), LTE-Advanced (LTE-C ... Advanced), 802.11x, Wi-Fi, Zigbee, Ultra-Wideband (UWB), 802.16x, 802.15, Home Node-B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near-Field Communications (NFC), 5th Generation (5G), New Radio (NR), any combination thereof, and / or any other currently existing or future-implemented communication standard and / or protocol without departing from the scope of the present invention.In some embodiments, the communications device 520 may include one or more antennas that are a single antenna, an array of antennas, a phased antenna, a switched antenna, a beamforming antenna, a beamsteering antenna, a combination thereof, and / or any other antenna configuration without departing from the scope of the present invention.

[0077] The processor(s) 510 are further coupled via bus 505 to a display 525, such as a plasma display, a liquid crystal display (LCD), a light-emitting diode (LED) display, a field emission display (FED), an organic light-emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high-definition display, a Retina® display, an in-plane switching (IPS) display, or any other suitable display for displaying information to a user. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, or the like, using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signaling, acoustic pulse recognition, frustrated total internal reflection, or the like. Any suitable display device and haptic I / O may be used without departing from the scope of the invention.

[0078] A keyboard 530 and cursor control device 535, such as a computer mouse, touchpad, etc., are further coupled to bus 505 to allow a user to interface with computing system 500. However, in certain embodiments, a physical keyboard and mouse may not be present, and the user may interact with the device solely through display 525 and / or a touchpad (not shown). Any type and combination of input devices may be used as a matter of design choice. In certain embodiments, no physical input devices and / or displays are present. For example, a user may interact with computing system 500 remotely through another computing system in communication with it, or computing system 500 may operate autonomously.

[0079] Memory 515 stores software modules that provide functionality when executed by processor(s) 510. The modules include an operating system 540 for computing system 500. The modules further include an anomaly detection and self-healing module 545 configured to perform all or a portion of the processes described herein or derivatives thereof. Computing system 500 may include one or more additional functional modules 550 that include additional functionality.

[0080] Those skilled in the art will appreciate that a "system" may be embodied as a server, embedded computing system, personal computer, console, personal digital assistant (PDA), mobile phone, tablet computing device, quantum computing system, or any other suitable computing device or combination of devices without departing from the scope of the present invention. Presenting the above-described functions as being performed by a "system" is not intended to limit the scope of the present invention in any way, but rather to provide an example of many embodiments of the present invention. Indeed, the methods, systems, and apparatuses disclosed herein may be implemented in localized and distributed forms consistent with computing techniques, including cloud computing systems. The computing system may be part of or otherwise accessible through a local area network (LAN), a mobile communications network, a satellite communications network, the Internet, a public or private cloud, a hybrid cloud, a server farm, any combination thereof, or the like. Any local or distributed architecture may be used without departing from the scope of the present invention.

[0081] It should be noted that some of the system features described herein are presented as modules to further emphasize implementation independence. For example, a module may be implemented as a hardware circuit comprising custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, etc.

[0082] Modules may also be implemented at least partially in software for execution by various types of processors. For example, an identified unit of executable code may include one or more physical or logical blocks of computer instructions, which may be organized, for example, as an object, a procedure, or a function. Nevertheless, executable identified modules need not be physically located together; they may include separate instructions stored in different locations that, when logically combined, comprise a module to achieve the purpose stated for the module. Furthermore, modules may be stored on non-transitory computer-readable media, such as, for example, a hard disk drive, a flash device, RAM, tape, and / or any other non-transitory computer-readable medium used to store data without departing from the scope of the present invention.

[0083] Indeed, a module of executable code may be a single instruction, many instructions, or even distributed across several different code segments, different programs, and multiple memory devices. Similarly, operational data may be identified and depicted herein within a module, and may be embodied and organized in any suitable form within any suitable type of data structure. Operational data may be collected as a single data set, or may be distributed in different locations across different storage devices, or may exist, at least in part, simply as electronic signals on a system or network.

[0084] FIG. 6 is an architectural diagram illustrating a system 600 configured to perform RPA anomaly detection and self-healing through AI / ML, according to an embodiment of the present invention. System 600 includes user computing systems such as desktop computers 602, 603, a tablet 604, and a smartphone 606. However, any desired computing system, including, but not limited to, a smartwatch, a laptop computer, and the like, may be used without departing from the scope of the present invention. Also, while three user computing systems are shown in FIG. 6, any suitable number of computing systems may be used without departing from the scope of the present invention. For example, in some embodiments, tens, hundreds, thousands, or millions of computing systems may be used.

[0085] Each computing system 602, 604, 606 has an AI / ML-enabled RPA robot 610 that uses an AI / ML model, such as AI / ML model 632, to perform specific tasks. For example, the RPA robot 610 may use an AI / ML model trained to use CV to recognize what a user is doing on the respective computing system. The computing systems 602, 604, 606 send information to a server 630 via a network 620 (e.g., a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, any combination thereof, etc.). In some embodiments, the server 630 may be part of a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, etc. In particular embodiments, the server 630 may host multiple software-based servers on a single computing system 630. The server 630, in this embodiment, includes the AI / ML model 632 that is invoked by the RPA robot 610 to perform operations.

[0086] The RPA robot 610 includes anomaly handling and / or self-healing logic that attempts to automatically detect and correct anomalies. This logic may be built into the RPA robot 610 itself, the AI / ML model 632, or a combination thereof. Including the anomaly handling and / or self-healing logic as part of the AI / ML model 632 may enable leveraging information from multiple or many computing systems, potentially providing more examples, examples of how users overcame problems, etc.

[0087] The RPA robot 610 may occasionally encounter a new, unhandled anomaly where it cannot identify a graphical element and / or normal interactions do not work. Such anomalies may be handled by an anomaly handler (e.g., a subset of activities in the RPA robot's 610 workflow(s) or logic in the AI / ML model 632) that attempts to automatically handle the anomaly. If the target graphical element still cannot be identified by the anomaly handler, the RPA robot 610 can process the exception. The RPA robot 610 and / or the AI / ML model 632, in some embodiments, may have a self-healing capability that attempts to fill in missing data from the anomaly without user input.

[0088] In some embodiments, data regarding anomalies may be stored in database 640 for review by application 652 of training computing system 650, which allows a user to label and / or correct anomaly-related screens that failed to process the RPA robot's 610 self-correction attempts. For example, if a previously unseen scenario occurs, such as a new control not recognized by the AI / ML model, the user may be prompted for guidance. The user may then label the controls on the screen and potentially provide actions to take if the controls behave differently from previous controls the AI / ML model was trained to recognize. Using this training data, the AI / ML model 632 may be retrained to improve accuracy. In some embodiments, the AI / ML model invoked by the RPA robot 610 may have multiple layers performing various functions, such as statistical modeling (e.g., hidden Markov models (HMMs)), and may utilize deep learning techniques (e.g., long short-term memory (LSTM) deep learning, encoding previous hidden states, etc.) to identify sequences of user interactions.

[0089] AI layer

[0090] In some embodiments, multiple AI layers may be used. Each AI layer is an algorithm (or model) that runs on data, and the AI ​​model itself may be a deep learning neural network (DLNN) of artificial "neurons" trained on training data. Layers may run in serial, parallel, or a combination thereof.

[0091] AI layers may include, but are not limited to, a sequence extraction layer, a clustering detection layer, a visual component detection layer, a text recognition layer (e.g., OCR), a speech-to-text translation layer, or any combination thereof. However, any desired number and type(s) of layers may be used without departing from the scope of the present invention. Using multiple layers allows the system to develop a global picture of what is happening on the screen. For example, one AI layer may perform OCR, another may detect buttons, another may compare sequences, etc. Patterns may be determined individually by one AI layer or collectively by multiple AI layers.

[0092] 7 is a flowchart illustrating a process 700 for performing RPA anomaly detection and self-healing via AI / ML, according to an embodiment of the present invention. The process begins at 705 with executing an RPA robot (and thus an RPA workflow) that performs UI automation using an AI / ML model. Using the AI / ML model, the RPA robot searches for a target graphical element in the UI that will be interacted with by an activity in the RPA workflow at 710. Once the target graphical element is uniquely found by the AI / ML model at 715, the RPA robot interacts with the target graphical element according to one or more RPA activities in the RPA workflow at 720. The RPA robot then continues executing the RPA workflow logic at 725 until it needs to interact with the next graphical element. The process then returns to step 710.

[0093] However, if the target graphical element cannot be uniquely found or interacted with by the AI / ML model at 715, the RPA robot or AI / ML model automatically attempts to correct the anomaly at 730. In some embodiments, the automatic attempt to correct the anomaly includes determining whether one or more characteristics distinguish the target graphical element from other similar graphical elements. In particular embodiments, determining whether one or more characteristics distinguish the target graphical element from other similar graphical elements includes analyzing graphical elements that surround the target graphical element within a radius, utilizing the order of the graphical elements in the UI, determining whether the target graphical element has one or more different visual characteristics, or a combination thereof. In some embodiments, the automatic attempt to correct the anomaly includes modifying visual characteristics of the UI and searching for the target graphical element in the UI using the modified visual characteristics. In particular embodiments, the automatic attempt to correct the anomaly includes taking a default action. In some embodiments, the default action may be or include searching a UI object library or UI object repository for one or more UI descriptors that enable the desired interaction with the target graphical element.

[0094] In some embodiments, the automatic attempt to correct the anomaly includes the RPA robot or AI / ML model attempting a self-healing process to fill in the missing data without user input. This may be achieved, for example, through an exploration phase in reinforcement learning. In particular embodiments, the RPA robot or AI / ML model is configured to determine whether the self-healing process is successful by monitoring whether one or more performance metrics improve in response to the self-healing process. In some embodiments, if the self-healing process is unsuccessful, the automatic attempt to correct the anomaly includes trying one or more different techniques and monitoring whether the one or more different techniques improve the one or more performance metrics. In particular embodiments, the self-healing process includes polling multiple users to provide proposed solutions to the anomaly and selecting a most optimal solution from the proposed solutions based on one or more performance metrics. For example, the most optimal solution may be selected based on speed, increased revenue, creation of more customer accounts, a combination thereof, etc.

[0095] If the automatic attempt to correct the anomaly is successful at 735, data regarding the automatic correction is provided (e.g., sent to a remote server and database, such as server 630 and database 640 in FIG. 6 ) for subsequent retraining of the AI / ML model at 740, and the process proceeds to step 720 and continues as described above. However, if the automatic attempt to correct the anomaly is not successful at 745, the user is prompted for a solution to identify the target graphical element, a method for interacting with the target graphical element, or both. For example, if a never-before-seen scenario occurs, such as a new control not recognized by the AI / ML model, the user can be prompted for guidance. The user may then label the control on the screen and potentially provide an action to take if the control behaves differently from a previous control the AI / ML model was trained to recognize. If the guidance provided by the user is successful in allowing the RPA robot to interact with the target graphical element, data regarding the user solution is provided at 740 for subsequent retraining of the AI / ML model, and the process proceeds to step 720 and continues as described above.

[0096] If the guidance provided by the user is not successful in enabling the RPA robot to interact with the target graphical element at 750, and the target graphical element is not necessary to complete the overall task of the RPA workflow at 755 and continued operation is possible, the RPA robot may continue performing the RPA workflow at 725. This may be possible, for example, if the target graphical element is not necessary to complete the overall task with sufficient accuracy. However, if continuing operation is not possible at 755, an exception is thrown at 760 and the process ends.

[0097] The process steps performed in FIG. 7 may be performed by a computer program encoding instructions to a processor(s) to perform at least a portion of the process(es) described in FIG. 7 according to an embodiment of the present invention. The computer program may be stored on a non-transitory computer-readable medium. The computer-readable medium may be, but is not limited to, a hard disk drive, a flash device, RAM, tape, and / or any other such medium or combination of media used to store data. The computer program may include coded instructions for controlling a processor(s) of a computing system (e.g., processor(s) 510 of computing system 500 of FIG. 5) to implement all or a portion of the process steps described in FIG. 7, which may also be stored on a computer-readable medium.

[0098] The computer program may be implemented in hardware, software, or a hybrid implementation. The computer program may be composed of modules in operable communication with each other and designed to send information or instructions to a display. The computer program may be configured to run on a general-purpose computer, an ASIC, or any other suitable device.

[0099] It will be readily understood that the components of the various embodiments of the present invention, as generally described and illustrated herein, may be arranged and designed in a wide variety of different configurations. Thus, the detailed description of the embodiments of the present invention, as represented in the accompanying figures, is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention.

[0100] The features, structures, or characteristics of the invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, references throughout this specification to "certain embodiments," "some embodiments," or similar language mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Thus, the appearances of "certain embodiments," "some embodiments," "other embodiments," or similar language throughout this specification do not necessarily refer to the same group of all embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0101] It should be noted that references to features, advantages, or similar language throughout this specification do not imply that all of the features and advantages that may be realized in the present invention are to be found in any single embodiment of the present invention, or in any embodiment of the present invention. Rather, language referring to features and advantages is understood to mean that the particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, discussions of features and advantages throughout this specification, and similar language, may, but do not necessarily, refer to the same embodiment.

[0102] Furthermore, the described features, advantages, and characteristics of the invention may be combined in any suitable manner in one or more embodiments. Those skilled in the relevant art will recognize that the invention may be practiced without a particular feature or advantage of one or more particular embodiments. In other instances, additional features and advantages may be recognized in particular embodiments, although they may not be present in all embodiments of the invention.

[0103] Those of ordinary skill in the art will readily appreciate that the invention as described above can be implemented using steps in a different order and / or with hardware elements in different configurations than those disclosed. Thus, while the invention has been described in terms of these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative configurations will become apparent while remaining within the spirit and scope of the invention. Accordingly, reference should be made to the appended claims to determine the scope of the invention.

Claims

1. Executing a Robotic Process Automation (RPA) workflow by an RPA robot, which performs user interface (UI) automation using an artificial intelligence (AI) / machine learning (ML) model; searching, by the RPA robot, for a target graphical element in the UI that is to be interacted with by an activity of the RPA workflow using the AI / ML model; and if the target graphical element cannot be uniquely found or interacted with by the AI / ML model, automatically attempting to correct the anomaly by the RPA robot or the AI / ML model; The computer-implemented method, wherein the automatic attempt to correct the anomaly includes modifying visual characteristics of the UI and searching for the target graphical element within the UI using the modified visual characteristics.

2. The computer-implemented method of claim 1 , wherein automatically attempting to correct the anomaly includes determining whether one or more characteristics distinguish the target graphical element from other similar graphical elements.

3. 3. The computer-implemented method of claim 2, wherein determining whether the one or more characteristics distinguish the target graphical element from the other similar graphical elements comprises analyzing graphical elements that surround the target graphical element within a radius, utilizing an order of the graphical elements in the UI, determining whether the target graphical element has one or more different visual characteristics, or a combination thereof.

4. The computer-implemented method of claim 1 , wherein the automatic attempt to correct the anomaly includes taking a default action.

5. The computer-implemented method of claim 4 , wherein the default action includes searching a UI object library or UI object repository for one or more UI descriptors that enable interaction with the target graphical element.

6. If the automatic attempt to correct the anomaly is successful, the method further comprises:

10. The computer-implemented method of claim 1, comprising providing, by the RPA robot, data related to automatic correction for subsequent retraining of the AI / ML model.

7. If the automatic attempt to correct the anomaly is unsuccessful, the method further comprises:

10. The computer-implemented method of claim 1, comprising prompting a user, by the RPA robot, for a solution to identify the target graphical element, a method to interact with the target graphical element, or both.

8. When the guidance provided by the user is successful in enabling the RPA robot to interact with the target graphical element, the method further comprises:

10. The computer-implemented method of claim 7, comprising providing, by the RPA robot, data regarding user solutions for subsequent retraining of the AI / ML model.

9. If the guidance provided by the user is not successful in enabling the RPA robot to interact with the target graphical element, and the target graphical element is not necessary to complete an overall task of the RPA workflow, the method further comprises: The computer-implemented method of claim 7 , comprising attempting to continue performance of the RPA workflow by the RPA robot.

10. The automatic attempt to correct the anomaly includes:

2. The computer-implemented method of claim 1, comprising attempting a self-repair process by the RPA robot or the AI / ML model to complete missing data without user input.

11. 11. The computer-implemented method of claim 10, wherein the RPA robot or the AI / ML model is configured to determine whether the self-repair process is successful by monitoring whether one or more performance metrics improve in response to the self-repair process.

12. If the self-repair process is unsuccessful, the method further comprises: The computer-implemented method of claim 10 , comprising trying one or more different techniques and monitoring whether the one or more different techniques improve the one or more performance metrics.

13. The self-repair process comprises: polling a plurality of users to provide suggested solutions to the anomaly; and and selecting a most optimal solution from the proposed solutions based on one or more performance metrics.

14. The computer-implemented method of claim 10 , wherein the attempt of the self-repairing process to complete the missing data without user input comprises using an exploration phase in reinforcement learning.

15. A non-transitory computer-readable medium having stored thereon a computer program, the computer program causing at least one processor to: Performing robotic process automation (RPA) workflows that use artificial intelligence (AI) / machine learning (ML) models to perform user interface (UI) automation; using the AI / ML model to find target graphical elements in the UI that are interacted with by activities of the RPA workflow; configured to automatically attempt to correct an anomaly if the AI / ML model is unable to uniquely find or interact with the target graphical element; A non-transitory computer-readable medium, wherein the automatic attempt to correct the anomaly includes modifying visual characteristics of the UI and searching for the target graphical element within the UI using the modified visual characteristics.

16. 16. The non-transitory computer-readable medium of claim 15, wherein the automatic attempt to correct the anomaly includes analyzing graphical elements surrounding the target graphical element within a radius, utilizing an order of the graphical elements in the UI, and determining whether the target graphical element has one or more different visual characteristics.

17. when the automatic attempt to correct the anomaly is successful, the computer program is further configured to cause the at least one processor to provide data regarding the automatic correction for subsequent retraining of the AI / ML model; 16. The non-transitory computer-readable medium of claim 15, wherein if the automatic attempt to correct the anomaly is unsuccessful, the computer program is further configured to cause the at least one processor to prompt a user with solutions for identifying the target graphical element, ways to interact with the target graphical element, or both.

18. when the guidance provided by the user is successful in enabling the computer program to interact with the target graphical element, the computer program is further configured to: cause the at least one processor to provide data regarding the user solution for subsequent retraining of the AI / ML model; 18. The non-transitory computer-readable medium of claim 17, wherein when the guidance provided by the user is not successful in enabling the computer program to interact with the target graphical element and the target graphical element is not necessary to complete an overall task of the RPA workflow, the computer program is further configured to cause the at least one processor to attempt to continue performing the RPA workflow.

19. 16. The non-transitory computer-readable medium of claim 15, wherein automatically attempting to correct the anomaly comprises attempting a self-healing process to fill in missing data without user input.

20. a memory for storing computer program instructions; and at least one processor configured to execute the computer program instructions, the computer program instructions causing the at least one processor to: The RPA robot executes a robotic process automation (RPA) workflow that uses an artificial intelligence (AI) / machine learning (ML) model to perform user interface (UI) automation; using the AI / ML model to search for target graphical elements in the UI that are to be interacted with by activities of the RPA workflow; if the target graphical element cannot be uniquely found or interacted with by the AI / ML model, then the RPA robot or the AI / ML model is configured to automatically attempt to correct an anomaly; automatically attempting to correct the anomaly includes determining whether one or more characteristics distinguish the target graphical element from other similar graphical elements, attempting a self-healing process to complete missing data without user input, or both; The computing system, wherein the automatic attempt to correct the anomaly includes modifying visual characteristics of the UI and searching for the target graphical element within the UI using the modified visual characteristics.

21. 21. The computing system of claim 20, wherein determining whether the one or more characteristics distinguish the target graphical element from the other similar graphical elements comprises analyzing graphical elements that surround the target graphical element within a radius, utilizing an order of the graphical elements in the UI, determining whether the target graphical element has one or more different visual characteristics, or a combination thereof.

Images