Robotic process automation anomaly detection and self-repair via artificial intelligence / machine learning

Abstract

Anomaly detection and self-healing for Robotic Process Automation (RPA) via Artificial Intelligence (AI) / Machine Learning (ML) are disclosed. RPA robots utilizing AI / ML models and Computer Vision (CV) can interpret and interact with most encountered graphical elements through normally learned interactions. However, such RPA robots may occasionally encounter new, unprocessed anomalies, where graphical elements cannot be identified and / or normal interactions will not function. Such anomalies can be handled by an anomaly processor. The RPA robot can possess self-healing functionality that attempts to automatically find information to resolve the anomaly.

Description

[0001] Cross-reference to related applications

[0002] This international application claims the interest and priority of U.S. Patent Application No. 17 / 070,250, filed October 14, 2020. The subject matter of that previously filed application is incorporated herein by reference in its entirety. Technical Field

[0003] This invention relates generally to user interface (UI) automation, and more specifically, to anomaly detection and self-healing for robotic process automation (RPA) via artificial intelligence (AI) / machine learning (ML). Background Technology

[0004] Currently, when an exception occurs in runtime RPA (e.g., in a production environment), the source of the problem is analyzed, and RPA developers provide a programmatic solution based on that analysis. However, this process tends to be time-consuming and expensive. Therefore, an improved approach could be beneficial. Summary of the Invention

[0005] Some embodiments of the present invention can provide solutions to problems and needs in the prior art that have not been fully identified, understood, or resolved by current UI automation technologies. For example, some embodiments of the present invention relate to anomaly detection and self-healing for RPA via AI / ML.

[0006] In one embodiment, the computer-implemented method includes an RPA workflow executed by an RPA robot to automate the UI using an AI / ML model. The computer-implemented method also includes the RPA robot using the AI / ML model to search for target graphical elements in the UI that the activities of the RPA workflow want to interact with. If a target graphical element is not uniquely found by the AI / ML model or cannot be interacted with, the RPA robot or the AI / ML model automatically attempts to correct the anomaly.

[0007] In another embodiment, a non-transitory computer-readable medium stores a computer program. This computer program is configured to cause at least one processor to execute an RPA workflow that performs UI automation using an AI / ML model. Using the AI / ML model, the computer program is also configured to cause at least one processor to search within the UI for target graphical elements that the activities of the RPA workflow want to interact with. If a target graphical element is not uniquely found by the AI / ML model or cannot be interacted with, the computer program is also configured to automatically attempt to correct the anomaly.

[0008] In another embodiment, a computing system includes a memory storing computer program instructions and at least one processor configured to execute the computer program instructions. The computer program instructions are configured to cause the at least one processor, by an RPA bot, to execute an RPA workflow that automates the UI using an AI / ML model. Using the AI / ML model, the computer program instructions are also configured to cause the at least one processor, by the RPA bot, to search within the UI for target graphical elements that activities within the RPA workflow want to interact with. When a target graphical element is not uniquely found by the AI / ML model or cannot be interacted with, the computer program instructions are further configured to cause the at least one processor to automatically attempt to correct the anomaly. This automatic attempt to correct the anomaly includes determining whether one or more features distinguish the target graphical element from other similar graphical elements, attempting a self-repair process to complete missing data in the absence of user input, or both. Attached Figure Description

[0009] To facilitate understanding of the advantages of certain embodiments of the present invention, a more detailed description of the invention, briefly described above, is provided with reference to the specific embodiments shown in the accompanying drawings. Although it should be understood that these drawings depict only typical embodiments of the invention and should not be considered as limiting the scope of the invention, the invention is further described and explained with additional specific details using the drawings, wherein:

[0010] Figure 1 This is an illustration of the architecture of a robotic process automation (RPA) system according to an embodiment of the present invention.

[0011] Figure 2 This is an illustration of the architecture of a deployed RPA system according to an embodiment of the present invention.

[0012] Figure 3 This is an architecture diagram illustrating the relationship between the designer, activities, and drivers according to an embodiment of the present invention.

[0013] Figure 4 This is an illustration of the architecture of an RPA system according to an embodiment of the present invention.

[0014] Figure 5 The diagram illustrates the architecture of a computing system according to an embodiment of the present invention, which is configured to perform anomaly detection and self-repair for RPA via AI / ML.

[0015] Figure 6 The diagram illustrates the architecture of a system according to an embodiment of the present invention, which is configured to perform anomaly detection and self-healing for RPA via AI / ML.

[0016] Figure 7This is a flowchart illustrating an anomaly detection and self-repair process for RPA via AI / ML according to an embodiment of the present invention.

[0017] Unless otherwise stated, similar reference numerals always denote corresponding features in all the accompanying drawings. Detailed Implementation

[0018] Some embodiments involve anomaly detection and self-healing for RPA via AI / ML. RPA bots utilizing AI / ML models and computer vision (CV) can interpret and interact with most graphical elements encountered in a UI through normally learned interactions. However, such RPA bots may occasionally encounter new, unhandled anomalies where graphical elements cannot be identified and / or normal interactions will not function. In some embodiments, such anomalies can be handled by the RPA bot's anomaly handler or AI / ML model.

[0019] When an anomaly occurs, such as when multiple submit buttons appear on the screen and the AI / ML model cannot determine which button to click on its own, the anomaly handler can attempt to determine if something distinguishes these graphic elements from each other. This may include, but is not limited to, examining the graphic elements surrounding the target graphic element(s) within a radius, examining the order of the graphic elements on the screen, and attempting to determine if something visually distinguishes the target graphic element from other similar graphic elements. In some embodiments, a default action may be taken, such as automatically selecting the graphic element that appears closest to the top or bottom of the screen.

[0020] In some embodiments, the default action may be to search for a UI descriptor in a UI object library, which includes a UI object repository of UI objects or a UI object storehouse of UI objects, that allows an RPA bot or AI / ML model to interact with a target graphical element. UI elements on a screen (e.g., text fields, buttons, labels, menus, checkboxes, etc.) may be grouped by application, application version, application screen, and set of UI elements, with each screen having multiple UI elements. As used herein, a “screen” is an image of an applied UI at a given point in time or an image of a portion of an applied UI (e.g., a screenshot). For a running UI automation, a screen may be the current UI of a computing system. An “application” or a version of a given application may be a union of screens in that context or an application running at runtime. In some embodiments, each UI element may be described by one or more UI descriptors. UI elements, UI descriptors, applications, and application screens are UI objects. In some embodiments, UI elements and screens may be further distinguished by specific types of UI elements (e.g., buttons, checkboxes, text fields, etc.) and specific types of screens (e.g., top window, modal window, pop-up window, etc.).

[0021] In some embodiments, the UI object library can be stored in a UI object repository. As used herein, a UI object repository is a collection of UI objects. In some embodiments, the UI object repository may be a NuGet library. TM Package feeds, web services, etc. In some embodiments, the UI object repository of the UI object library can facilitate improved management, reuse, and reliability of UI descriptors within a project. In some embodiments, UI descriptors can be added to the UI library and published or redistributed for global reuse within the UI object repository.

[0022] To make UI objects reusable, they can be extracted into a UI object library that can be referenced by RPA processes. For example, when a selector or other UI descriptor is modified due to a new version of the application, the UI object library can be creatively created (or redistributed) to include the modified UI descriptor. The RPA process using the UI object library can then invoke the modified version of the UI descriptor. In some embodiments, the reference to the new application programming interface (API) associated with the new library can be manually modified by the developer for the RPA process, modified via software tools (e.g., tools that update UI descriptor references via the RPA process when selectors or other components change), or modified automatically.

[0023] In some embodiments, the package of a UI object (e.g., NuGet) TMThe package can be attached as a dependency. However, in some embodiments, the UI object can be obtained via one or more API calls to the web service. This allows the UI object to be stored, retrieved, and used remotely during automation.

[0024] UI elements added to the UI object library can represent selectors used in RPA workflows. To automate specific actions within the UI, RPA bots can interact with various windows, buttons, dropdown lists, and / or other graphical elements. Traditionally, this is done by using the expected screen position of the UI element. However, this is not reliable.

[0025] Some embodiments use selectors that overcome the problems associated with fixed coordinate identifiers by storing the attributes of UI elements and their parents in XML fragments. While in some embodiments where the UI is static, selectors can be generated automatically, some software programs, such as some web applications, have changing layouts and attribute nodes with unstable values. These changes may not be easily predictable, and this previously required the manual generation of some selectors. However, some embodiments can overcome this problem by using other types of UI descriptors, for example.

[0026] A selector is a type of UI descriptor that, in some embodiments, can be used to detect UI elements. In some embodiments, a selector has the following structure:

[0027] <node_1 / ><node_2 / > …<node_N / >

[0028] The last node represents the GUI element of interest, and all preceding nodes represent the parent of that element.<node_1 / > It often refers to the root node and represents the top window of the application.

[0029] Each node may have one or more attributes that assist in the correct identification of a specific level of the selected application. In some embodiments, each node has the following format:

[0030] <ui_system attr_name_1=’attr_value_1’…attr_name_N=’attr_value_N’ / >

[0031] Each attribute can have an assigned value, and attributes with unchanging values ​​can be selected. This is because changes to attribute values ​​each time the application begins can cause the selector to fail to correctly identify associated elements.

[0032] UI object library descriptors can be directly added to RPA workflow activities to save developer time that might otherwise be required to create custom selectors for that activity. The object library is defined herein as a collection of UI descriptors corresponding to one or more screens from a particular version of the application. A UI descriptor is a set of instructions for finding UI elements. In some embodiments, a UI descriptor is an encapsulated data / structure format that includes(multiple) UI element selectors,(multiple) anchor selectors,(multiple) computer vision (CV) descriptors,(multiple) unified target descriptors, screen image captures (context), element image captures, other metadata (e.g., application and application version), combinations thereof, etc. The encapsulated data / structure format can be extensible to future platform updates and is not limited to the above limitations. Any suitable UI descriptor for identifying UI elements on a screen can be used without departing from the scope of the invention. A unified target descriptor links multiple types of UI descriptors together. A unified target descriptor can function like a finite state machine (FSM), where a first UI descriptor mechanism is applied in a first context, a second UI descriptor in a second context, and so on.

[0033] Traditionally, making selectors reliable requires testing, understanding the inherent details of the selectors, and so on. This makes creating / improving selectors difficult or impossible for individuals without technical knowledge. By providing libraries of selectors for common UI elements and tasks, some implementations allow non-technical individuals to develop automation for production tasks. A user can instruct a version of the application's screen to automate, and the RPA development application can access that screen. The user can then interact with the screen to define what he or she wants to do.

[0034] In some embodiments, as new UI descriptors are created and / or existing UI descriptors are modified, a global database of UI objects can be established. This global database is shareable, collaborative, and potentially open-source. In some embodiments, taxonomy and ontology can be used. Applications, versions, screens, UI elements, descriptors, etc., can be defined as taxonomic categories, which are hierarchical structures of subcategories.

[0035] However, many real-world concepts are not easily adapted to taxonomic organization. Instead, they may be closer to concepts in mathematical ontology. In ontology, the relationships between categories do not necessarily have to be hierarchical. For example, the scenario where a button on a screen, when clicked, takes the user to another screen cannot be easily captured by the taxonomy of that screen because the next screen is not in a hierarchy. In some embodiments, interactions between UI elements on the same screen or on different screens can be adapted, and information about how UI elements relate to each other (potentially on different screens) can be provided.

[0036] Consider the example of clicking a confirmation button leading to an employee screen. An ontological structure allows the designer application to suggest employees to the user on the next screen. Ontological information about the relationships between these screens is provided by the confirmation button, enabling the designer application to do so. By defining a graphical structure (which doesn't necessarily have to be tree-like, but is relevant to what the application is actually doing), more complex and richer relationships can be captured.

[0037] UI descriptors can be extracted from activities in an RPA workflow and added to a structured architecture that groups UI descriptors by UI application, screen, and UI element. In some embodiments, a UI descriptor may be part of a project for broad reuse, part of a global repository for testing purposes, or part of a UI object library for global sharing across projects. The object library, in some embodiments, is a wrapper around UI descriptors grouped by application, application version, and screen. In some embodiments, UI descriptors may be defined and added to the object library, which may be installed as a dependency in other projects after deployment. This makes the object library of UI descriptors reusable. The UI application, in some embodiments, is an application targeted with multiple versions, each with multiple screens.

[0038] For various reasons, UI object repositories and UI descriptor reuse can be advantageous. For example, if the location, appearance, and / or functionality of a UI element in an application changes, the UI descriptor can be modified, and the change can then be propagated to activities using the UI descriptor. Therefore, reusability can be provided at the level of identifying graphical elements in the UI.

[0039] This can be particularly advantageous for adapting to new versions of an application. For new versions, software developers may only need to update the UI descriptors and / or develop a few new selectors from scratch, significantly reducing development time. For example, for a webpage, Hypertext Markup Language (HTML) can be inspected to obtain the path to the UI element corresponding to the selector. The selector for the corresponding UI element can be different in different web browsers and / or in different versions of the same web browser. This concept can also be applied to visual desktops, servers, smartphones, and tablet applications. A UI object library can contain actual selectors for an application; based on the above, actual selectors are just one type of UI descriptor. A UI object library can have multiple UI descriptors of different types for one or more applications. UI descriptors can also refer to the different technologies used to build the application. For example, different UI descriptors can be created and used for standalone desktop, web, and mobile applications. For one technology, you might want to use a selector; for another technology, you might want to use a CV descriptor, and so on.

[0040] UI descriptors can work with the Unified Target, which encompasses all UI detection mechanisms through which image detection and qualifying can be performed in some embodiments. The Unified Target can combine multiple techniques for identifying and automating UI elements into a single cohesive approach. The Unified Target can prioritize selector-based and driver-based UI detection mechanisms and fall back to CV to find an image if the first two mechanisms fail. In some embodiments, the selector editor and UI browser can be configured to support the Unified Target.

[0041] The following provides an example of an object library structure:

[0042] SAP

[0043] Version 1

[0044] Screen 1

[0045] ·Object 1

[0046] ·Object 2

[0047] ·…

[0048] Screen 2…

[0049] ·…

[0050] Version 2…

[0051] ·……

[0052] Salesforce…

[0053] ·…

[0054] It should be noted that the above hierarchy is provided only as an example. Any desired number of levels and associated elements can be used within the hierarchy without departing from the scope of the invention. In some embodiments, the user can define the application map as he or she wishes. Furthermore, in some embodiments, some nodes in the tree may only play a managerial role and not a functional role. For example, UI elements may be grouped together into containers without UI descriptors. In some embodiments, the containers may exist solely for grouping purposes.

[0055] To utilize UI object libraries and / or UI object repositories, some embodiments of the RPA bot or AI / ML model, when a target graphical element cannot be identified, default to searching the UI object library or UI object repository for UI descriptors that can interact with the target graphical element. For example, the RPA bot or AI / ML model can search for UI descriptors using the same name, the same or similar attributes, or the same but different versions of the target application. If such UI descriptors(s) are found, the RPA bot or AI / ML model can automatically try(s) the UI descriptors(s). If interaction with the target graphical element is successful, the RPA bot or AI / ML model can be modified in the future to use the successful UI descriptors, appending or replacing previously tried UI descriptors. The UI object library or UI object repository can be local or remote on the RPA bot or AI / ML model searching for them.

[0056] If the exception handler still cannot identify the target graphical element and / or the default action fails, the RPA bot can gracefully handle exceptions. For example, if the target graphical element is not required to complete the overall task with sufficient accuracy, the exception handler can allow the RPA bot to continue its workflow, prompt the user whether to continue, or query labeled training data to be used in the future to resolve the exception. Such exception handling makes AI / ML models more robust to real-world differences and changes at runtime. This can occur when a new version of the application is released, or when encountering different browser settings, etc.

[0057] In some embodiments, self-healing capabilities can be provided so that RPA bots and / or AI / ML models attempt to complete missing data from anomalies without user input. RPA developers can provide certain inputs to AI / ML model variables, pull data from a database, etc. In some embodiments, AI tools can be used to generate the code without human design time.

[0058] In some embodiments, self-healing can leverage various techniques. For example, for document understanding or other applications, if human verification is frequently required, it can be inferred that the AI / ML model is not good enough. The AI / ML model can utilize various techniques to attempt to improve its own performance as a "self-healing" measure. For example, the AI / ML model can try different optical character recognition (OCR) engines, modify the properties of images (e.g., brightness, hue, contrast, conversion to grayscale, etc.), search for images that look similar, and examine the techniques(s) applied to these images, and / or the results. The AI / ML model can monitor whether human verification effort is reduced, whether the speed and / or efficiency of execution is increased (e.g., the process runs faster and / or steps in the RPA workflow can be accelerated or eliminated), or whether the return on investment (ROI) is improved to determine whether the AI / ML model's self-healing effect is working. If not, the AI / ML model can try different techniques and / or use different information. The AI / ML model can then learn how to complete missing, incorrect, and / or incomplete data at runtime based on this self-healing method.

[0059] In some embodiments, reinforcement learning can be used. Reinforcement learning includes an exploration phase and an exploitation phase. During the exploration phase, the system may attempt to find solutions randomly from a start state to an end state using a reward function to guide the search of the state space. The reward function explores intermediate transitions and steps through both short-term and long-term rewards. For example, in cases where an anomaly occurs—such as the inability to find a target graphical element in the UI or a failure to interact with the target graphical element—the winning state could be the correct interaction with the target graphical element that the activity in the RPA workflow should achieve. If a solution is found, the RPA bot or AI / ML model can be modified to try that solution in addition to or as a replacement for previous methods.

[0060] Some embodiments determine what the policy should be via a policy network, rather than performing completely random exploration. This drastically reduces the number of simulations required to reach a winning state, enabling AI / ML models trained via reinforcement learning to be developed and deployed faster, trained with fewer computational hardware resources, or both. In fact, in some embodiments, reinforcement learning can be performed on the user's own computing system (e.g., desktop computer, laptop computer, smartphone, etc.), which is traditionally not possible.

[0061] For example, in the context of interacting with a user interface, a robot can randomly tap on the screen to determine what is clickable. Reinforcement learning can then be used to advance the desired automation. For instance, given a spreadsheet of email addresses and the desired email text, reinforcement learning can be used to discover which clickable button opens the email application, where different text should be entered in the email, and the button to click to send the email. In this example, a policy network can be used to first determine whether sending the email is even possible or desirable. Once that determination is made, the policy network can then determine whether the email address comes from the spreadsheet and whether the email body text can be copied into the email before it is sent.

[0062] Some implementations can be used for robotic process automation (RPA). Figure 1 This is an architectural diagram illustrating an RPA system 100 according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that allows developers to design and implement workflows. The designer 110 can provide solutions for application integration and automation of third-party applications, management of information technology (IT) tasks, and business IT processes. The designer 110 can facilitate the development of automation projects, which are graphical representations of business processes. In short, the designer 110 facilitates the development and deployment of workflows and robots.

[0063] Automation projects automate rule-based processes by allowing developers to control the execution order and the relationships between a custom set of steps developed within the workflow (defined as "activities" in this document). A commercial example of an embodiment of Designer 110 is UiPath Studio. TM Each activity may include actions such as clicking a button, reading a file, or writing to a log panel. In some embodiments, workflows may be nested or embedded.

[0064] Some types of workflows may include, but are not limited to, sequences, flowcharts, flow charts (FSMs), and / or global exception handlers. Sequences may be particularly suitable for linear processes, implementing the flow from one activity to another without compromising the workflow. Flowcharts may be particularly suitable for more complex business logic, integrating decisions and connecting activities in more diverse ways through multiple branching logic operators. FSMs may be particularly suitable for large workflows. FSMs can use a limited number of states in their execution, triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suitable for determining workflow behavior when execution errors are encountered and are suitable for debugging processes.

[0065] Once the workflow is developed in designer 110, the execution of the business process is orchestrated by coordinator 120, which in turn orchestrates one or more robots 130 to execute the workflow developed in designer 110. A commercial example of an embodiment of coordinator 120 is the UiPath Orchestrator. TM The coordinator 120 facilitates the management of the creation, monitoring, and deployment of resources in the environment. The coordinator 120 can act as an integration point or aggregation point for third-party solutions and applications.

[0066] Coordinator 120 can manage the queue of robots 130, connecting and executing robots 130 from a centralized point. The types of robots 130 that can be managed include, but are not limited to, manned robots 132, unattended robots 134, development robots (similar to unattended robots 134 but used for development and testing purposes), and non-production robots (similar to manned robots 132 but used for development and testing purposes). Manned robots 132 are triggered by user events and operate alongside humans on the same computing system. Manned robots 132 can be used with coordinator 120 for centralized process deployment and logging media. Manned robots 132 can assist human users in performing various tasks and can be triggered by user events. In some embodiments, processes cannot be initiated from coordinator 120 on this type of robot and / or they cannot run under a locked screen. In some embodiments, manned robots 132 can only be initiated from a robot tray or from a command prompt. In some embodiments, manned robots 132 should operate under human supervision.

[0067] Unattended robot 134 operates unattended in a virtual environment and can automate multiple processes. Unattended robot 134 can be responsible for remote execution, monitoring, scheduling, and supporting work queues. In some embodiments, debugging for all robot types can be run in designer 110. Both manned and unattended robots can automate various systems and applications, including but not limited to mainframes, web applications, virtual machines, and enterprise applications (e.g., by...). Applications that are produced (such as those in production) and computing system applications (e.g., desktop and laptop computer applications, mobile device applications, wearable computer applications, etc.).

[0068] The coordinator 120 may have various capabilities, including but not limited to provisioning, deployment, version control, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining a connection between the robot 130 and the coordinator 120 (e.g., a web application). Deployment may include ensuring that packaged versions are correctly delivered to the assigned robot 130 for execution. In some embodiments, version control may include the management of unique instances of some processes or configurations. Configuration may include the maintenance and delivery of robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include maintaining the tracking of robot identification data and maintaining user permissions. Logging may include storing logs in a database (e.g., an SQL database) and / or other storage mechanisms (e.g., ...). It also indexes data, providing the ability to store and quickly query large datasets. The coordinator 120 can provide interconnectivity by acting as a centralized communication point for third-party solutions and / or applications.

[0069] Robot 130 is an execution agent that runs workflows built into Designer 110. A commercial example of some embodiments of (multiple) robots 130 is UiPath Robots. TM In some embodiments, robot 130 has Microsoft installed by default. Services managed by the Service Control Manager (SCM). As a result, this type of robot 130 can open interactive windows under the local system account. Conversation, and has The right to services.

[0070] In some embodiments, the robot 130 can be installed in user mode. For such robots 130, this means they have the same rights as a user who has already installed a given robot 130. This feature can also be used for high-density (HD) robots, ensuring full utilization of the maximum potential of each machine. In some embodiments, any type of robot 130 can be configured in an HD environment.

[0071] In some embodiments, robot 130 is divided into several components, each dedicated to a specific automation task. Robot components in some embodiments include, but are not limited to, SCM-managed robot services, user-mode robot services, actuators, agents, and command lines. SCM-managed robot service management and monitoring... The session acts as a proxy between the coordinator 120 and the execution host (i.e., the computing system on which the robot 130 executes). These services are trusted and manage credentials for the robot 130. The console application is launched by the SCM under the local system.

[0072] In some embodiments, user-mode robot service management and monitoring The session acts as a proxy between the coordinator 120 and the execution host. The user-mode robot service can be trusted and manage credentials for robot 130. If the SCM-managed robot service is not installed, it can start automatically. application.

[0073] The actuator can be A given job runs within a session (i.e., they can execute workflows). The executor can know the dots per inch (DPI) setting for each monitor. An agent can be... Presentation Foundation (WPF) applications display available jobs in the system tray window. Agents can be clients of the service. Agents can request to start or stop a job and change settings. The command line is a client of the service. The command line is a console application that can request to start a job and wait for its output.

[0074] Decomposing the components of robot 130 as explained above helps developers, support users, and the computing system more easily run, identify, and track what each component is performing. This allows for configuring specific behaviors for each component, such as setting different firewall rules for executors and services. In some embodiments, the executor may always know the DPI setting for each monitor. As a result, workflows can be executed at any DPI, regardless of the configuration of the computing system on which they are created. In some embodiments, projects from designer 110 can also be independent of browser scaling levels. For applications that do not know the DPI or are intentionally marked as not knowing it, DPI can be disabled in some embodiments.

[0075] Figure 2 This is an architectural diagram illustrating a deployed RPA system 200 according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be... Figure 1 The RPA system 100 may be an integral part thereof. It should be noted that the client side, server side, or both may include any desired number of computing systems without departing from the scope of the invention. On the client side, the robot application 210 includes an actuator 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not run on the computing system 210. The actuator 212 operates. Several business processes can run simultaneously, such as... Figure 2 As shown in the example. In this embodiment, agent 214 (e.g., The service is a single point of contact for all executors 212. In this embodiment, all messages are logged to the coordinator 230, which further processes them via the database server 240, the indexer server 250, or both. (See above regarding...) Figure 1 The actuator 212 discussed may be a robot component.

[0076] In some embodiments, a robot represents an association between a machine name and a username. A robot can manage multiple actuators simultaneously. In computing systems that support running multiple interactive sessions simultaneously (e.g., On Server 2012, multiple bots can run simultaneously, each using a unique username in a separate environment. During the conversation. This was referred to above as the HD robot.

[0077] Agent 214 is also responsible for sending the robot's status (e.g., periodically sending "heartbeat" messages indicating that the robot is still operational) and downloading the required version of the package to be executed. In some embodiments, communication between agent 214 and coordinator 230 is always initiated by agent 214. In notification scenarios, agent 214 may open a WebSocket channel, which is later used by coordinator 230 to send commands to the robot (e.g., start, stop, etc.).

[0078] On the server side, a presentation layer (web application 232, Open Data Protocol (OData) Representative State Delivery (REST) ​​Application Programming Interface (API) endpoint 234, and notification and monitoring 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240 and indexer server 250) are included. The coordinator 230 includes the web application 232, the OData REST API endpoint 234, the notification and monitoring 236, and the API implementation / business logic 238. In some embodiments, most actions performed by the user in the interface of the coordinator 230 (e.g., via browser 220) are performed by calling various APIs. Such actions may include, but are not limited to, starting a job on a robot, adding / removing data from a queue, scheduling a job to run unattended, etc., without departing from the scope of the invention. The web application 232 is the visual layer of the server platform. In this embodiment, the web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the invention. In this embodiment, the user interacts with a webpage from web application 232 via browser 220 to perform various actions to control coordinator 230. For example, the user can create robot groups, assign packages to robots, analyze logs of each robot and / or each process, start and stop robots, etc.

[0079] In addition to web application 232, coordinator 230 also includes a service layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the invention. The REST API is used by web application 232 and agent 214. In this embodiment, agent 214 is a manager of one or more robots on a client computer.

[0080] The REST API in this embodiment covers configuration, logging, monitoring, and queuing functionality. In some embodiments, the configuration endpoint can be used to qualify and configure application users, permissions, bots, assets, publications, and environments. The logging REST endpoint can be used to log various information, such as errors, explicit messages sent by bots, and other environment-specific information. Bots can use the deployment REST endpoint to query the wrapper version that should be executed if a start job command is used in coordinator 230. The queuing REST endpoint can be responsible for queue and queue item management, such as adding data to the queue, retrieving transactions from the queue, and setting the status of transactions.

[0081] The monitoring REST endpoint can monitor web application 232 and agent 214. The notification and monitoring API 236 can be a REST endpoint used to register agent 214, deliver configuration settings to agent 214, and send / receive notifications from the server and agent 214. In some embodiments, the notification and monitoring API 236 can also use WebSocket communication.

[0082] In this embodiment, the persistence layer includes a pair of servers—a database server 240 (e.g., an SQL server) and an indexer server 250. In this embodiment, the database server 240 stores configurations for robots, robot groups, associated processes, users, roles, schedules, etc. In some embodiments, this information is managed via a web application 232. The database server 240 can manage queues and queue items. In some embodiments, the database server 240 can store messages logged by the robot logs (in addition to or instead of the indexer server 250).

[0083] In some embodiments, an optional indexer server 250 stores and indexes information logged by the robot logs. In some embodiments, the indexer server 250 can be disabled through configuration settings. In some embodiments, the indexer server 250 uses... It is an open-source full-text search engine project. Messages logged by the robot (e.g., using activities like log messages or writing lines) can be sent to the indexer server 250 via (multiple) logging REST endpoints, where they are indexed for future use.

[0084] Figure 3This is an architecture diagram illustrating the relationship 300 between a designer 310, activities 320, 330, a driver 340, and an AI / ML model 350 according to an embodiment of the present invention. As described above, a developer uses the designer 310 to develop a workflow executed by a robot. The workflow may include user-defined activities 320 and UI automation activities 330. In some embodiments, user-defined activities 320 and / or UI automation activities 330 may invoke one or more AI / ML models 350, which may be locally located on a computing system on which the robot is operating and / or remotely operating. Some embodiments are capable of identifying non-textual visual components in an image, referred to herein as computer vision (CV). Some CV activities associated with such components may include, but are not limited to, clicking, typing, text capture, hovering, element presence, refreshing range, highlighting, etc. In some embodiments, clicking uses, for example, CV, optical character recognition (OCR), fuzzy text matching, and multi-anchoring to identify an element and click it. Typing can use the above to identify an element and type within it. Text capture can identify the location of specific text and scan it using OCR. Hovering can identify an element and hover over it. The presence of elements can be checked using the techniques described above. In some embodiments, hundreds or even thousands of activities may be implemented in the designer 310. However, any number and / or type of activities may be available without departing from the scope of the invention.

[0085] UI automation activities 330 are a specific subset of lower-level activities written in lower-level code (e.g., CV activities) and facilitating interaction with the screen. UI automation activities 330 facilitate these interactions via drivers 340 and / or AI / ML models 350 that allow the robot to interact with desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc. UI automation activities 330 may use one or more AI / ML models 350 to determine the execution of interactions with the computing system. In some embodiments, AI / ML models 350 may enhance or completely replace drivers 340. In fact, in some embodiments, drivers 340 are not included.

[0086] Driver 340 can interact with the OS at a low level to locate hooks, monitor key presses, etc. These can facilitate... Integration of technologies such as "click" and "click" functions. For example, the "click" activity performs the same role in these different applications via driver 340.

[0087] Figure 4This is an architectural diagram illustrating an RPA system 400 according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be or include Figure 1 and Figure 2 The RPA system 100 and / or 200. The RPA system 400 includes multiple client computing systems 410 that run robots. The computing systems 410 are able to communicate with the coordinator computing system 420 via a web application running thereon. The coordinator computing system 420 is then able to communicate with a database server 430 and an optional indexer server 440.

[0088] about Figure 1 and Figure 3 It should be noted that although web applications are used in these embodiments, any suitable client / server software can be used without departing from the scope of the invention. For example, the coordinator can run a server-side application that communicates with non-web-based client software applications on the client computing system.

[0089] Figure 5 This diagram illustrates the architecture of a computing system according to an embodiment of the present invention, configured to perform anomaly detection and self-healing for RPA via AI / ML. In some embodiments, computing system 500 may be one or more computing systems depicted and / or described herein. Computing system 500 includes a bus 505 or other communication mechanism for transmitting information, and multiple processors 510 coupled to bus 505 for processing information. The multiple processors 510 may be any type of general-purpose or special-purpose processor, including a central processing unit (CPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The multiple processors 510 may also have multiple processing cores, and at least some cores may be configured to perform specific functions. In some embodiments, multi-parallel processing may be used. In some embodiments, at least one or more processors 510 may be a neuromorphic circuit including processing elements that simulate biological neurons. In some embodiments, the neuromorphic circuit may not require typical components of a von Neumann computing architecture.

[0090] The computing system 500 also includes a memory 515 for storing information and instructions to be executed by the processor(s) 510. The memory 515 may consist of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage (e.g., a hard disk or optical disk), or any other type of non-transitory computer-readable medium, or any combination thereof. The non-transitory computer-readable medium may be any available medium accessible by the processor(s) 510, and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both.

[0091] Furthermore, the computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communication network via wireless and / or wired connections. In some embodiments, the communication device 520 may be configured to use Frequency Division Multiple Access (FDMA), Single-Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), CDMA2000, Wideband CDMA (W-CDMA), High-Speed ​​Downlink Packet Access (HSDPA), and High-Speed ​​Uplink Packet Access (HSUPA). The communication device 520 may include High-Speed ​​Packet Access (HSPA), Long Term Evolution (LTE), LTE-Advanced (LTE-A), 802.11x, Wi-Fi, Purple Bee, Ultra Wideband (UWB), 802.16x, 802.15, Home Node B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near Field Communication (NFC), 5G, New Radio (NR), and any combination thereof, and / or any other existing or future communication standards and / or protocols, without departing from the scope of the invention. In some embodiments, the communication device 520 may include one or more antennas, which may be single, arrayed, phased, switched, beamforming, beamcontrolled, combinations thereof, and / or any other antenna configuration without departing from the scope of the invention.

[0092] Multiple processors 510 are also coupled to a display 525 via a bus 505, such as a plasma display, liquid crystal display (LCD), light-emitting diode (LED) display, field emission display (FED), organic light-emitting diode (OLED) display, flexible OLED display, flexible substrate display, projection display, 4K display, high-definition display, etc. The display 525 may be a coplanar conversion (IPS) display or any other display suitable for displaying information to a user. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, etc., using resistive, capacitive, surface acoustic wave (SAW) capacitor, infrared, optical imaging, dispersive signal technology, acoustic impulse recognition, suppressed total internal reflection, etc., without departing from the scope of this invention. Any suitable display device and haptic I / O may be used.

[0093] Keyboard 530 and cursor control device 535, such as a computer mouse, touchpad, etc., are also coupled to bus 505 to enable the user to interface with computing system 500. However, in some embodiments, a physical keyboard and mouse may not be present, and the user may interact with the device solely through display 525 and / or touchpad (not shown). Any type and combination of input devices can be chosen as a design option. In some embodiments, there are no physical input devices and / or displays. For example, the user may interact remotely with computing system 500 through another computing system with which it communicates, or computing system 500 may operate autonomously.

[0094] Memory 515 stores software modules that provide functionality when executed by processor(s) 510. These modules include an operating system 540 for the computing system 500. The module also includes an anomaly detection and self-healing module 545 configured to perform all or part of the processes described herein or derivatives thereof. The computing system 500 may include one or more additional function modules 550 containing additional functionalities.

[0095] Those skilled in the art will understand that "system" can be embodied as a server, embedded computing system, personal computer, console, personal digital assistant (PDA), cellular phone, tablet computing device, quantum computing system, or any other suitable computing device or combination thereof, without departing from the scope of the invention. Presenting the foregoing functionality as being performed by a "system" is not intended to limit the scope of the invention in any way, but rather to provide an example of many embodiments of the invention. In fact, the methods, systems, and apparatuses disclosed herein can be implemented in localized and distributed forms in accordance with computing technologies, including cloud computing. A computing system can be part of or accessible from a local area network (LAN), mobile communication network, satellite communication network, the Internet, public or private cloud, hybrid cloud, server farm, any combination thereof, etc. Any localized or distributed architecture can be used without departing from the scope of the invention.

[0096] It should be noted that some system features described in this specification have been presented as modules to more specifically emphasize their implementation independence. For example, modules can be implemented as hardware circuits, including custom-designed very large-scale integrated circuit (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. Modules can also be implemented in programmable hardware devices, such as field-programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, etc.

[0097] Modules can also be implemented at least partially in software for execution by various types of processors. The identified units of executable code can, for example, comprise one or more physical or logical blocks of computer instructions, which can, for example, be organized as objects, processes, or functions. Nevertheless, the executable files of the identified modules do not need to be physically located together, but can comprise different instructions stored in different locations that, when logically combined, constitute the module and achieve the module's stated purpose. Furthermore, modules can be stored on a computer-readable medium, which can be, for example, a hard disk drive, flash memory device, RAM, magnetic tape, and / or any other such non-transitory computer-readable medium for storing data without departing from the scope of the invention.

[0098] In fact, an executable code module can be a single instruction or multiple instructions, and can even be distributed across multiple different code segments, different programs, and across multiple storage devices. Similarly, operational data in this document can be identified and represented within a module, and can be presented in any suitable form and organized in any suitable data structure. This operational data can be collected as a single dataset, or it can be distributed across different locations including different storage devices, and can exist at least in part as electronic signals on a system or network.

[0099] Figure 6 This diagram illustrates the architecture of a system 600 according to an embodiment of the present invention. The system 600 is configured to perform anomaly detection and self-healing for RPA via AI / ML. System 600 includes user computing systems, such as desktop computers 602 and 603, tablet computers 604, and smartphones 606. However, any desired computing system can be used without departing from the scope of the invention, including but not limited to smartwatches, laptop computers, etc. Furthermore, although... Figure 6 Three user computing systems are shown, but any suitable number of computing systems can be used without departing from the scope of the invention. For example, in some embodiments, dozens, hundreds, thousands, or millions of computing systems may be used.

[0100] Each computing system 602, 604, 606 has an AI / ML-capable RPA robot 610 that uses an AI / ML model, such as AI / ML model 632, to perform certain tasks. For example, the RPA robot 610 can use an AI / ML model trained to use computer vision (CV) to identify what a user is doing on the corresponding computing system. Computing systems 602, 604, 606 send information to server 630 via network 620 (e.g., local area network (LAN), mobile communication network, satellite communication network, the Internet, and any combination thereof). In some embodiments, server 630 may be part of a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, etc. In some embodiments, server 630 may host multiple software-based servers on a single computing system 630. In this embodiment, server 630 includes an AI / ML-capable RPA robot 632, which is invoked by RPA robot 610 to perform operations.

[0101] The RPA robot 610 includes anomaly handling and / or self-healing logic that attempts to automatically detect and correct anomalies. This logic can be incorporated into the RPA robot 610 itself, the AI / ML model 632, or a combination thereof. Including anomaly handling and / or self-healing logic as part of the AI / ML model 632 can allow for the utilization of information from multiple or many computing systems, potentially providing more samples, examples of how users can overcome problems, etc.

[0102] RPA robot 610 may occasionally encounter new, unprocessed anomalies where graphical elements cannot be identified and / or normal interactions will not function. Such anomalies can be handled by an anomaly handler (e.g., a subset of activities in the RPA robot 610's workflow(s) or logic within the AI / ML model 632), which attempts to handle the anomaly automatically. Where the anomaly handler still cannot identify the target graphical element, RPA robot 610 can handle the exception. In some embodiments, RPA robot 610 and / or AI / ML model 632 may have self-healing capabilities that attempt to complete missing data from an anomaly without user input.

[0103] In some embodiments, anomaly-related data may be stored in a database 640 for examination by an application 652 of a training computation system 650, which allows the user to flag screens associated with unhandled anomalies and / or correct attempts at self-correction by the RPA robot 610. For example, if a previously unseen scenario (such as a new control not recognized by the AI / ML model) occurs, prompts may be provided to the user. The user can then flag controls on the screen, and if a control functions differently from previous controls that the AI / ML model was trained to recognize, actions to take are potentially provided. Using the training data, the AI / ML model 632 can be trained to increase accuracy. In some embodiments, the AI / ML model invoked by the RPA robot 610 may have multiple layers performing various functions, such as statistical modeling (e.g., Hidden Markov Models (HMMs)) and utilizing deep learning techniques (e.g., Long Short-Term Memory (LSTM) deep learning, encoding of previous hidden states, etc.) to identify sequences of user interactions.

[0104] AI layer

[0105] In some embodiments, multiple AI layers may be used. Each AI layer is an algorithm (or model) that runs on data, and the AI ​​model itself may be a deep learning neural network (DLNN) of trained artificial “neurons” trained on training data. The layers may run in series, in parallel, or in combination.

[0106] AI layers may include, but are not limited to, sequence extraction layers, cluster detection layers, visual component detection layers, text identification layers (e.g., OCR), audio-to-text conversion layers, or any combination thereof. However, any desired number and type of layers may be used without departing from the scope of this invention. Using multiple layers allows the system to develop a global picture of what is happening on the screen. For example, one AI layer may perform OCR, another may detect buttons, yet another may compare sequences, etc. Patterns may be determined by a single AI layer or by multiple AI layers working together.

[0107] Figure 7This is a flowchart illustrating a process 700 of performing anomaly detection and self-correction for RPA via AI / ML according to an embodiment of the present invention. At 705, the process begins by executing an RPA robot (and therefore an RPA workflow) that uses an AI / ML model to perform UI automation. At 710, the RPA robot uses the AI / ML model to search the UI for target graphical elements that activities in the RPA workflow want to interact with. At 715, when the target graphical element is uniquely found by the AI / ML model, at 720, the RPA robot interacts with the target graphical element based on one or more RPA activities in the RPA workflow. Then, at 725, the RPA robot continues executing the RPA workflow logic until interaction with the next graphical element is required. The process then returns to step 710.

[0108] However, at 715, when the target graphic element is not uniquely found by the AI / ML model or cannot be interacted with, at 730, the RPA robot or AI / ML model automatically attempts to correct the anomaly. In some embodiments, the automatic attempt to correct the anomaly includes determining whether one or more features distinguish the target graphic element from other similar graphic elements. In some embodiments, determining whether one or more features distinguish the target graphic element from other similar graphic elements includes analyzing graphic elements surrounding the target graphic element within a radius, utilizing the order of graphic elements in the UI, and determining whether the target graphic element has one or more distinct visual characteristics, or a combination thereof. In some embodiments, the automatic attempt to correct the anomaly includes changing the visual characteristics of the UI and searching for the target graphic element in the UI using the changed visual characteristics. In some embodiments, the automatic attempt to correct the anomaly includes taking a default action. In some embodiments, the default action may be or include searching a UI object library or UI object repository for one or more UI descriptors that enable desired interaction with the target graphic element.

[0109] In some embodiments, automated attempts to correct anomalies include an RPA bot or AI / ML model attempting a self-repair process to complete the missing data without user input. This can be accomplished, for example, via an exploration phase in reinforcement learning. In some embodiments, the RPA bot or AI / ML model is configured to determine the success of the self-repair process by monitoring whether one or more performance metrics improve in response to the self-repair process. In some embodiments, automated attempts to correct anomalies when the self-repair process is unsuccessful include trying one or more different techniques and monitoring whether the different techniques improve one or more performance metrics. In some embodiments, the self-repair process includes polling multiple users for suggested solutions to the anomaly and selecting the optimal solution from the suggested solutions based on one or more performance metrics. For example, the optimal solution could be selected based on speed, increased revenue, creation of more consumer accounts, or a combination thereof.

[0110] At point 735, upon successful automatic attempt to correct the anomaly, at point 740, data related to the automatic correction is provided (e.g., to remote servers and databases, for example). Figure 6 The data is sent to server 630 and database 640 for subsequent retraining of the AI / ML model, and the process proceeds to step 720 and continues as discussed above. However, at 745, if the automatic attempt to correct the anomaly fails, the user is prompted to identify the target graphical element, how to interact with the target graphical element, or a solution for both. For example, if a previously unseen scenario (such as a new control not recognized by the AI / ML model) occurs, guidance can be provided to the user. The user can then mark the control on the screen, and if the control works differently from previous controls that the AI / ML model was trained to recognize, actions to be taken are potentially provided. At 740, if the guidance provided by the user successfully enables the RPA robot to interact with the target graphical element, data related to the user's solution is provided for subsequent retraining of the AI / ML model, and the process proceeds to step 720 and continues as discussed above.

[0111] At point 750, if the user-provided guidance fails to enable the RPA robot to interact with the target graphical element, and the target graphical element does not necessarily need to complete the overall task of the RPA workflow, and continued operation at point 755 is possible, the RPA robot can continue the execution of the RPA workflow at point 725. This is possible, for example, if the target graphical element is not required to complete the overall task of the RPA workflow with sufficient accuracy. However, if continued operation is not possible at point 755, an exception is thrown at point 760 and the process terminates.

[0112] Figure 7 The process steps executed in the process can be executed by a computer program, which encodes instructions for (multiple) processors to perform the task. Figure 7 The computer program may be embodied on a non-transitory computer-readable medium. The computer-readable medium may be, but is not limited to, hard disk drives, flash memory devices, RAM, magnetic tape, and / or any other such medium or combination of media for storing data. The computer program may include processor(s) for controlling a computing system (e.g., ...). Figure 5 The computing system 500 has (multiple) processors 510 to achieve Figure 7 The coded instructions for all or part of the process steps described herein may also be stored on a computer-readable medium.

[0113] Computer programs can be implemented in hardware, software, or a hybrid manner. A computer program can consist of communicable modules designed to exchange information or instructions for display. A computer program can be configured to run on a general-purpose computer, an ASIC, or any other suitable device.

[0114] It is readily understood that the components of various embodiments of the present invention, as generally described and illustrated in the accompanying drawings, can be arranged and designed in a wide variety of different configurations. Therefore, the detailed description of embodiments of the present invention, as shown in the accompanying drawings, is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention.

[0115] The features, structures, or characteristics of the invention described throughout this specification can be combined in any suitable manner in one or more embodiments. For example, references to "certain embodiments," "some embodiments," or similar language throughout this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. Therefore, the phrases "in some embodiments," "in some embodiments," "in other embodiments," or similar language appearing throughout this specification do not necessarily refer to the same set of embodiments, and the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0116] It should be noted that references to features, advantages, or similar language throughout this specification do not imply that all features and advantages achievable by the invention should be included in or in any single embodiment of the invention. Rather, references to features and advantages are to be understood as meaning that a particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Therefore, the discussion of features and advantages, as well as similar language, throughout this specification may, but does not necessarily, refer to the same embodiment.

[0117] Furthermore, the described features, advantages, and characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the invention can be practiced without one or more specific features or advantages of a particular embodiment. In other instances, additional features and advantages not presented in all embodiments of the invention may be recognized in certain embodiments.

[0118] It will be readily understood by those skilled in the art that the invention described above can be implemented with steps in a different order and / or with hardware elements in a different configuration than those disclosed. Therefore, although the invention has been described based on these preferred embodiments, certain modifications, variations, and alternative structures will be apparent to those skilled in the art while remaining within the spirit and scope of the invention. Therefore, reference should be made to the appended claims to determine the boundaries and limits of the invention.

Claims

1. A computer-implemented method, comprising: The RPA workflow is executed by a Robotic Process Automation (RPA) robot, which uses an Artificial Intelligence (AI) / Machine Learning (ML) model to perform user interface (UI) automation. One of the RPA robots uses one of the AI / ML models in the AI / ML model to search for target graphical elements in the user interface (UI) that the activity to interact with through the RPA workflow. as well as When the target graphical element is not uniquely found by the AI / ML model or cannot be interacted with, causing an anomaly in the operation of the RPA robot, the RPA robot or the AI / ML model automatically attempts to correct the anomaly. The automatic attempt to correct the anomaly includes: determining whether one or more features distinguish the target graphic element from other similar graphic elements, wherein The determination includes: analyzing graphic elements surrounding the target graphic element within a radius, using the order of the graphic elements in the UI to determine whether the target graphic element has one or more distinct visual characteristics, or a combination thereof, and wherein The automatic attempt also includes: changing the visual characteristics of the UI and using the changed visual characteristics to search for the target graphic element in the UI.

2. The computer-implemented method of claim 1, wherein the automatic attempt to correct the anomaly includes taking a default action.

3. The computer-implemented method according to claim 2, wherein the default action includes searching a UI object library or UI object store for one or more UI descriptors, wherein the one or more UI descriptors implement interaction with the target graphical element.

4. The computer-implemented method according to claim 1, wherein when the automatic attempt to correct the anomaly succeeds, the method further comprises: The RPA robot provides data related to automatic correction for subsequent retraining of the AI / ML model.

5. The computer-implemented method of claim 1, wherein when the automatic attempt to correct the anomaly fails, the method further comprises: The RPA robot prompts the user to identify the target graphic element, how to interact with the target graphic element, or a solution to both.

6. The computer-implemented method of claim 5, wherein when the user-provided guidance successfully enables the RPA robot to interact with the target graphical element, the method further comprises: The RPA robot provides data related to the user solution for subsequent retraining of the AI / ML model.

7. The computer-implemented method of claim 5, wherein when the user-provided guidance fails to enable the RPA robot to interact with the target graphical element, and the target graphical element does not necessarily complete the overall task of the RPA workflow, the method further comprises: The RPA robot attempts to continue executing the RPA workflow.

8. The computer-implemented method of claim 1, wherein the automatic attempt to correct the anomaly comprises: The RPA robot or the AI / ML model attempts a self-repair process to complete the missing data without user input.

9. The computer-implemented method of claim 8, wherein the RPA robot or the AI / ML model is configured to determine whether the self-healing process is successful by monitoring one or more performance metrics in response to improvements in the self-healing process.

10. The computer-implemented method of claim 8, wherein if the self-repair process fails, the method further comprises: Try one or more different techniques and monitor whether the one or more different techniques improve one or more performance metrics.

11. The computer-implemented method of claim 8, wherein the self-repair process comprises: Poll multiple users to provide suggested solutions to the anomaly; as well as The optimal solution among the proposed solutions is selected based on one or more performance metrics.

12. The computer-implemented method of claim 8, wherein the attempt to complete the self-repair process of making the missing data complete without user input includes the use of an exploration phase in reinforcement learning.

13. A non-transitory computer-readable medium storing a computer program configured to cause at least one processor to: The Robotic Process Automation (RPA) workflow is executed, which uses an artificial intelligence (AI) / machine learning (ML) model to perform user interface (UI) automation. Using one of the AI / ML models, search the user interface (UI) for target graphical elements that the activity wants to interact with through the RPA workflow; as well as When the target graphical element is not uniquely found by the AI / ML model or cannot be interacted with, causing an anomaly in the operation of the Robotic Process Automation (RPA) robot, the system automatically attempts to correct the anomaly. The automatic attempt to correct the anomaly includes: determining whether one or more features distinguish the target graphic element from other similar graphic elements, wherein The determination includes: analyzing graphic elements surrounding the target graphic element within a radius, using the order of the graphic elements in the UI to determine whether the target graphic element has one or more distinct visual characteristics, or a combination thereof, and wherein The automatic attempt also includes: changing the visual characteristics of the UI and using the changed visual characteristics to search for the target graphic element in the UI.

14. The non-transitory computer-readable medium of claim 13, wherein... Upon successful automatic attempt to correct the anomaly, the computer program is further configured to cause the at least one processor to provide data related to the successful correction of the anomaly for subsequent retraining of the AI / ML model. When the automatic attempt to correct the anomaly fails, the computer program is also configured to cause the at least one processor to prompt the user to identify the target graphic element, how to interact with the target graphic element, or a solution to both.

15. The non-transitory computer-readable medium of claim 14, wherein... When the computer program is successfully enabled to interact with the target graphical element by guidance provided by the user, the computer program is also configured to cause the at least one processor to provide data related to the user's solution for subsequent retraining of the AI / ML model, and If the user-provided guidance fails to enable the computer program to interact with the target graphical element, and the target graphical element does not need to complete the overall task of the RPA workflow, the computer program is also configured to cause the at least one processor to attempt to continue execution of the RPA workflow.

16. The non-transitory computer-readable medium of claim 13, wherein the automatic attempt to correct the anomaly includes attempting a self-repair process to complete the missing data without user input.

17. A computing system, comprising: Memory that stores computer program instructions; as well as At least one processor, the at least one processor being configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to: The RPA workflow is executed by a Robotic Process Automation (RPA) robot, which uses an Artificial Intelligence (AI) / Machine Learning (ML) model to perform user interface (UI) automation. One of the RPA robots uses one of the AI / ML models in the AI / ML model to search for target graphical elements in the user interface (UI) that the activity to interact with through the RPA workflow. as well as When the target graphical element is not uniquely found by the AI / ML model or cannot be interacted with, causing an anomaly in the operation of the RPA robot, the RPA robot or the AI / ML model automatically attempts to correct the anomaly. The automatic attempt to correct the anomaly includes: determining whether one or more features distinguish the target graphic element from other similar graphic elements, wherein The determination includes: attempting a self-repair process to complete the missing data without user input, or both; analyzing the graphic elements surrounding the target graphic element within a radius; using the order of the graphic elements in the UI to determine whether the target graphic element has one or more distinct visual characteristics, or a combination thereof; and wherein The automatic attempt also includes: changing the visual characteristics of the UI and using the changed visual characteristics to search for the target graphic element in the UI.

Images