Graphical element detection using a combination of user interface descriptor attributes according to two or more graphical element detection techniques

Abstract

Graphical element detection using a combination of user interface (UI) descriptor attributes according to two or more graphical element detection techniques is disclosed. The UI descriptor can be used to compare attributes for a given UI descriptor with attributes of UI elements found in the UI at runtime. At runtime, the attributes for the UI elements found in the UI can be searched for a match with attributes of a corresponding RPA workflow activity, and if an exact match or a match within a match threshold is found, the UI can be identified and interacted with accordingly.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Non-Provisional Patent Application No. 16 / 990,709, filed on August 11, 2020. The subject matter of that earlier-filed application is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates generally to graphical element detection, and more particularly to graphical element detection using a combination of user interface (UI) descriptor attributes according to two or more graphical element detection techniques. Background Art

[0004] For Robotic Process Automation (RPA) in UI, graphic element detection can be performed using selectors, computer vision (CV), or optical character recognition (OCR) for each UI action. However, these techniques are applied separately and are not optimal for all scenarios. Therefore, an improved method may be beneficial. Summary of the invention

[0005] Certain embodiments of the present invention may provide solutions to problems and needs in the art that have not been fully identified, understood, or solved by current graphic element detection techniques. For example, some embodiments of the present invention relate to graphic element detection using a combination of UI descriptor attributes according to two or more graphic element detection techniques (e.g., selector, CV, OCR, etc.).

[0006] In an embodiment, a computer-implemented method for detecting graphical elements in a UI includes: receiving, by a designer application, a selection of an activity to be configured in an RPA workflow. The computer-implemented method also includes: receiving, by the designer application, a selection and / or modification of: one or more graphical element detection techniques from a plurality of graphical element detection techniques, one or more UI descriptor attributes of the graphical element detection techniques, or a combination thereof. The computer-implemented method also includes: configuring, by the designer application, the activity based on the received selection and / or modification.

[0007] In another embodiment, a computer program is implemented on a non-transitory computer-readable medium. The program is configured to cause at least one processor to receive a selection of an activity to be configured in an RPA workflow. The computer program is also configured to cause at least one processor to receive a selection and / or modification of: one or more graphic element detection technologies among a plurality of graphic element detection technologies, one or more UI descriptor attributes of a graphic element detection technology, or a combination thereof. The computer program is also configured to cause at least one processor to configure the activity based on the received selection and / or modification.

[0008] In yet another embodiment, a computer program is implemented on a non-transitory computer-readable medium. The program is configured to cause at least one processor to analyze a UI at runtime to identify UI element attributes. The computer program is also configured to cause at least one processor to compare the UI element attributes with UI descriptor attributes according to multiple graphic element detection techniques for an activity of an RPA workflow. When a UI element matching the attributes of the multiple graphic element detection techniques is found via an exact match or a threshold match, the computer program is configured to cause at least one processor to take an action associated with the activity involving the UI element. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to readily understand the advantages of certain embodiments of the present invention, a more particular description of the invention briefly described above will be presented by reference to specific embodiments illustrated in the accompanying drawings. While it should be understood that these drawings depict only typical embodiments of the invention and, therefore, should not be considered limiting of the scope of the invention, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0010] Figure 1 is an architectural diagram illustrating a robotic process automation (RPA) system according to an embodiment of the present invention.

[0011] Figure 2 is an architectural diagram illustrating an RPA system deployed according to an embodiment of the present invention.

[0012] Figure 3 is an architectural diagram illustrating the relationship between a designer, an activity, and a driver according to an embodiment of the present invention.

[0013] Figure 4 is an architectural diagram illustrating an RPA system according to an embodiment of the present invention.

[0014] Figure 5 is an architectural diagram illustrating a computing system configured to perform graphical element detection using a combination of UI descriptor attributes according to two or more graphical element detection techniques in accordance with an embodiment of the present invention.

[0015] Figure 6 An example graphical element detection technology configuration interface for an RPA designer application according to an embodiment of the present invention is illustrated.

[0016] Fig. 7A and Figure 7B Another example graphical element detection technology configuration interface for an RPA designer application according to an embodiment of the present invention is illustrated.

[0017] Figure 8 is a flow chart illustrating a process for configuring an RPA workflow to perform graphical element detection using a combination of UI descriptor attributes according to two or more graphical element detection techniques according to an embodiment of the present invention.

[0018] Fig. 9 is a flow chart illustrating a process for performing graphical element detection using a combination of UI descriptor attributes according to two or more graphical element detection techniques according to an embodiment of the present invention.

[0019] Unless otherwise indicated, similar reference numerals indicate corresponding features throughout the drawings. DETAILED DESCRIPTION

[0020] Some embodiments relate to graphical element detection using a combination of UI descriptor attributes according to two or more graphical element detection techniques (e.g., selectors, CV, OCR, etc.). "Graphic element" and "UI element" are used interchangeably herein. A UI descriptor identifies a UI element at its core (e.g., a text field, a button, a label, a menu, a check box, etc.). Some types of UI descriptors include, but are not limited to, selectors, CV descriptors, image matching descriptors, OCR descriptors, unified target descriptors that can utilize multiple different types of UI descriptors in series or in parallel, etc. A UI descriptor can be used to compare the attributes for a given UI descriptor with the attributes of a UI element found in the UI at runtime.

[0021] In some embodiments, the UI descriptor stores attributes of the corresponding UI element and its parent element, for example in an Extensible Markup Language (XML) fragment. At runtime, attributes for UI elements found in the UI can be searched for matches with attributes for the corresponding RPA workflow activity, and if an exact match or a "close enough" match is found within the matching threshold, the UI element can be identified and interacted accordingly. Attributes may include text-based identifiers (IDs), classes, roles, and the like. For CV, attributes may include the type of target element and a relationship to one or more anchor elements that can be used in a multi-anchor matching approach. For OCR, attributes may include, for example, text in the form of a stored string and text found via OCR, and during execution, the stored string is fuzzy matched with the OCR. Any suitable attribute and graphic element detection techniques may be used without departing from the scope of the present invention.

[0022] Some embodiments search for a mix of automatically defined or custom attributes from two or more different graphical element detection techniques. For example, some selector attributes, some CV descriptor attributes, and some OCR descriptor attributes can be used to jointly and potentially simultaneously identify graphical elements at runtime in a manner that can be more accurate than using any graphical element detection technique alone. For example, if there are groups of check boxes on the screen with different visual appearances, it may make sense to use an image matching descriptor for identifying the check box images in combination with a CV descriptor that provides multi-anchor matching information related to the position of a particular check box relative to a nearby anchor UI element. This can improve accuracy compared to only a single graphical element detection technique.

[0023] In some embodiments, the combined graphic element detection technology can be configured at design time using an RPA designer application (e.g., UiPath Studio™). For example, an out-of-the-box click activity can be configured to view the type of target graphic element to be identified and / or certain characteristics of the target graphic element to be identified to automatically determine which combination of graphic element detection technology and corresponding attributes to use. In some embodiments, the RPA developer can modify the default logic of the activity, and / or provide graphic element detection technology and its attributes to be used manually. For example, the RPA developer can select the type of graphic element detection technology to be applied. In some embodiments, the RPA developer can select from a list of attributes provided for a given technology selection, the developer can manually enter the attributes, etc.

[0024] The combination of attributes used according to the techniques may be a subset and / or all of the attributes for each technique. For example, in some embodiments, all attributes for all selected techniques may be used. However, in certain embodiments, one attribute according to the selector may be used, two attributes according to the CV descriptor may be used, all attributes according to the OCR descriptor may be used, etc.

[0025] As used herein, a "screen" is an image of an application UI or a portion of an application UI at a certain point in time. In some embodiments, UI elements and screens can be further distinguished into specific types of UI elements (e.g., buttons, check boxes, text fields, etc.) and screens (e.g., top windows, modal windows, pop-up windows, etc.).

[0026] Some embodiments use a UI descriptor that stores the attributes of a UI element and its parent element in an XML fragment. In modern computing systems, the operating system typically represents each user interface as a hierarchical data structure commonly referred to as a UI tree. An example UI tree may include a document object model (DOM) based on a web page presented by a web browser application.

[0027] In some embodiments, a selector is a type of UI descriptor that can be used to detect UI elements. In some embodiments, a selector has the following structure:

[0028] <node_1 / ><node_2 / > ...<node_N / >

[0029] The last node represents the GUI element of interest, and all previous nodes represent the parents of that element.<node_1> Often referred to as the root node, and represents the top window of the application.

[0030] Each node may have one or more attributes that assist in correctly identifying a specific level of the selected application. In some embodiments, each node has the following format:

[0031] <ui_system attr_name_1='attr_value_1' ... attr_name_N='attr_value_N' / >

[0032] Each property can have an assigned value, and properties can be selected to have constant values. This is because changing the property value each time the app is launched may cause the selector to not correctly identify the associated element.

[0033] A UI descriptor is a set of instructions for finding UI elements. In some embodiments, a UI descriptor is an encapsulated data / structure format, which includes (multiple) UI element selectors, (multiple) anchor selectors, (multiple) CV descriptors, (multiple) OCR descriptors, (multiple) unified target descriptors combining two or more types of UI descriptors, screen image capture (context), element image capture, other metadata (such as applications and application versions), combinations thereof, and the like. The encapsulated data / structure format can be expanded with future updates of the platform and is not limited to the above definition. Any suitable UI descriptor for identifying UI elements on the screen may be used without departing from the scope of the present invention. UI descriptors can be extracted from activities in an RPA workflow and added to a structured scheme that groups UI descriptors by UI application, screen, and UI element.

[0034] In some embodiments, the UI descriptor can work with a unified target that contains multiple or all UI detection mechanisms (image detection and definition are performed by this mechanism). The unified target can merge multiple technologies that identify and automate UI elements into a single cohesive method. The unified target descriptor links multiple types of UI descriptors together serially, uses these UI descriptors in parallel, or first uses at least one technology (such as a selector) for a period of time, and then runs at least one other technology in parallel if the first technology does not find a match within the time period. In some embodiments, the unified target descriptor can function like a finite state machine (FSM), where in the first context, the first UI descriptor mechanism is applied; in the second context, the second UI descriptor is applied, and so on. In some embodiments, if the first two mechanisms are unsuccessful, the unified target can prioritize selector-based and drive-based UI detection mechanisms, and use CV, image matching, and / or other mechanisms to find graphic elements.

[0035] In some embodiments, fuzzy matching can be used, in which one or more attributes should match with a certain accuracy (e.g., 70% match, 80% match, 99% match, etc.), should match within a certain range, should match using a string metric (e.g., Levenshtein distance, Hamming distance, Jaro-Winkler distance, etc.), combinations thereof, etc. One of ordinary skill in the art will appreciate that similarity measures can quantify the amount of similarity and the amount of mismatch between two attribute values. In addition, in various embodiments, the similarity threshold can represent the maximum amount of mismatch or the minimum amount of similarity required for a match.

[0036] Depending on the chosen way of calculating the similarity measure, the similarity threshold can have various interpretations. For example, the similarity threshold can indicate the maximum count of characters that can differ between two strings, or indicate a mismatch degree score calculated as a proportion of the total character count (e.g., the combined string length). In some embodiments, the similarity threshold can be rescaled to a predetermined interval, such as between 0 and 1, between 0 and 100, between 7 and 34, etc. In a non-limiting example, a relatively high similarity threshold (e.g., close to 1 or 100%) indicates a requirement for an almost exact match, i.e., only slightly different values ​​of the fuzzy attributes in the runtime target are allowed to differ from the values ​​of the corresponding attributes in the design time target. In contrast, when the similarity threshold is relatively low (e.g., close to 0), almost any value of the corresponding fuzzy attribute is considered a match.

[0037] In some embodiments, the matching tolerance can be different on a per-attribute basis. For example, one or more attributes may require an exact match (e.g., a certain exact name may be expected to be found), and a fuzzy match may be performed on one or more other attributes. In some embodiments, the number and / or type of attributes used per graphic element detection technique may be custom specified by the RPA developer.

[0038] In some embodiments, attributes can be stored as attribute-value pairs and / or attribute-value-tolerance pairs (e.g., fuzzy matching). In some embodiments, attribute-value pairs can indicate the name and type of the UI element represented by the corresponding node. However, it should be understood by those skilled in the art that, without departing from the scope of the present invention, in addition to the list of attribute-value pairs, there can also be multiple ways to represent the position of a specific node in the UI tree.

[0039] In some embodiments, these attribute-value pairs and / or attribute-value-tolerance pairs can be stored in tags, and each tag can include a sequence of characters interspersed with implementation-specific delimiters (e.g., starting with "<" and ending with " / >"). In some embodiments, the attribute-value pair can indicate the name and type of the UI element represented by the corresponding node. However, it should be understood by those skilled in the art that, without departing from the scope of the present invention, in addition to a list of attribute-value pairs, there can be multiple ways to represent the position of a specific node within the UI tree.

[0040] In order to enable the RPA robot to perform successful and ideally clear identification, some embodiments use an element ID that characterizes the corresponding UI element to represent each UI element. In some embodiments, the element ID indicates the position of the target node in the UI tree, where the target node represents the corresponding UI element. For example, the element ID can identify the target node / UI element as a member of a selected node subset. The selected node subset can form a pedigree, that is, a descending line running through the UI tree, in which each node is either an ancestor or a descendant of another node.

[0041] In some embodiments, the element ID includes an ordered sequence of node indicators that track the pedigree path through the UI tree and the path that ends in the corresponding target node / UI element. Each node indicator can represent a member of the object hierarchy of the corresponding UI and its position in a sequence consistent with the corresponding hierarchy. For example, each member of the sequence can represent a descendant (e.g., a child node) of the previous member, and the subsequent member can be used as a descendant (e.g., a child node). In a hypertext markup language (HTML) example, an element ID representing a separate form field can indicate that the corresponding form field is a child of an HTML form, which in turn is a child of a specific part of a web page, etc. In some embodiments, the pedigree does not need to be complete.

[0042] Some embodiments may use one or more multi-anchor matching attributes. Anchors are other UI elements that can be used to assist in uniquely identifying a target UI element. For example, if multiple text fields are included in the UI, then searching only for the text field is not enough to uniquely identify a given text field. Therefore, some embodiments look for additional information in order to uniquely identify a given UI element. Using the text field example, a text field for entering a name may appear to the right of a label "Name". The name label can be set to an "anchor" to help uniquely identify the text field as a "target".

[0043] In some embodiments, various positional associations and / or geometric associations between a target and an anchor may be used to uniquely identify a target, potentially within one or more tolerances. For example, the center of the bounding box of an anchor and a target may be used to define a line segment. The line segment may then be required to have a certain length within the tolerance and / or a slope within the tolerance to uniquely identify the target using a target / anchor pair. However, in some embodiments, any desired positioning of a position associated with a target and / or an anchor may be used without departing from the scope of the present invention. For example, the point used to draw a line segment may be at the center, the upper left corner, the upper right corner, the lower left corner, the lower right corner, any other position on the border of the bounding box, any position within the bounding box, a position outside the bounding box as identified relative to the bounding box properties, etc. In some embodiments, a target and one or more anchors may have different positions within or outside their bounding boxes for geometric matching.

[0044] Based on the above, a single anchor may not always be sufficient to uniquely identify a target element on the screen with a certain confidence. For example, consider a web form where two text fields for entering a name appear to the right of a corresponding label "name" in different locations on the screen. In this example, one or more additional anchors may be useful for uniquely identifying a given target. Geometric properties between an anchor and the target (e.g., line segment length, angle, and / or relative position with tolerance) may be used to uniquely identify the target. The user may be required to continue adding anchors until the match strength of the target exceeds a threshold.

[0045] As used herein, the terms "user" and "developer" are used interchangeably. A user / developer may or may not have programming and / or technical knowledge. For example, in some embodiments, a user / developer may create an RPA workflow by configuring activities in an RPA workflow without manual coding. In some embodiments, for example, this may be accomplished by clicking and dragging various features.

[0046] Certain embodiments may be deployed for Robotic Process Automation (RPA). Figure 1 1 is an architectural diagram illustrating an RPA system 100 according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that allows developers to design and implement workflows. The designer 110 can provide solutions for application integration and automation of third-party applications, management of information technology (IT) tasks, and business IT processes. The designer 110 can support the development of automation projects, which are graphical representations of business processes. In short, the designer 110 supports the development and deployment of workflows and robots.

[0047] Automation projects enable automation of rule-based processes by providing developers with control over the order of execution and relationships between a set of custom steps (defined herein as "activities") developed in a workflow. One commercial example of an embodiment of the designer 110 is UiPath Studio™. Each activity can include an action, such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows can be nested or embedded.

[0048] Some types of workflows may include, but are not limited to, sequences, flow charts, FSMs, and / or global exception handlers. Sequences may be particularly suitable for linear processes, thereby being able to flow from one activity to another without confusing the workflow. Flow charts may be particularly suitable for more complex business logic, thereby enabling the integration of decisions made in more different ways by multiple branching logic operators and the connection of activities. FSMs may be particularly suitable for larger workflows. FSMs may use a limited number of states in their execution, which are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suitable for determining workflow behavior and for debugging processes when encountering execution errors.

[0049] Once a workflow is developed in the designer 110, the execution of the business process is orchestrated by a conductor 120, which orchestrates one or more robots 130 that execute the workflow developed in the designer 110. One commercial example of an embodiment of the conductor 120 is UiPath Orchestrator™. The conductor 120 supports the management of the creation, monitoring, and deployment of resources in the environment. The conductor 120 can act as an integration point, or as one of the aggregation points, with third-party solutions and applications.

[0050] The commander 120 can manage a fleet of robots 130, thereby connecting and executing robots 130 from a centralized point. The types of robots 130 that can be managed include, but are not limited to, attended robots 132, unattended robots 134, development robots (similar to unattended robots 134, but for development and testing purposes), and non-production robots (similar to attended robots 132, but for development and testing purposes). Attended robots 132 can be triggered by user events or scheduled to occur automatically and operate on the same computing system with humans. Attended robots 132 can be used with the commander 120 for centralized process deployment and recording media. Attended robots 132 can help human users complete various tasks and can be triggered by user events. In some embodiments, processes cannot be started from the commander 120 on this type of robot and / or these processes cannot be run under a lock screen. In some embodiments, attended robots 132 can only be started from the robot tray or according to a command prompt. In some embodiments, attended robots 132 should operate under human supervision.

[0051] Unattended robots 134 run in an unattended manner in a virtual environment or on a physical machine and can automate many processes. Unattended robots 134 can be responsible for remote execution, monitoring, scheduling, and providing support for work queues. In some embodiments, debugging for all robot types can be run according to the designer 110. Both attended and unattended robots can automate various systems and applications, including but not limited to mainframes, web applications, VMs, enterprise applications (such as those provided by SAP), and other systems. ® 、SalesForce ® 、Oracle ® and computing system applications (such as desktop and notebook applications, mobile device applications, wearable computer applications, etc.).

[0052] The commander 120 may have various capabilities, including but not limited to provisioning, deployment, versioning, configuration, queueing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining a connection between a robot 130 and a commander 120 (e.g., a web application). Development may include ensuring that a package version is correctly delivered to an assigned robot 130 for execution. In some embodiments, versioning may include the management of a unique instance of a process or configuration. Configuration may include maintaining and delivering robot environments and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs into a database (e.g., a SQL database) and / or another storage mechanism (e.g., ElasticSearch). ® , which provides the ability to store and quickly query larger data sets). Director 120 can provide interconnectivity by serving as a centralized point of communication for third-party solutions and / or applications.

[0053] Robot 130 is an execution agent that runs the workflow built into designer 110. One commercial example of some embodiments of robot(s) 130 is UiPath Robots™. In some embodiments, robot 130 is installed by default with Microsoft Windows ® Services managed by the Service Control Manager (SCM). As a result, such robots 130 can open interactive Windows under the Local System account. ®session, and has Windows ® Permissions for services.

[0054] In some embodiments, a robot 130 can be installed in user mode. For such a robot 130, this means that it has the same permissions as the user who has installed the given robot 130. This feature can also be used by high-density (HD) robots, which ensure that each machine is fully utilized at its maximum potential. In some embodiments, any type of robot 130 can be configured in an HD environment.

[0055] In some embodiments, the robot 130 is divided into several components, each dedicated to a specific automation task. In some embodiments, the robot components include but are not limited to the robot service managed by the SCM, the user mode robot service, the executor, the agent, and the command line. The robot service managed by the SCM manages and monitors Windows ®

[0056] The SCM starts a session and acts as a delegate between the commander 120 and the execution host (ie, the computing system on which the robot 130 is executed). These services are trusted by the robot 130 and manage the credentials of the robot 130. The console application is launched by the SCM under the local system.

[0057] In some embodiments, the user-mode robot service manages and monitors Windows ® The user-mode robot service can be trusted by the robot 130 and manage the credentials of the robot 130. In the case where the SCM-managed robot service is not installed, Windows ® Applications can be started automatically.

[0058] The executor can be installed on Windows ® The executor can execute the workflow. The executor can know the dots per inch (DPI) setting of each monitor. The agent can be Windows ® A Presentation Foundation (WPF) application that displays available jobs in a system tray window. An agent can be a client of a service. An agent can request to start or stop jobs and change settings. A command line is a client of a service. A command line is a console application that can request to start a job and wait for its output.

[0059] As explained above, generally keeping the components of the robot 130 separate helps developers, support users, and computing systems more easily run, identify, and track what each component is executing. Special behaviors can be configured by component in this way, such as setting different firewall rules for executors and services. In some embodiments, the executor can always know the DPI setting of each monitor. As a result, regardless of the configuration of the computing system that created the workflow, the workflow can be executed at any DPI. In some embodiments, projects from the designer 110 can also be independent of the browser zoom level. In some embodiments, DPI can be disabled for applications that are DPI unaware or intentionally marked as unaware.

[0060] Figure 2 is an architectural diagram illustrating an RPA system 200 deployed according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be Figure 1 The RPA system 100 of the present invention or may be a part of the RPA system. It should be noted that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 210 includes an executor 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not be running on the computing system 210. The executor 212 is running a process. Figure 2 As shown in , several business items can be run simultaneously. In this embodiment, the agent 214 (such as Windows ® The Executor 212 service is a single point of contact for all executors 212. All messages in this embodiment are logged to the Director 230, which further processes the messages via the database server 240, the indexer server 250, or both. Figure 1 As discussed, effector 212 may be a robotic component.

[0061] In some embodiments, a robot represents an association between a machine name and a user name. A robot can manage multiple executors simultaneously. On a computing system that supports multiple interactive sessions running simultaneously (e.g., Windows ® Server 2012), multiple robots can run simultaneously, each in a separate Windows instance with a unique username. ® In the above text, this is referred to as the HD robot.

[0062] The agent 214 is also responsible for sending the status of the robot (e.g., periodically sending a "heartbeat" message indicating that the robot is still functioning) and downloading the required version of the package to be executed. In some embodiments, the communication between the agent 214 and the commander 230 is always initiated by the agent 214. In the notification scenario, the agent 214 can open a WebSocket channel, which is later used by the commander 230 to send commands to the robot (e.g., start, stop, etc.).

[0063] On the server side, a presentation layer (web application 232, Open Data Protocol (OData) Representational State Transfer (REST) ​​application programming interface (API) endpoint 234, and notification and monitoring 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240 and indexer server 250) are included. The commander 230 includes a web application 232, an OData REST API endpoint 234, notification and monitoring 236, and an API implementation / business logic 238. In some embodiments, most actions performed by a user in the interface of the commander 230 (e.g., via a browser 220) are performed by calling various APIs. Without departing from the scope of the present invention, such actions may include, but are not limited to, starting a job on a robot, adding / removing data from a queue, scheduling a job to run in an unattended manner, and the like. The web application 232 is the visual layer of the server platform. In this embodiment, the web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the present invention. In this embodiment, the user interacts with a web page from web application 232 via browser 220 to perform various actions to control commander 230. For example, the user can create a robot group, assign packages to robots, analyze logs for each robot and / or each process, start and stop robots, etc.

[0064] In addition to the web application 232, the director 230 also includes a service layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the present invention. The REST API is consumed by both the web application 232 and the agent 214. In this embodiment, the agent 214 is an overseer of one or more robots on a client computer.

[0065] In this embodiment, the REST API covers configuration, logging, monitoring, and queuing functionality. In some embodiments, the configuration endpoint can be used to define and configure application users, permissions, robots, assets, releases, and environments. The logging REST endpoint can be used to log different information, such as (for example) errors, explicit messages sent by the robot, and other environment-specific information. If the start job command is used in the commander 230, the deployment REST endpoint can be used by the robot to query the package version that should be executed. The queuing REST endpoint can be responsible for queue and queue item management, such as adding data to the queue, getting transactions from the queue, setting the status of transactions, etc.

[0066] The monitoring REST endpoint can monitor the web application 232 and the agent 214. The notification and monitoring API 236 can be a REST endpoint for registering the agent 214, delivering configuration settings to the agent 214, and for sending / receiving notifications from the server and the agent 214. In some embodiments, the notification and monitoring API 236 can also use WebSocket communication.

[0067] In this embodiment, the persistence layer includes a pair of servers - a database server 240 (e.g., SQL server) and an indexer server 250. In this embodiment, the database server 240 stores the configuration of robots, robot groups, associated processes, users, roles, schedules, etc. In some embodiments, this information is managed by the web application 232. The database server 240 can manage queues and queue items. In some embodiments, the database server 240 can store messages logged by the robot (in addition to or instead of the indexer server 250).

[0068] In some embodiments, the optional indexer server 250 stores information logged by the robot and indexes the information. In some embodiments, the indexer server 250 can be disabled through a configuration setting. In some embodiments, the indexer server 250 uses ElasticSearch

[0069] ®

[0070] , which is an open source project full-text search engine. Messages logged by the robot (e.g., using activities such as logging messages or writing lines) can be sent to the indexer server 250 through the logging REST endpoint(s), where they are indexed for future use.

[0071] Figure 3310, activities 320, 330 and drivers 340 according to an embodiment of the present invention. According to the above, the developer uses the designer 310 to develop a workflow executed by a robot. The workflow may include user-defined activities 320 and UI automation activities 330. Some embodiments are capable of identifying non-textual visual components in an image, referred to herein as computer vision (CV). Some CV activities related to such components may include, but are not limited to, clicking, typing, getting text, hovering, element existence, refreshing range, highlighting, etc. In some embodiments, clicking uses, for example, CV, optical character recognition (OCR), fuzzy text matching, and multi-anchors to identify an element, and clicks on the element. Typing can use the above content to identify an element and the type in the element. Getting text can identify the location of a specific text and use OCR to scan the location. Hovering can identify an element and hover over the element. Element existence can use the techniques described above to check whether an element exists on the screen. In some embodiments, there may be hundreds or even thousands of activities that can be implemented in the designer 310. However, any number and / or types of activities may be available without departing from the scope of the present invention.

[0072] UI automation activities 330 are a subset of special lower-level activities (e.g., CV activities) written using lower-level code and facilitate interaction with applications through the UI layer. In some embodiments, for example, UI automation activities 300 can simulate user input through window messages, etc. UI automation activities 330 support these interactions via drivers 340 that allow the robot to interact with the desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc.

[0073] Driver 340 can interact with the OS at a low level to find hooks, monitor keys, etc. Drivers can help with Chrome ® , IE ® , Citrix ® , SAP ® For example, the "click" activity plays the same role in these different applications via driver 340.

[0074] Figure 4 4 is an architectural diagram illustrating an RPA system 400 according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be or include Figure 1 and / or Figure 2The RPA system 100 and / or 200 of the present invention. The RPA system 400 includes a plurality of client computing systems 410 running robots. The computing systems 410 can communicate with a commander computing system 420 via a web application running thereon. The commander computing system 420 can in turn communicate with a database server 430 and an optional indexer server 440.

[0075] Relative to Figure 1 and Figure 3 It should be noted that although web applications are used in these embodiments, any suitable client and / or server software may be used without departing from the scope of the present invention. For example, the director may run a server-side application that communicates with a non-web-based client software application on a client computing system.

[0076] Figure 5 is an architectural diagram illustrating a computing system 500 configured to perform graphical element detection using a combination of UI descriptor attributes according to two or more graphical element detection techniques according to an embodiment of the present invention. In some embodiments, the computing system 500 may be one or more of the computing systems depicted and / or described herein. The computing system 500 includes a bus 505 or other communication mechanism for transmitting information, and (multiple) processors 510 coupled to the bus 505 to process information. The (multiple) processors 510 may be any type of general or special purpose processor, including a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The (multiple) processors 510 may also have multiple processing cores, and at least some of the cores may be configured to perform specific functions. In some embodiments, multi-parallel processing may be used. In some embodiments, at least one of the (multiple) processors 510 may be a neuromorphic circuit including a processing element that mimics a biological nerve. In some embodiments, the neuromorphic circuit may not require typical components of the von Neumann computing architecture.

[0077] The computing system 500 also includes a memory 515 for storing information and instructions to be executed by the processor(s) 510. The memory 515 may be composed of any combination of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage (such as a magnetic disk or optical disk), or any other type of non-transitory computer-readable medium, or a combination of the above. Non-transitory computer-readable media may be any available media that can be accessed by the processor(s) 510, and may include volatile media, non-volatile media, or both. The media may also be removable, non-removable, or both.

[0078] Additionally, the computing system 500 includes a communication device 520 (such as a transceiver) to provide access to a communication network via wireless and / or wired connections. In some embodiments, the communication device 520 can be configured to use frequency division multiple access (FDMA), single carrier FDMA (SC-FDMA), time division multiple access (TDMA), code division multiple access (CDMA), orthogonal frequency division multiplexing (OFDM), orthogonal frequency division multiple access (OFDMA), global system for mobile (GSM) communications, general packet radio service (GPRS), universal mobile telecommunications system (UMTS), cdma2000, wideband CDMA (W-CDMA), high speed downlink packet access (HSDPA), etc., without departing from the scope of the present invention. The communication device 520 may include one or more antennas that are singular, array, phased, switched, beamformed, beamsteered, combinations thereof, and / or any other antenna configurations without departing from the scope of the present invention.

[0079] The processor(s) 510 are further coupled to a display 525 via bus 505, such as a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, a Retina

[0080] ®

[0081] Display, in-plane switching (IPS) display, or any other suitable display for displaying information to a user. Display 525 can be configured as a touch (tactile) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, etc. using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse recognition, frustrated total internal reflection, etc. Any suitable display device and tactile I / O can be used without departing from the scope of the present invention.

[0082] A keyboard 530 and a cursor control device 535 (such as a computer mouse, touchpad, etc.) are further coupled to bus 505 to enable a user to interface with computing system 500. However, in some embodiments, a physical keyboard and mouse may not be present, and a user may interact with the device only through display 525 and / or touchpad (not shown). Any type and combination of input devices may be used as a matter of design choice. In some embodiments, there are no physical input devices and / or displays. For example, a user may interact with computing system 500 remotely via another computing system in communication with it, or computing system 500 may operate autonomously.

[0083] The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. The modules include an operating system 540 for the computing system 500. The modules also include a combined technology graphic element detection module 545 configured to perform all or part of the processes described herein or derivatives thereof. The computing system 500 may include one or more additional functional modules 550 that include additional functionality.

[0084] It should be understood by those skilled in the art that, without departing from the scope of the present invention, the "system" may be implemented as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device or combination of devices. Presenting the functions described above as being performed by a "system" is not intended to limit the scope of the present invention in any way, but is intended to provide an example of many embodiments of the present invention. In fact, the methods, systems, and apparatus disclosed herein may be implemented in localized and distributed forms consistent with computing technologies including cloud computing systems.

[0085] It should be noted that some of the system features described in this specification have been presented as modules in order to more specifically emphasize their implementation independence. For example, a module can be implemented as a hardware circuit including a custom very large scale integration (VLSI) circuit or gate array, an off-the-shelf semiconductor (such as a logic chip, a transistor, or other discrete components). A module can also be implemented in a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, a graphics processing unit, etc.

[0086] Modules may also be implemented at least in part in software so that they are executed by various types of processors. For example, the identified unit of executable code may include one or more physical blocks or logic blocks of computer instructions, which may be, for example, organized into objects, processes, or functions. Nevertheless, the executable file of the identified module does not need to be physically located in one place, but may include disparate instructions stored in different locations, which include the module and achieve the intended purpose of the module when they are combined together in a logical manner. In addition, modules may be stored on a computer-readable medium, which may be, for example, a hard disk drive, a flash memory device, a RAM, a magnetic tape, and / or any other such non-transient computer-readable medium for storing data, without departing from the scope of the present invention.

[0087] In fact, a module of executable code can be a single instruction or many instructions, and can even be distributed on several different code segments, distributed between different programs, and distributed across multiple memory devices. Similarly, operating data can be identified and illustrated as within a module in this article, and the operating data can be implemented in any suitable form and organized in any suitable type of data structure. Operating data can be collected as a single data set, or can be distributed in different locations included on different storage devices, and can exist at least in part only as an electronic signal on a system or network.

[0088] According to the above, some embodiments employ attributes according to two or more graphic element detection techniques to identify graphic elements in a UI. In some embodiments, these techniques may be configured for RPA workflow activities in an RPA designer application at design time. Figure 6 An example graphical element detection technology configuration interface for an RPA designer application 600 according to an embodiment of the present invention is illustrated. The RPA designer application 600 includes an RPA workflow development pane 610. When a user clicks on an activity that interacts with a graphical element in the UI, a graphical element detection technology configuration pane 620 shows possible graphical element detection technologies and corresponding properties for the activity.

[0089] The graphic element detection technology configuration pane 620 provides the RPA developer with the ability to select from a variety of graphic element detection technologies and their associated properties. Triangle 622 allows the RPA developer to show or hide the property list for each graphic element detection technology. Graphic element detection technology selection checkbox 624 allows the RPA developer to select or deselect all properties for a given graphic element detection technology at once. Property checkbox 626 allows the RPA developer to select or deselect the corresponding property.

[0090] In some embodiments, fuzzy matching can be employed for attributes. By selecting the fuzzy matching radio button 628, the RPA developer can cause the corresponding fuzzy matching threshold field 629 to appear. The RPA developer can then specify the threshold for fuzzy matching. In some embodiments, other interactive mechanisms can be used, including but not limited to sliders, manual editing of XML, etc. For example, a tag including an attribute such as the following can be displayed:

[0091] <wnd app='java.exe' cls='SunAwtFrame' title="SwingSet2"

[0092] matching:title='fuzzy' fuzzylevel:title='0.8' / >

[0093] Add the attributes "matching:title" and "fuzzylevel:title" to <wnd / > tag. The matching:title attribute has a value of "fuzzy" to indicate that the value of the "title" attribute of the current tag will be fuzzy matched at runtime. The value of the fuzzylevel:title attribute indicates the desired fuzzy similarity threshold specified by the developer (0.8 in this example). It should be appreciated by those of ordinary skill in the art that there may be many other ways of encoding the fuzzy flag specifying the selected attribute for fuzzy matching without departing from the scope of the present invention.

[0094] In some embodiments, the RPA designer application can provide the RPA developer with the ability to manually search for attributes. For example, the RPA designer application 600 includes an attribute search field 630 that can perform a search for attributes as the RPA developer types text. In some embodiments, a list of matching attributes (if any) can appear with each character typed.

[0095] In this example, the click activity 612 has been configured to click a check box. However, any suitable activity that attempts to interact with one or more graphical elements may be used and configured without departing from the scope of the present invention. The (multiple) graphical element detection techniques used for each suitable activity may be configured in this manner to perform the desired RPA action. In some embodiments, an RPA developer may choose to use only a single graphical element detection technique for one or more activities, and use multiple simultaneous graphical element detection techniques for one or more other activities. This provides a highly flexible RPA development tool.

[0096] Fig. 7A and Figure 7BAnother example graphical element detection technology configuration interface 700 for an RPA designer application according to an embodiment of the present invention is illustrated. Like the graphical element detection technology configuration interface 600, the graphical element detection technology configuration interface 700 includes an RPA workflow development pane 710 with a click activity 712, a graphical element detection technology configuration pane 720, and an attribute search field 730. However, in this embodiment, other graphical element detection technologies are nested within a unified target technology 721. If the unified target technology 721 is selected (e.g., in Fig. 7A ), then a unified goal approach is used that applies the other techniques serially for a period of time, in parallel, or applies at least one of the techniques and then employs at least one other technique in parallel if no match is found for a period of time.

[0097] If the unified target technology 721 is deselected, the developer can select and deselect the desired technology and / or attributes. Triangle 722 allows the RPA developer to show or hide the list of attributes for each graphic element detection technology. The graphic element detection technology selection check box 724 allows the RPA developer to select or deselect all attributes for a given graphic element detection technology at once. The attribute check box 726 allows the RPA developer to select or deselect corresponding attributes. In some embodiments, fuzzy matching can be used for attributes. By selecting the fuzzy matching radio button 728, the RPA developer can cause the corresponding fuzzy matching threshold field 729 to appear.

[0098] Figure 8 is a flow chart illustrating a process 800 for configuring an RPA workflow to perform graphical element detection using a combination of UI descriptor attributes according to two or more graphical element detection techniques according to an embodiment of the present invention. In some embodiments, process 800 may be performed by Figure 6 or Fig. 7A and Figure 7BThe RPA designer application 600 or 700 is executed. The process begins by receiving an activity selection in an RPA workflow at 810, which activity selection is to be configured to perform graphic element detection using a combination of UI descriptor attributes according to two or more graphic element detection technologies. In some embodiments, at 820, the type of graphic element detection technology and / or UI descriptor attributes are automatically selected based on the action implemented by the activity (e.g., click, get text, hover, etc.), the graphic element type (e.g., button, text field, etc.) and / or the specific graphic element indicated by the RPA developer (i.e., which element the user selected on the screen and which other elements exist in the application). With respect to a specific graphic element, for example, if the RPA developer clicks an OK button, but there are two OK buttons on the screen, some attributes can be automatically added to distinguish between two otherwise identical OK buttons. For example, when a UI tree is used, the UI tree is usually constructed in this way: when the RPA developer indicates a graphic element on the screen, at least some attributes in the UI tree are different for the case of the graphic element compared to the case for other graphic elements. At 830 , the RPA designer application may receive selections and / or modifications of properties of the graphical element detection technique from the RPA developer in order to custom configure the activity.

[0099] Then at 840, the RPA designer application configures the activity based on the selected graphical element detection technique and attributes. If more activities are to be configured, then the RPA developer can select another activity and the process returns to step 810. After the desired activity(s) are configured, at 850, the RPA designer application generates an RPA robot to implement the RPA workflow including the configured activity(s). The process then ends or continues to Fig. 9 .

[0100] Fig. 9 is a flow chart illustrating a process 900 for performing graphical element detection using a combination of UI descriptor attributes according to two or more graphical element detection techniques according to an embodiment of the present invention. In some embodiments, process 900 may be performed at runtime by Figure 6 or Fig. 7A and Figure 7B The RPA designer application 600 or 700 is implemented by the RPA robot created. The process begins by analyzing the UI (e.g., a screenshot, an image of an application window, etc.) at 910 to identify UI element attributes. The UI element attributes may include, but are not limited to, images, text, relationships between graphic elements, hierarchical representations of graphic elements in the UI, etc. The identification may be performed via CV, OCR, API calls, analysis of text files (e.g., HTML, XML, etc.), combinations thereof, etc.

[0101] After the UI has been analyzed, at 920, the UI element attributes are compared with the configured UI descriptor attributes of the two or more graphic element detection techniques for the activity. For example, this can be based on Figure 8 The configuration of the activity in step 840 of FIG. 840 is performed. If a UI element matching the attributes of the graphic element detection technology is found via an exact match or a threshold match at 930, then an action associated with the activity involving the UI element (e.g., clicking a button, entering text, interacting with a menu, etc.) is performed at 940. If there are more activities at 950, then the process continues to step 920 for the next activity. However, if no UI element matching the attributes of the graphic element detection technology is found at 930, then an exception is thrown at 960 and the process ends.

[0102] According to an embodiment of the present invention, Figure 8 and Fig. 9 The process steps performed in the embodiment may be performed by a computer program encoded with instructions for causing a processor(s) to perform Figure 8 and Fig. 9 The computer program may be implemented on a non-transitory computer readable medium. The computer readable medium may be, but is not limited to, a hard drive, a flash memory device, a RAM, a magnetic tape, and / or any other such medium or combination of media for storing data. The computer program may include a program for controlling the processor(s) of a computing system (e.g. Figure 5 The processor(s) 510 of the computing system 500 implements Figure 8 and Fig. 9 The coded instructions of all or part of the process steps described in the present invention may also be stored on a computer-readable medium.

[0103] The computer program may be implemented in hardware, software or a hybrid implementation. The computer program may consist of modules that are operatively in communication with each other and are designed to pass information or instructions for display. The computer program may be configured to operate on a general purpose computer, an ASIC or any other suitable device.

[0104] It will be readily understood that, as generally described and illustrated in the figures herein, the components of the various embodiments of the present invention may be arranged and designed in a variety of different configurations. Therefore, as represented in the accompanying drawings, the detailed description of the embodiments of the present invention is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention.

[0105] The features, structures, or characteristics of the present invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, throughout this specification, references to "certain embodiments," "some embodiments," or similar language refer to specific features, structures, or characteristics described in conjunction with the embodiments being included in at least one embodiment of the present invention. Therefore, throughout this specification, the appearance of the phrases "in certain embodiments," "in some embodiments," "in other embodiments," or similar language do not necessarily all refer to the same set of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0106] It should be noted that throughout this specification, references to features, advantages, or similar language do not mean that all features and advantages that can be achieved with the present invention should be or are in any single embodiment of the present invention. Instead, language referring to features and advantages is understood to mean that a particular feature, advantage, or characteristic described in conjunction with an embodiment is included in at least one embodiment of the present invention. Therefore, throughout this specification, discussions of features and advantages and similar language may, but do not necessarily, refer to the same embodiment.

[0107] In addition, the described features, structures, and characteristics of the present invention may be combined in any suitable manner in one or more embodiments. It will be appreciated by those skilled in the relevant art that the present invention may be practiced without one or more of the specific features or advantages of a particular embodiment. In other cases, additional features and advantages that may not be present in all embodiments of the present invention may be identified in certain embodiments.

[0108] It will be readily understood by those skilled in the art that the invention as described above may be practiced using steps in a different order and / or using hardware elements in a configuration different from that disclosed. Therefore, although the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative constructions will be apparent while still being within the spirit and scope of the invention. Therefore, in order to determine the limits and boundaries of the invention, reference should be made to the appended claims.

Claims

1. A computer-implemented method for detecting a graphical element in a user interface (UI), comprising: A selection interface is provided by the designer application, the selection interface supporting selection of a graphic element detection technology from a plurality of graphic element detection technologies on a per-activity basis for use in a separate activity of a robotic process automation (RPA) workflow; receiving, by the designer application, a selection of an activity to be configured in the RPA workflow from the selection interface; Receiving, by the designer application, a selection and / or modification of the following for the selected activity: one or more graphic element detection technologies among the plurality of graphic element detection technologies, one or more UI descriptor attributes of the one or more graphic element detection technologies, or a combination thereof; as well as The selected activity is configured by the designer application based on the received selections and / or modifications.

2. The computer-implemented method of claim 1 , further comprising: One or more of the plurality of graphical element detection techniques, one or more of the UI descriptor attributes, or both are automatically configured by the designer application.

3. A computer-implemented method according to claim 2, wherein the automatic configuration is performed by the designer application based on: the action implemented by the activity, the type of target graphical element, the presence of one or more other graphical elements in the UI, or a combination of the above.

4. The computer-implemented method of claim 1 , wherein the process of claim 1 is repeated for at least one additional activity in the RPA workflow.

5. The computer-implemented method of claim 1 , further comprising: An RPA robot is generated by the designer application, and the RPA robot is configured to implement the configured activity.

6. The computer-implemented method of claim 5, wherein the RPA robot is configured to use a subset of the UI descriptor attributes for at least one graphical element detection technique among the plurality of graphical element detection techniques.

7. The computer-implemented method of claim 5, further comprising: The RPA robot analyzes the UI at runtime to identify UI element attributes; comparing, by the RPA robot, the UI element attributes with the configured UI descriptor attributes for the activity; and When a UI element matching the attributes of the plurality of graphical element detection techniques is found via an exact match or a threshold match: An action associated with the activity involving the UI element is taken by the RPA robot.

8. The computer-implemented method of claim 7, wherein the UI element attributes include images, text, relationships between graphical elements in the UI, a hierarchical representation of the graphical elements in the UI, or a combination thereof.

9. The computer implemented method of claim 7, wherein the process of claim 7 is repeated for at least one additional activity.

10. The computer-implemented method of claim 1, wherein the types of UI descriptors include two or more of: a selector, a computer vision (CV) descriptor, an image matching descriptor, and an optical character recognition (OCR) descriptor.

11. A non-transitory computer readable medium comprising a computer program configured to cause at least one processor to: Providing a selection interface that supports selecting a graphic element detection technology from a plurality of graphic element detection technologies on a per-activity basis for use in a separate activity of a robotic process automation (RPA) workflow; receiving a selection of an activity to be configured in the RPA workflow from the selection interface; Receiving a selection and / or modification of: one or more of the plurality of graphic element detection techniques, one or more UI descriptor attributes of the plurality of graphic element detection techniques, or a combination thereof; as well as The selected activity is configured based on the received selections and / or modifications.

12. The non-transitory computer readable medium of claim 11, wherein the computer program is further configured to cause the at least one processor to: One or more of the plurality of graphical element detection techniques, one or more of the UI descriptor attributes, or both are automatically configured.

13. The non-transitory computer-readable medium of claim 12, wherein the automatic configuration is performed based on: an action implemented by the activity, a type of a target graphical element, the presence of one or more other graphical elements in the UI, or a combination thereof.

14. The non-transitory computer readable medium of claim 11, wherein the process of claim 11 is repeated for at least one additional activity.

15. The non-transitory computer readable medium of claim 11, wherein the computer program is further configured to cause the at least one processor to: An RPA robot is generated, wherein the RPA robot is configured to implement the configured activity.

16. The non-transitory computer-readable medium of claim 15, wherein the RPA robot is configured to use a subset of the UI descriptor attributes for at least one graphical element detection technique among the plurality of graphical element detection techniques. 17 . The non-transitory computer-readable medium of claim 11 , wherein the type of UI descriptor comprises two or more of: a selector, a computer vision (CV) descriptor, an image matching descriptor, and an optical character recognition (OCR) descriptor.

18. A non-transitory computer readable medium storing a computer program, the computer program being configured to cause at least one processor to: Analyze the user interface UI at runtime to identify UI element attributes; comparing the UI element attributes with UI descriptor attributes according to a plurality of graphic element detection techniques for an activity of a Robotic Process Automation (RPA) workflow; and When a UI element matching the attributes of the plurality of graphical element detection techniques is found via an exact match or a threshold match: An action associated with the activity involving the UI element is taken.

19. The non-transitory computer-readable medium of claim 18, wherein the UI element attributes include images, text, relationships between graphical elements in the UI, a hierarchical representation of the graphical elements in the UI, or a combination thereof.

20. The non-transitory computer readable medium of claim 18, wherein the process of claim 18 is repeated for at least one additional activity.

Images