User Interface (UI) Mapper for Robotic Process Automation

Abstract

A user interface (UI) mapper for robotic process automation (RPA) is disclosed. The UI mapper can initially capture UI elements for faster fetching of UI elements for later use, allowing RPA developers to "map" UI elements for automating applications. This can enable subsequent developers, who may not have programming knowledge, to build RPA workflows using these predefined "target" UI elements.

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Nonprovisional Patent Application No. 17 / 019,679, filed September 14, 2020. The subject matter of this previously filed application is hereby incorporated by reference in its entirety.

[0002] The present invention relates generally to robotic process automation (RPA), and more specifically to a user interface (UI) mapper for RPA. [Background technology]

[0003] For RPA automation with a user interface (UI), developers typically build a descriptor (e.g., a selector) for each UI action. However, initially there are typically no descriptors or other UI objects. Growing a repository of UI objects organically (i.e., item-by-item, process-by-process, and record-by-record) is slow. Therefore, an improved approach can be beneficial. Summary of the Invention

[0004] Certain embodiments of the present invention may provide solutions to problems and needs in the field that have not yet been fully identified, appreciated, or solved by current RPA technology. For example, some embodiments of the present invention relate to UI mappers for RPA.

[0005] In an embodiment, a computer program for providing a UI mapper configured to map UI elements is stored on a non-transitory computer-readable medium. The computer program is configured for at least one processor to provide an interface that facilitates indication of UI elements in a screen of an application. The computer program is also configured for at least one processor to receive one or more indications of the UI elements in the screen and include the one or more indicated UI elements in a UI tree of the application. The UI mapper is configured to operate in a live mode that provides the interface and receives the indication of the UI elements on the screen. The UI mapper is also configured to operate in an offline mode, where the UI mapper facilitates selection of a different screen.

[0006] In another embodiment, a computer-implemented method for mapping UI elements for RPA includes providing, by a UI mapper, an interface that facilitates indication of UI elements in a screen of an application. The computer-implemented method also includes receiving, by the UI mapper, one or more indications of UI elements in the screen and including, by the UI mapper, the one or more indicated UI elements in a UI tree of the application. The computer-implemented method further includes adding, by the UI mapper or a UI object browser, the one or more indicated UI elements to an object repository for later reuse. The UI mapper includes a UI descriptor pane configured with one or more applications, one or more respective screens per application, and one or more respective UI elements for each screen included in the UI tree.

[0007] In yet another embodiment, a computer program for providing a UI mapper configured to map UI elements is stored on a non-transitory computer-readable medium. The computer program is configured for at least one processor to provide an interface facilitating indication of UI elements in a screen of an application. The computer program is also configured for the at least one processor to receive one or more indications of UI elements in the screen and include the one or more indicated UI elements in a UI tree of the application. The computer program is further configured for the at least one processor to display a properties pane including properties of the selected application, screen, or UI element. When a UI element is selected, the properties pane includes properties of a target UI element and one or more UI descriptors for identifying the target UI element. [Brief description of the drawings]

[0008] So that the advantages of certain embodiments of the invention may be readily understood, a more particular description of the invention briefly described above will now be depicted with reference to specific embodiments which are illustrated in the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not to be considered as limiting its scope, but the invention will be depicted and explained with further specificity and detail through the use of the following accompanying drawings, in which:

[0009] [Figure 1] FIG. 1 is an architectural diagram illustrating a robotic process automation (RPA) system in accordance with an embodiment of the present invention.

[0010] [Diagram 2] FIG. 1 is an architectural diagram illustrating a deployed RPA system according to an embodiment of the present invention.

[0011] [Diagram 3]FIG. 2 is an architecture diagram illustrating the relationships between designers, activities, and drivers according to an embodiment of the present invention.

[0012] [Figure 4] FIG. 1 is an architectural diagram illustrating an RPA system according to an embodiment of the present invention.

[0013] [Diagram 5] FIG. 1 is an architectural diagram illustrating a computing system configured to provide a UI mapper for RPA in accordance with an embodiment of the present invention.

[0014] [Figure 6A] 1 illustrates a UI mapper in a collapsed mode, according to an embodiment of the present invention.

[0015] [Figure 6B] 1 illustrates a UI mapper in expanded mode, according to an embodiment of the present invention.

[0016] [Figure 6C] 1 illustrates a live application screen with no UI elements selected, according to an embodiment of the present invention.

[0017] [Figure 6D] 1 illustrates a live application screen after a target and anchor UI element have been selected, according to an embodiment of the present invention.

[0018] [Figure 7] 11 is a flowchart illustrating a UI mapper process in accordance with an embodiment of the present invention.

[0019] Unless otherwise noted, like reference characters denote corresponding features consistently throughout the attached drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Detailed Description of the Embodiments Some embodiments relate to a UI mapper for RPA. The UI mapper may capture UI elements first to fetch them faster for later use. The UI mapper may "map" UI elements for an RPA developer to automate an application, allowing a later developer, who may not have programming knowledge, to build an RPA workflow using these predefined "target" UI elements.

[0021] UI elements on a screen (e.g., text fields, buttons, labels, menus, checkboxes, etc.) may be grouped by application, application version, application screen, and collection of UI elements. Each screen typically has multiple UI elements. As used herein, a "screen" refers to a state of an application. This state may have a particular image or appearance at a time, and a hierarchical structure of elements, such as UI elements, containers, etc., may exist "under the hood." An "application" or version of a given application may be a combination of screens in this context. Each UI element may be described by one or more UI descriptors in some embodiments.

[0022] UI elements, applications, and application screens are UI objects. UI elements, applications, and application screens may have a particular ordered structure in some embodiments. For example, an application may have an application screen, and an application screen may have UI elements. In some embodiments, an application may have multiple application screens, and each screen may have multiple UI elements. In some embodiments, UI elements and screens may be further differentiated into particular types of UI elements (e.g., buttons, checkboxes, text fields, etc.) and screens (e.g., top windows, modal windows, pop-up windows, etc.).

[0023] The UI object browser may access libraries of UI objects that may be grouped by application, application version, application screen, collection of UI elements, combinations thereof, etc. In some embodiments, the UI object libraries may be stored in a UI object repository. As used herein, a UI object repository is a collection of UI object libraries. In some embodiments, the UI object repository may be a NuGet™ feed, a web service, etc.

[0024] The UI object browser, in some embodiments, may be used to navigate, manage, and edit UI object libraries in a UI object repository. The UI object repository of a UI object library in some embodiments may facilitate management, reuse, and improved reliability of UI descriptors in a project. In some embodiments, UI descriptors may be added to a UI library and published or republished for global reuse in the UI object repository. The UI object browser may facilitate reuse of the UI element specification framework and its derivatives by providing access to the UI object repository and its UI object libraries.

[0025] To make the UI objects reusable, they may be extracted into a UI object library that can be referenced by RPA processes. For example, if a selector or other UI descriptor changes due to a new version of an application, the library may be recreated (or republished) to include the changed UI descriptor. RPA processes that use the UI object library may then call the changed version of the UI descriptor. References to new application programming interfaces (APIs) associated with the new library may be changed manually for the RPA process by a developer, through a software tool (e.g., a tool that goes through the RPA process and updates the UI descriptor references when the selector or other components change), or in some embodiments automatically by any suitable mechanism.

[0026] In some embodiments, a package of UI objects may be attached as a dependency (e.g., a NuGet™ package). However, in certain embodiments, the UI objects may be obtained via one or more API calls to a web service, allowing the UI objects to be stored remotely and retrieved and used during automation.

[0027] UI elements added to the UI object library may represent selectors to use in the RPA workflow. To automate specific actions in the UI, the RPA robot may interact with various windows, buttons, drop-down lists, and / or other graphical elements. Typically, this is done using the expected screen positions of the UI elements. However, this is unreliable.

[0028] Some embodiments use selectors that overcome the problems associated with identifying fixed coordinates by storing attributes of UI elements and their parents in an Extensible Markup Language (XML) fragment. In some embodiments where the UI is static, selectors may be generated automatically, but in some software programs, such as some web applications, layouts change and attribute nodes exist that have volatile values. These changes may not be easily predicted, which previously required some selectors to be generated manually. However, the object browser of some embodiments may overcome this problem.

[0029] A selector is a type of UI descriptor that can be used to find UI elements in some embodiments. A selector, in some embodiments, has the following structure: <node_1 / ><node_2 / > ...<node_N / >

[0030] The final node represents the target GUI element, and all previous nodes represent the parents of that element.<node_1> is usually called the root node and represents the top window of the application.

[0031] Each node may have one or more attributes that aid in the correct identification of the particular level of the selected application. Each node, in some embodiments, has the following format: <ui_system attr_name_1=’attr_value_1’...attr_name_N=’attr_value_N’ / >

[0032] All attributes may have values ​​assigned to them, and attributes with constant values ​​may be chosen because changing the value of an attribute every time an application launches may cause the selector to not properly identify the associated element.

[0033] UI object library descriptors can be added directly to RPA workflow activities, saving developers time that would otherwise be required to create custom selectors for the activities. The object browser can provide a database that stores the created selectors in an object library, allowing reuse of UI descriptors. An object library is defined herein as a collection of UI descriptors that correspond to one or more screens from a particular version of an application. A UI descriptor is a set of instructions for locating UI elements. A UI descriptor in some embodiments is an encapsulated data / structure format that includes UI element selector(s), anchor selector(s), computer vision (CV) descriptor(s), integrated target descriptor(s), screen image capture (context), element image capture, other metadata (e.g., application and application version), combinations thereof, and the like. The encapsulated data / structure format may be extensible by future updates to the platform and is not limited to the above definition. Any suitable UI descriptor for identifying UI elements on a screen may be used without departing from the scope of the present invention. An integrated target descriptor concatenates multiple types of UI descriptors. The integrated target descriptor may function like a finite state machine (FSM): in a first context, a first UI descriptor mechanism is applied, in a second context, a second UI descriptor is applied, and so on.

[0034] In some embodiments, the RPA designer application may ask the user what type(s) of application(s) they plan to automate. For example, the user may specify Excel®, Workday®, SAP®, etc. The RPA application already contains screen descriptors in the UI object library for these applications, so the application may be programmed with logic on how to automate these applications. Out-of-the-box selectors for different versions of these applications may be available, and the user may be able to specify which version(s) to automate.

[0035] In some embodiments, fuzzy matching may be employed, where one or more attributes must match within a certain range and with a certain accuracy (e.g., 70% match, 80% match, 99% match, etc.) using a string metric (e.g., Levenshtein distance, Hamming distance, Jaro-Winkler distance, etc.), combinations thereof, etc. One skilled in the art will appreciate that a similarity measure can quantify not only the amount of similarity, but also the amount of mismatch between two attribute values. Additionally, in various embodiments, a similarity threshold may represent a maximum amount of mismatch or a minimum amount of similarity required for a match.

[0036] Depending on the selected method of calculating the similarity measure, the similarity threshold may have various interpretations. For example, the similarity threshold may indicate the maximum count of characters that may differ between the two strings, or may indicate the fractional degree of mismatch calculated as a percentage of the total count of characters (e.g., the length of the combined strings). In some embodiments, the similarity threshold may be rescaled to a predefined interval, such as between 0 and 1, between 0 and 100, between 7 and 34, etc. In one non-limiting example, a relatively high similarity threshold (e.g., close to 1 or 100%) indicates a requirement of near perfect match, i.e., the values ​​of the fuzzy attributes in the run-time target are allowed to deviate only very slightly from the values ​​of each attribute in the design-time target. On the other hand, if the similarity threshold is relatively low (e.g., close to 0), nearly all values ​​of the respective fuzzy attributes are considered to match.

[0037] In certain embodiments, matching tolerances may vary by attribute criteria. For example, an exact match may be required for one or more attributes (e.g., one may wish to find a specific exact name), and fuzzy matching may be performed for one or more other attributes. The number and / or type of attributes used from each graphical element detection technique may be custom specified by the RPA developer in some embodiments.

[0038] In some embodiments, attributes may be stored as attribute-value pairs and / or attribute-value-tolerance pairs (e.g., fuzzy matching). The attribute-value pairs may, in some embodiments, indicate the name and type of the UI element represented by the respective node. However, one skilled in the art will appreciate that there may be multiple ways to represent the location of a particular node in a UI tree other than as a list of attribute-value pairs without departing from the scope of the present invention.

[0039] These attribute-value pairs and / or attribute-value-tolerance pairs may be stored in tags in some embodiments, where each tag may include a string of characters with the sequence bookended by an implementation-specific delimiter (e.g., starting with "<" and ending with " / >"). The attribute-value pairs may, in some embodiments, indicate the name and type of the UI element represented by the respective node. However, one skilled in the art will appreciate that there may be multiple ways to represent the location of a particular node in a UI tree other than as a list of attribute-value pairs without departing from the scope of the present invention.

[0040] To enable successful and ideally unambiguous identification by the RPA robot, in some embodiments, each UI element is represented using an element ID that characterizes the respective UI element. In some embodiments, the element ID indicates the location of the target node in the UI tree, where the target node represents the respective UI element. For example, the element ID may identify the target node / UI element as a member of a subset of selected nodes. The subset of selected nodes can form a genealogy, i.e., a descent through the UI tree, where each node is either an ancestor or a descendant of another node.

[0041] In some embodiments, the element IDs include an ordered sequence of node indicators that trace a genealogical path through the UI tree, with the path terminating at a respective target node / UI element. Each node indicator may represent a member of the respective UI object hierarchy and its position in the sequence that is consistent with the respective hierarchy. For example, each member of the sequence may represent a descendant (e.g., a child node) of the previous member, which in turn is a descendant (e.g., a child node) of the next member, etc. In one HyperText Markup Language (HTML) example, element IDs representing individual form fields may indicate that each form field is a child of an HTML form, which in turn is a child of a particular section of a web page, etc. The genealogy need not be complete in some embodiments.

[0042] In some embodiments, one or more multi-anchor matching attributes may be used. Anchors are other UI elements that may be used to help uniquely identify a target UI element. For example, if a UI contains multiple text fields, searching the text fields alone is not sufficient to uniquely identify a given text field. Thus, in some embodiments, additional information is sought to uniquely identify a given UI element. Using the example of a text field, to the right of a label that reads "First Name," a text field for entering a first name may appear. This first name label may be set as an "anchor" to help uniquely identify the text field that is the "target."

[0043] In some embodiments, various positions and / or geometric associations between targets and anchors may be used, potentially within one or more tolerances, to uniquely identify a target. For example, the centers of the bounding boxes of the anchor and target may be used to define a line segment. This line segment may then be required to have a certain length within a tolerance and / or slope within a tolerance to uniquely identify the target using the target / anchor pair. However, any desired positions of positions associated with targets and / or anchors may be used in some embodiments without departing from the scope of the invention. For example, the points for drawing the line segments may be in the center, top left corner, top right corner, bottom left corner, bottom right corner, any other position on the boundary of the bounding box, any position within the bounding box, a position outside the bounding box, etc., as specified in conjunction with the bounding box characteristics. In certain embodiments, the target and one or more anchors may have different positions within or outside their bounding boxes that are used for geometric matching.

[0044] As can be seen, a single anchor may not always be sufficient to uniquely identify a target element on the screen with a certain degree of reliability. For example, consider a web form that displays two text fields for entering a first name, each to the right of a label "First Name" at different locations on the screen. In this example, one or more additional anchors may be useful in uniquely identifying a given target. Geometric properties between the anchor and the target (e.g., length, angle, and / or relative position of a line segment with a tolerance) may be used to uniquely identify the target. The user may be required to continue adding anchors until the match strength for the target exceeds a threshold.

[0045] As used herein, the terms "user" and "developer" are used interchangeably. A user / developer may or may not have programming and / or technical knowledge. For example, in some embodiments, a user / developer may create an RPA workflow by configuring activities within the RPA workflow without manual coding. In certain embodiments, this may be done, for example, by clicking and dragging and dropping various functions.

[0046] UI descriptors may be extracted from activities in an RPA workflow and added to a structured schema that groups UI descriptors by UI application, screen, and UI element. The UI descriptors may be part of one project for widespread reuse, part of a global repository for testing purposes, or part of a UI object library for global cross-project sharing in some embodiments. The object library in some embodiments is an encapsulation of UI descriptors grouped by application, application version, and screen. UI descriptors may be defined and added to the object library, which in some embodiments may be installed in other projects as a dependency after publishing. This allows the object library of UI descriptors to be reused. The UI application in some embodiments is a target application that has multiple versions, each version with multiple screens.

[0047] The UI Object Repository and reuse of UI descriptors can be beneficial for a variety of reasons. For example, if the location, appearance, and / or functionality of a UI element in an application changes, the UI descriptor can be modified and the changes can then be propagated to activities that use the UI descriptor. In this way, reusability can be provided at the specific level of the graphical elements in the UI.

[0048] This may be particularly beneficial to accommodate new versions of an application. For a new version, software developers only need to update the UI descriptors and / or develop a small number of new selectors from scratch, significantly reducing development time. For example, for a web page, the HyperText Markup Language (HTML) may be inspected to obtain paths to UI elements for corresponding selectors. The selectors of the corresponding UI elements may be different for different web browsers and / or different versions of the same web browser. This concept may also be applied to visual desktop, server, smartphone, and tablet applications. The UI object library may include the actual selectors for the application, which is just one type of UI descriptor, as described above. The UI object library may have multiple UI descriptors of different types for one or more applications. The UI descriptors may refer to different technologies on which the application is built. For example, different UI descriptors may be created and used for standalone desktop, web, and mobile applications. In one technology you may want to use selectors, in another you may want to use CV descriptors, etc.

[0049] The UI descriptor, in some embodiments, may work with a unified target that encompasses all UI detection mechanisms by which image detection and definition are performed. The unified target may merge multiple techniques for identifying and automating UI elements into a single cohesive approach. The unified target may prioritize selector-based and driver-based UI detection mechanisms and fall back to CV to find images if the first two mechanisms are not successful. In some embodiments, the selector editor and UI explorer may be configured to support the unified target.

[0050] In some embodiments, the designer application includes a section or panel as an "object browser" that loads object libraries from one or more sources. Using this object browser, a user can visualize the entire set of UI objects and use any UI descriptor through drag and drop and wizards or contextual actions. This can allow a user to select an action to be performed in an RPA workflow activity, such as clicking, getting text, typing, etc.

[0051] An example of the structure of an object library is shown below. ·SAP Version 1 Screen 1 Object 1 Object 2 ·… Screen 2… ·… Version 2… ·… ·Salesforce… ·…

[0052] It should be noted that the above hierarchy is provided as an example only. Any desired number of levels in the hierarchy and elements associated with each level may be used without departing from the scope of the present invention. In certain embodiments, a user may define the application map as he or she desires. Additionally, some nodes in the tree may, in some embodiments, only serve an administrative role and not a functional role. For example, UI elements may be grouped together in a container that does not have a UI descriptor. The container may, in some embodiments, exist solely for grouping purposes.

[0053] In some embodiments, if the OLDB is configured in UiPath Studio™, for example, "Show on Screen" may look at the OLDB and pick up a UI descriptor if available. If a UI descriptor is not available, a UI descriptor may be defined by the user by indicating the missing UI elements. A UI descriptor may then be generated and exposed in the UI object library.

[0054] The use of UI Descriptors and UI Object Repositories in some embodiments creates relationships and data that can be further expanded. In some embodiments, UI steps from an automated process can be mapped to a single screen. For example, a screen containing a form where a user adds and captures some data and then hits submit can be documented with one screenshot that contains a screen capture and each UI step performed on that screen. For example: (1) amount copied; (2) number copied; (3) name added; (4) click "OK".

[0055] User Interface (UI) Mapper

[0056] When a screen has many different UI elements, fetching them one by one for each screen can be time consuming. Therefore, some embodiments employ a UI mapper that captures multiple UI objects from the screens of an application at once. This can make fetching UI objects for subsequent screens faster. Some embodiments can use computer vision (CV) to automatically capture the screens into an object repository and use a UI mapper to add UI elements from the screens to the object repository.

[0057] In the first phase, the user may manually capture UI elements and / or use computer vision (CV) to detect UI elements on the screen and suggest them to the user. The user may manage the UI elements by rearranging them, grouping them into windows, providing UI element scopes other than the screen level, etc. With respect to using other UI element scopes, situations may arise where the user wishes to use one or more containers for its grouped components rather than the entire application screen. For example, a container could be an individual pane within a screen, a dropdown of a menu, etc.

[0058] In some embodiments, the UI Mapper is a tool that allows an RPA developer to quickly build a repository of UI objects and arrange them into a hierarchy of applications and screens so that the UI objects can be easily used in later automations. In some embodiments, the UI Mapper can be launched from the toolbar of the Object Browser. The UI Mapper tool is, from a high-level perspective, like a combination of an Object Browser and a Selection Screen. The Object Browser can be used as a library / tree for organizational purposes, while the Selection Screen and the Properties Pane (a.k.a. "helpers") are used to configure individual targets. Adding objects to the repository can be done in bulk or individually in some embodiments. Either approach can allow for advanced editing of UI object settings. In some embodiments, users can add applications, screens, and other organizational groupings, add UI elements in bulk or one at a time, delete UI objects (e.g., UI elements, screens, applications, groups, etc.), edit existing UI elements (e.g., add / remove anchors, edit selectors, etc.), highlight UI elements (e.g., one UI element, multiple UI elements, all UI elements, etc.), interact with the target application, etc.

[0059] Certain embodiments may be employed in Robotic Process Automation (RPA). FIG. 1 is an architectural diagram illustrating an RPA system 100, according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that enables developers to design and implement workflows. The designer 110 provides solutions for application integration, as well as automating third-party applications, management information technology (IT) tasks, and business IT processes. The designer 110 may facilitate the development of automation projects, which are graphical representations of business processes. In simple terms, the designer 110 facilitates the development and deployment of workflows and robots.

[0060] Automation projects enable rule-based process automation by giving developers control over the order of execution and relationships between custom sets of steps developed in workflows, defined herein as "activities." One commercial example of an embodiment of the Designer 110 is UiPath Studio™. Each activity may include an action such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0061] The types of workflows may include, but are not limited to, sequences, flowcharts, FSMs, and / or global exception handlers. Sequences may be particularly suitable for linear processes, allowing flow from one activity to another without cluttering the workflow. Flowcharts may be particularly suitable for more complex business logic, allowing for the integration of decisions and the connection of activities in more diverse ways through multiple branching logic operators. FSMs may be particularly suitable for large workflows. FSMs may use a finite number of states triggered by conditions (i.e., transitions) or activities during their execution. Global exception handlers may be particularly suitable for determining the behavior of a workflow when an execution error is encountered, or for debugging the process.

[0062] Once a workflow is developed in Designer 110, the execution of the business process is orchestrated by Conductor 120, which coordinates one or more Robots 130 that execute the workflow developed in Designer 110. One commercial example of an embodiment of Conductor 120 is UiPath Orchestrator™. Conductor 120 facilitates management of the creation, monitoring, and deployment of resources in an environment. Conductor 120 may act as, or one of, the integration points with third-party solutions and applications.

[0063] The conductor 120 may manage all the robots 130, connecting and executing them from a central point. Types of robots 130 that may be managed include, but are not limited to, attended robots 132, unattended robots 134, development robots (similar to unattended robots 134 but used for development and testing purposes), and non-production robots (similar to attended robots 132 but used for development and testing purposes). Attended robots 132 may be triggered by user events or scheduled to occur automatically and may operate side-by-side with humans on the same computing system. Attended robots 132 may be used with the conductor 120 for a centralized process deployment and logging medium. Attended robots 132 may assist human users in accomplishing various tasks and may be triggered by user events. In some embodiments, processes cannot be started from the conductor 120 on this type of robot and / or they cannot be run under a locked screen. In certain embodiments, the attended robot 132 may only be launched from the robot tray or from a command prompt. The attended robot 132 preferably operates under human supervision in some embodiments.

[0064] Unattended robots 134 can run unattended on virtual environments or physical machines and automate many processes. Unattended robots 134 can be responsible for providing remote execution, monitoring, scheduling, and work queue support. Debugging for all robot types can be performed from the designer 110 in some embodiments. Both attended and unattended robots can automate a variety of systems and applications, including, but not limited to, mainframes, web applications, VMs, enterprise applications (e.g., those produced by SAP®, SalesForce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0065] The conductor 120 may have various capabilities including, but not limited to, provisioning, deployment, versioning, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between the robots 130 and the conductor 120 (e.g., web applications). Deployment may include ensuring correct delivery of package versions to the robots 130 assigned for execution. Versioning may include managing unique instances of some processes or configurations in some embodiments. Configuration may include maintaining and delivering robot environments and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot specific data and maintaining user permissions. Logging may include storing and indexing logs in a database (e.g., an SQL database) and / or another storage mechanism (e.g., ElasticSearch®, which provides the ability to store large data sets and perform queries quickly). The conductor 120 may provide interconnectivity by operating as a central point of communication for third-party solutions and / or applications.

[0066] The robot 130 is an execution agent that executes the workflow built in the designer 110. One commercial example of some embodiments of the robot(s) 130 is UiPath Robots™. In some embodiments, the robot 130 installs the Microsoft Windows Service Control Manager (SCM) management service by default. As a result, such a robot 130 can open an interactive Windows session under the local system account and may have Windows service rights.

[0067] In some embodiments, the robot 130 may be installed in user mode, meaning that for such a robot 130, it has the same rights as the user to which the given robot 130 is installed. This feature may also be available for high density (HD) robots, ensuring maximum utilization of each machine. In some embodiments, either type of robot 130 may be configured in an HD environment.

[0068] The robot 130 in some embodiments is divided into multiple components, each specialized for a particular automation task. The robot components in some embodiments include, but are not limited to, the SCM managed robot service, the user mode robot service, the executor, the agent, and the command line. The SCM managed robot service manages and monitors the Windows session and acts as a proxy between the conductor 120 and the execution host (i.e., the computing system on which the robot 130 executes). These services are entrusted with and manage the credentials of the robot 130. The console application is launched by the SCM under the local system.

[0069] The user-mode robot service in some embodiments manages and monitors the Windows session and acts as a proxy between the conductor 120 and the execution host. The user-mode robot service may be delegated and manage the credentials of the robot 130. If the SCM management robot service is not installed, a Windows application may be launched automatically.

[0070] An Executor may execute a given job under a Windows session (i.e., execute a workflow). An Executor may be aware of per-monitor dots-per-inch (DPI) settings. An Agent may be a Windows Presentation Foundation (WPF) application that displays available jobs in a system tray window. An Agent may be a client of a service. An Agent may ask to start or stop a job or change settings. A Command Line is a client of a service. A Command Line is a console application that can request the start of a job and wait for its output.

[0071] As explained above, the separation of the components of the robot 130 helps developers, support users, and computing systems to more easily execute, identify, and track what each component is doing. In this way, special behaviors can be configured for each component, such as setting different firewall rules for executors and services. The executor may always be aware of per-monitor DPI settings in some embodiments. As a result, workflows may execute at any DPI regardless of the configuration of the computing system on which the workflow was created. Also, in some embodiments, projects from the designer 110 may be made independent of the browser zoom level. For applications that are not DPI aware or are marked as not DPI aware intentionally, in some embodiments, DPI may be disabled.

[0072] FIG. 2 is an architecture diagram illustrating a deployed RPA system 200 according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be or be part of the RPA system 100 of FIG. 1. It should be noted that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 210 includes an executor 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not be running on the computing system 210. The executor 212 is running a process. As shown in FIG. 2, multiple business projects may be running simultaneously. The agent 214 (e.g., a Windows service) is a single connection point for all the executors 212 in this embodiment. All messages in this embodiment are logged to the conductor 230, which further processes them via the database server 240, the indexer server 250, or both. As described above with respect to FIG. 1, the executor 212 may be a robotic component.

[0073] In some embodiments, a robot represents an association between a machine name and a username. A robot may manage multiple executors simultaneously. In computing systems that support multiple interactive sessions running simultaneously (such as Windows Server 2012), multiple robots may run simultaneously, each running in a separate Windows session using a unique username. This is referred to as the HD Robot above.

[0074] The agent 214 is also responsible for transmitting the robot's status (e.g., periodically sending "heartbeat" messages to indicate that the robot is still functioning) and downloading required versions of packages to be fulfilled. Communication between the agent 214 and the conductor 230 is, in some embodiments, always initiated by the agent 214. In a notification scenario, the agent 214 may open a WebSocket channel that is later used by the conductor 230 to send commands (e.g., start, stop, etc.) to the robot.

[0075] The server side includes a presentation layer (web application 232, Open Data Protocol (OData) Representational State Transfer (REST) ​​Application Programming Interface (API) endpoints 234, notifications and monitoring 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240, indexer server 250). Conductor 230 includes web application 232, OData REST API endpoints 234, notifications and monitoring 236, and API implementation / business logic 238. In some embodiments, most actions that a user performs in the conductor 220 interface (e.g., via browser 220) are performed by calling various APIs. Such actions may include, but are not limited to, launching jobs on a robot, adding / removing data in a queue, scheduling jobs to run unattended, etc., without departing from the scope of the invention. Web application 232 is the visual layer of the server platform. In this embodiment, web application 232 uses HyperText Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the present invention. A user interacts with web pages from web application 232, in this embodiment via browser 220, to perform various operations to control conductor 230. For example, a user may create robot groups, assign packages to robots, analyze per-robot and / or per-process logs, start and stop robots, etc.

[0076] In addition to the web application 232, the conductor 230 also includes a services layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the invention. The REST API is consumed by both the web application 232 and the agent 214, which in this embodiment is a supervisor of one or more robots on a client computer.

[0077] The REST API of the present embodiment covers configuration, logging, monitoring, and queuing functionality. The configuration endpoint may be used in some embodiments to define and configure users, permissions, robots, assets, releases, and environments of an application. The logging REST endpoint may be used to log various information, such as errors, explicit messages sent by the robot, and other environment specific information. The deployment REST endpoint may be used by the robot to query the version of the package that should be executed when a start job command is used in the conductor 230. The queuing REST endpoint may be responsible for managing queues and queue items, such as adding data to the queue, getting transactions from the queue, and setting the status of transactions.

[0078] Monitoring REST endpoints may monitor the web application 232 and the agents 214. Notification and monitoring API 236 may be a REST endpoint used to register agents 214, deliver configuration settings to agents 214, and send and receive notifications from the server and agents 214. Notification and monitoring API 236 may use WebSocket communications in some embodiments.

[0079] The persistence layer, in this embodiment, includes a pair of servers - a database server 240 (e.g., SQL Server) and an indexer server 250. The database server 240 in this embodiment stores configurations for robots, robot groups, associated processes, users, roles, schedules, etc. This information is managed, in some embodiments, via a web application 232. The database server 240 may also manage queues and queue items. In some embodiments, the database server 240 may also store messages logged by robots (in addition to or instead of the indexer server 250).

[0080] Optionally in some embodiments, indexer server 250 stores and indexes information logged by the robots. In certain embodiments, indexer server 250 may be disabled via a configuration setting. In some embodiments, indexer server 250 uses ElasticSearch®, a full-text search engine from an open source project. Messages logged by the robots (e.g., using activities such as log messages or line writes) may be sent via logging REST endpoint(s) to indexer server 250, where they are indexed for future use.

[0081] FIG. 3 is an architecture diagram illustrating the relationship 300 between a designer 310, activities 320, 330, and a driver 340, according to an embodiment of the present invention. As can be seen, a developer uses the designer 310 to develop a workflow to be performed by the robot. The workflow may include user-defined activities 320 and UI automation activities 330. In some embodiments, non-textual visual components in an image may be identified, referred to herein as computer vision (CV). Some CV activities related to such components may include, but are not limited to, click, type, get text, hover, detect presence or absence of element, update scope, highlight, and the like. In some embodiments, click may identify an element and click on it, for example, using CV, optical character recognition (OCR), fuzzy text matching, and multi-anchors. Type may identify an element using the above and type within an element. Get text may locate a particular text and scan it using OCR. Hover may identify an element and hover over it. Detect presence or absence of element may check to detect the presence or absence of an element on the screen using the techniques described above. In some embodiments, there may be hundreds or thousands of activities that may be implemented in the designer 310. However, any number and / or types of activities may be utilized without departing from the scope of the invention.

[0082] UI automation activities 330 are a subset of specialized low-level activities that are written in low-level code (e.g., CV activities) and facilitate interactions with applications through the UI layer. In certain embodiments, UI automation activities 300 may simulate user input, for example, through window messages. UI automation activities 330 facilitate these interactions through drivers 340 that enable the robot to interact with desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc.

[0083] Drivers 340 may interact with the OS at a low level, looking for hooks, monitoring keys, etc. They may facilitate integration with Chrome®, IE®, Citrix®, SAP®, etc. For example, a "click" activity plays the same role in these different applications via drivers 340.

[0084] FIG. 4 is an architecture diagram illustrating an RPA system 400, according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be or may include the RPA systems 100 and / or 200 of FIG. 1 and / or FIG. 2. The RPA system 400 includes multiple client computing systems 410 that execute robots. The computing systems 410 may communicate with a conductor computing system 420 via web applications executed thereon. The conductor computing system 420 may in turn communicate with a database server 430 and an optional indexer server 440.

[0085] 1 and 3, it should be noted that although web applications are used in these embodiments, any suitable client and / or server software may be used without departing from the scope of the present invention. For example, the conductor may run a server-side application on a client computing system that communicates with a non-web-based client software application.

[0086] FIG. 5 is an architecture diagram illustrating a computing system 500 configured to provide a UI mapper for RPA, according to an embodiment of the present invention. In some embodiments, the computing system 500 may be one or more computing systems depicted and / or described herein. The computing system 500 includes a bus 505 or other communication mechanism for communicating information and a processor(s) 510 coupled to the bus 505 for processing information. The processor(s) 510 may be any type of general or application-specific processor, including a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor(s) 510 may also have multiple processing cores, and at least some of the cores may be configured to perform specific functions. In some embodiments, multiple parallel processing may be used. In certain embodiments, at least one of the processor(s) 510 may be a neuromorphic circuit that includes processing elements that mimic biological neurons. In some embodiments, neuromorphic circuitry may not require typical components of a von Neumann computing architecture.

[0087] The computing system 500 further includes a memory 515 for storing information and instructions executed by the processor(s) 510. The memory 515 may be comprised of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage such as a magnetic or optical disk, or other types of non-transitory computer-readable media, or any combination thereof. The non-transitory computer-readable media may be any available media accessible by the processor(s) 510 and may include volatile media, non-volatile media, or both. Additionally, the media may be removable, non-removable, or both.

[0088] Additionally, the computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communication network via wireless and / or wired connections. In some embodiments, the communications device 520 may be configured to support any of the following wireless technologies: Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiple Access (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile (GSM) communications, General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), cdma2000, Wideband CDMA (W-CDMA), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), High-Speed ​​Packet Access (HSPA), Long Term Evolution (LTE), LTE Advanced (LTE-A), LTE-Advanced (LTE-B), LTE-Advanced-B, LTE-Advanced-C ... Advanced), 802.11x, Wi-Fi, Zigbee, Ultra-Wide Band (UWB), 802.16x, 802.15, Home Node-B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near-Field Communications (NFC), 5th Generation (5G), New Radio (NR), any combination thereof, and / or any other currently existing or future implemented communications standards and / or protocols without departing from the scope of the present invention.In some embodiments, the communications device 520 may include one or more antennas that are a singular antenna, an array of antennas, phased antennas, switched antennas, beamforming antennas, beamsteering antennas, combinations thereof, and / or any other antenna configuration without departing from the scope of the invention.

[0089] The processor(s) 510 are further coupled via bus 505 to a display 525, such as a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, a Retina® display, an in-plane switching (IPS) display, or any other suitable display for displaying information to a user. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, or the like, using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signaling, acoustic pulse recognition, frustrated total internal reflection, or the like. Any suitable display device and haptic I / O may be used without departing from the scope of the invention.

[0090] A keyboard 530 and cursor control device 535, such as a computer mouse, touchpad, and the like, are further coupled to the bus 505 to allow a user to interface with the computing system 500. However, in certain embodiments, a physical keyboard and mouse may not be present and the user may interact with the device solely through the display 525 and / or touchpad (not shown). Any type and combination of input devices may be used as a matter of design choice. In certain embodiments, no physical input devices and / or displays are present. For example, a user may interact with the computing system 500 remotely via another computing system in communication with it, or the computing system 500 may operate autonomously.

[0091] The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. The modules include an operating system 540 for the computing system 500. The modules further include a UI mapping module 545 configured to perform all or a portion of the processes described herein or derivatives thereof. The computing system 500 may include one or more additional functional modules 550 that include additional functionality.

[0092] Those skilled in the art will appreciate that the "system" may be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device or combination of devices without departing from the scope of the present invention. Presenting the above-described functions as being performed by a "system" is not intended to limit the scope of the present invention in any way, but rather to provide an example of many embodiments of the present invention. Indeed, the methods, systems, and apparatuses disclosed herein may be implemented in localized and distributed forms consistent with computing techniques, including cloud computing systems. The computing system may be part of or otherwise accessible in a local area network (LAN), a mobile communications network, a satellite communications network, the Internet, a public or private cloud, a hybrid cloud, a server farm, any combination thereof, and the like. Any local or distributed architecture may be used without departing from the scope of the present invention.

[0093] It should be noted that some of the system features described herein are presented as modules to better emphasize implementation independence. For example, a module may be implemented as a hardware circuit comprising custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, and the like.

[0094] Modules may also be implemented at least in part in software for execution by various types of processors. For example, an identified unit of executable code may include one or more physical or logical blocks of computer instructions, which may be organized, for example, as an object, procedure, or function. Nevertheless, executable identified modules need not be physically located together, but may include separate instructions stored in different locations, which when logically combined, comprise the modules, to accomplish the purpose stated for the modules. Furthermore, modules may be stored on non-transitory computer readable media, such as, for example, hard disk drives, flash devices, RAM, tape, and / or any other non-transitory computer readable media used to store data without departing from the scope of the present invention.

[0095] Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed across several different code segments, different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within a module, and may be embodied and organized in any suitable form within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed in different locations across different storage devices, or may exist, at least in part, simply as electronic signals on a system or network.

[0096] In some embodiments, the UI mapper operates in a live mode (also referred to herein as a "recorder mode") or an offline mode (also referred to herein as a "navigate mode"). The live and offline modes may be mutually exclusive, as the offline mode allows the user to navigate between screens of a recorded application and the live mode allows the user to indicate UI elements of a given screen. In some embodiments, when in the offline mode, the live application screen and its overlay may be frozen and / or hidden. This functionality is described in further detail with respect to FIGS. 6A-6D below.

[0097] FIG. 6A illustrates a UI Mapper 600 in a collapsed mode, according to an embodiment of the present invention. The UI Mapper 600 may be launched from a UI object browser of an RPA designer application in some embodiments. The UI Mapper 600 includes a Project UI Descriptors pane 610 having a UI tree structure organized by Applications 612, Screens 614, and UI Elements 616. In some embodiments, new applications and screens may be added from the toolbar buttons or by clicking an add button, or both. For example, if a user clicks the add button 602 or a hover-only add button (+) associated with a given tree item (e.g., add button 615), an associated application or screen may be added.

[0098] In this embodiment, one application 612, screen 614, and UI elements are shown. However, any number of applications, screens, and UI elements may be used and configured based on the nature of the application without departing from the scope of the present invention. If other UI elements are scoped, such as a pane of UI elements, this container may appear as a child of the application or screen. If a screen is used for UI elements used in multiple screens, this general screen may also appear at its own level in the tree or at the same level as the other screens.

[0099] If a new application is selected, then a new application and screen may be created in the UI tree in the appropriate location, and the properties of the UI elements may be displayed in the properties pane (e.g., see properties pane 620 of FIG. 6B). The "Name" field (e.g., name field 622 of FIG. 6B) may be pre-filled, for example, based on the name of the component in the selector information, and may be editable by the user. Selecting a new application may simultaneously initiate a "live" mode of the application in some embodiments, where the user may point to UI elements on the application screen. See, e.g., FIGS. 6B-6D.

[0100] After pointing to the application, if applicable, the application path and arguments are collected and stored, and a new screen is created containing the run-time data. This may include, for example, the application path and arguments (e.g., the file name and path to open the application with that particular file), as well as a selector. This set of information may be used to create the screen.

[0101] In some embodiments, after indicating an application, a new independent screen may be created without creating the application in the UI tree. Consider a case where a user wants to have a screen that does not belong to a specific application and / or a UI element that does not belong to a specific screen. For example, if a particular button appears in multiple screens or other states of an application, it may not make sense to use separate UI descriptor instances of the same button on all such screens. A user may specify such a button as a UI element in a general screen that is not in a specific application state.

[0102] In the embodiment of FIG. 6A and FIG. 6B, there are two editing modes: live mode and offline mode. In FIG. 6A, the UI mapper 600 is in offline mode, and in FIG. 6B, the UI mapper 600 is in live mode. These modes allow the user to add new elements or modify existing elements. The live mode (also referred to herein as "recorder mode") may be invoked via the recorder mode button 604, which disables the navigate button 606. Conversely, clicking the navigate button 606 disables the recorder mode button 604 and the user may operate in offline mode. The main difference between the live mode and offline mode in this embodiment is that in live mode, the application itself cannot be operated since interactions only add UI elements to the repository, whereas in navigate mode, new UI elements cannot be added by directing them in the application since interactions are sent to the application.

[0103] Turning to Figures 6B-6D, in recorder mode, a prompting function is initiated for a live application screen 640. As can be seen in Figures 6C and 6D, the live application screen 640 includes various options configured by checkboxes and radio buttons, as well as text fields. Initially, no UI elements are pointed to in the live application screen 640. See Figure 6C. The user may manually capture a UI element by clicking on the UI element and providing UI descriptor information, or CV may automatically attempt to identify the UI element and provide suggestions to the user. For example, CV may suggest capturing visible elements such as buttons, form fields, drop-down menus, etc.

[0104] Turning to FIG. 6D , a user selects a target UI element 642 and two anchors 644, 646 to uniquely identify the target UI element 642 using a multi-anchor approach. After the target UI element 642 is designated, the properties of the target UI element 642 may be added to and displayed to the user in a properties pane 620, for example, when the user hovers over and clicks the edit button 617. In this embodiment, the properties include a UI element name 622, a description 623, a Top Level Selector 626, a Target property 630, and an Anchor property 628. Expanding the Anchor property 628 may, in some embodiments, indicate a similar set of UI descriptor options for finding the respective anchor for the Target property 630.

[0105] Target Properties 630 includes checkboxes 632, 634, 638 for selecting the desired UI element detection technique(s), i.e., Selector, Fuzzy Selector, and Image Matching, respectively, in this embodiment. The user may manually modify the attributes of Selector Technique and Fuzzy Selector Technique using text areas 633, 635, respectively. The user may modify Selector Accuracy 636 using a slider or by manually entering a value. Image Accuracy 639 may be modified using the respective slider or by manually entering a value.

[0106] Using the properties pane 620, the user can modify various properties for each UI element, including, but not limited to, deleting anchors, deleting targets, editing selectors, editing UI element names and descriptions, etc. In some embodiments, the UI mapper must be in live mode for the user to add anchors. Using the project UI descriptor pane 610, the user may move the currently selected UI element 616 to a different position in the tree hierarchy. Once the user has finished configuring the UI element, the user may click the confirm element button 624, which resumes the recording process to capture the next desired UI element and / or screen.

[0107] In some embodiments, the user may click on any desired UI element 616 in the UI tree of the project UI descriptor pane 610. If valid (e.g., the UI descriptor(s) were able to find the target UI element in the live application screen), the target UI element is highlighted on the live application screen 640, as shown in FIG. 6D, and the settings of the UI element may be further edited as desired. If the target UI element is not valid, in some embodiments, the properties of the target UI element are still displayed and may be edited, but new elements (targets / anchors) may not be added until the error(s) are corrected. Also, validation may occur again if edits are made.

[0108] The user may also click within a screen 614 in the UI tree. In some embodiments, there may be no visual response to this action, but the next element added may be added to the selected screen instead of the previous screen. UI elements, screens, and applications may be deleted in some embodiments by right-clicking on the respective element and selecting a delete option (not shown). The user may stop recording by clicking the navigate button 606 on the toolbar.

[0109] Offline editing in the navigate (offline) mode of Figure 6A may be similar to the live mode of Figure 6B, except that highlighting and live validation of elements in the live application screen 640 and adding of anchors may not be performed. The properties pane 620 may allow manual editing of the properties shown in Figure 6B, even in the navigate mode. The user can return to the live mode via the recorder mode button 604.

[0110] Highlighting an element may be performed using the highlight button 608. This may be initiated for a single UI element or for all UI elements in some embodiments. For a single UI element, highlighting the UI element may mean locating it, so this action may switch modes from navigate mode to record mode. In the live application screen 640, the target UI element may be automatically highlighted.

[0111] When the user highlights a screen, all UI elements within the screen may be searched and the target may be highlighted. For a screen, the highlight button 608 may act like a toggle button and remain on until the user presses it again. In this state, the user may switch to recording mode and add / edit other UI elements, if desired. During the search for elements, a cancelable "toaster" may be displayed, displaying each UI element as it is found, rather than waiting until the end of the process.

[0112] 7 is a flow chart illustrating a UI mapper process 700 in accordance with an embodiment of the present invention. The process begins at 705 by providing an interface that facilitates the indication of UI elements within a screen of an application. When the user selects a new application at 710, a new application and screen are created in the UI tree and the application path and arguments for the application are collected and stored at 715.

[0113] CV is performed on the screen to automatically identify UI elements, and the automatically identified UI elements are used to provide one or more UI element suggestions at 720. One or more indications of UI elements in the screen are received at 725, and the one or more indicated UI elements are included in a UI tree of the application at 730.

[0114] In some embodiments, the UI mapper is configured to operate in a recorder mode to provide an interface and receive indications of UI elements on a screen. In particular embodiments, the UI mapper is configured to operate in a navigate mode, where the UI mapper facilitates selection of different screens. In some embodiments, the UI mapper is configured to operate in both modes. The recorder mode and the navigate mode are mutually exclusive in particular embodiments.

[0115] In some embodiments, the UI mapper is launched from a UI object browser of the RPA designer application. In certain embodiments, the UI mapper includes a UI descriptor pane configured with one or more applications, one or more respective screens per application, and one or more respective UI elements per screen included in the UI tree. In some embodiments, when a UI element is selected, the properties pane includes a target UI element and properties of one or more UI descriptors for identifying the target UI element. In certain embodiments, the properties pane further includes at least one anchor associated with the target UI element. In some embodiments, the UI mapper is configured to facilitate, via the properties pane, deleting an anchor, deleting a target UI element, editing one or more UI descriptors, editing the name and description of the target UI element, or a combination thereof.

[0116] Some embodiments implement a highlight mode function at 735, for example, if the user wants to search for a selected UI element on a screen or if the user wants to see all UI elements. The highlight mode function may include, for the former, receiving a selection of a UI element in the UI mapper and searching the screen for the selected UI element, and / or, for the latter, receiving a selection of a screen and displaying each UI element found in the screen. In some embodiments, at least one new separate screen is created at 740 that includes one or more UI elements that are common to multiple screens of the application.

[0117] In some embodiments, one or more containers for the screens are provided at 745. The one or more containers contain a subset of the screens. A properties pane containing properties of the selected application, screen, or UI element is displayed at 750. See, for example, FIG. 6B. The one or more indicated UI elements are added to an object repository at 755 for later reuse.

[0118] If the user wants to move to another screen of the application at 755, the user can put the UI mapper into navigate mode, select the next screen, return to recorder mode, and return to step 720 for that screen. In some embodiments, the user can select a different application and proceed to step 705 for that application. If no more screens or applications are to be displayed, then processing ends.

[0119] The process steps performed in FIG. 7 may be performed by a computer program encoding instructions to a processor(s) to perform at least a portion of the process(es) described in FIG. 7 according to an embodiment of the present invention. The computer program may be stored on a non-transitory computer readable medium. The computer readable medium may be, but is not limited to, a hard disk drive, a flash device, a RAM, a tape, and / or any other such medium or combination of media used to store data. The computer program may include coded instructions for controlling a processor(s) of a computing system (e.g., processor(s) 510 of computing system 500 of FIG. 5) to implement all or a portion of the process steps described in FIG. 7, which may also be stored on a computer readable medium.

[0120] The computer program may be implemented in hardware, software, or a hybrid implementation. The computer program may be composed of modules in operable communication with each other and designed to send information or instructions to a display. The computer program may be configured to run on a general purpose computer, an ASIC, or any other suitable device.

[0121] It will be readily understood that the components of the various embodiments of the present invention, as generally described and illustrated herein, may be arranged and designed in a wide variety of different configurations. Thus, the detailed description of the embodiments of the present invention, as represented in the accompanying figures, is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention.

[0122] The features, structures, or characteristics of the invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, reference throughout this specification to "certain embodiments," "some embodiments," or similar language means that the particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Thus, the appearance of "certain embodiments," "some embodiments," "other embodiments," or similar language throughout this specification does not necessarily refer to the same group of all embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0123] It should be noted that references to features, advantages, or similar language throughout this specification do not imply that all of the features and advantages that may be realized in the present invention should be in any single embodiment of the present invention, or that any embodiment of the present invention. Rather, language referring to features and advantages is understood to mean that the particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, discussions of features and advantages throughout this specification, as well as similar language, may, but do not necessarily, refer to the same embodiment.

[0124] Furthermore, the described features, advantages, and characteristics of the invention may be combined in any suitable manner in one or more embodiments. Those skilled in the relevant art will recognize that the invention may be practiced without the specific features or advantages of one or more particular embodiments. In other instances, additional features and advantages may be recognized in certain embodiments, although they may not be present in all embodiments of the invention.

[0125] Those of ordinary skill in the art will readily appreciate that the invention as described above can be practiced using steps in a different order and / or with hardware elements in different configurations than those disclosed. Thus, while the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative configurations will become apparent while remaining within the spirit and scope of the invention. Accordingly, reference should be made to the appended claims to determine the scope of the invention.

Claims

1. 1. A computer program product for providing a user interface (UI) mapper configured to map UI elements stored on a non-transitory computer readable medium, the computer program product comprising: Providing an interface for instructing UI elements on a screen of an application; receiving one or more indications of the UI elements of the screen; configured to include the indicated one or more UI elements in a UI tree of the application; the UI mapper is configured to operate in a live mode to provide the interface and receive the indication of the UI elements on the screen, and to operate in an offline mode in which the UI mapper makes selections of different screens; The UI mapper prevents operation of the application during the live mode, and the UI mapper sends interactions to the application during the offline mode to indicate new UI elements within the application to prevent the new UI elements from being added.

2. The computer program product of claim 1 , wherein the live mode and the offline mode are mutually exclusive.

3. The computer program product of claim 1 , wherein the UI mapper is launched from a UI object browser of an RPA designer application.

4. 2. The computer program product of claim 1 , wherein the UI mapper includes a UI descriptor pane configured with one or more applications, one or more respective screens for each application, and one or more respective UI elements for each screen included in the UI tree.

5. The computer program further comprises: The computer program product of claim 1 , configured to create a new application and screen in the UI tree when a user selects the new application.

6. The computer program further comprises: The computer program product of claim 1 configured to collect and store application paths and arguments of the application.

7. The computer program further comprises: The computer program product of claim 1 , configured to create at least one new independent screen that includes one or more UI elements that are common to different screens of the application.

8. The computer program further comprises: performing computer vision (CV) on the screen to automatically identify UI elements; The computer program product of claim 1 , configured to use the automatically identified UI elements to provide suggestions for one or more UI elements.

9. The computer program further comprises: The computer program product of claim 1 configured to display a properties pane containing properties of a selected application, screen, or UI element.

10. The computer program product of claim 9 , wherein when a UI element is selected, the property pane includes properties of a target UI element and one or more UI descriptors for identifying the target UI element.

11. The computer program product of claim 10 , wherein the property pane further comprises at least one anchor associated with the target UI element.

12. 11. The computer program product of claim 10, wherein the UI mapper is configured to delete an anchor, delete the target UI element, edit the one or more UI descriptors, edit a name and description of the target UI element, or combinations thereof, via the property pane.

13. The UI mapper includes a highlight function, and when the highlight function is enabled, the computer program further comprises:

2. The computer program product of claim 1 , configured to receive a selection of a UI element in the UI mapper and search for the selected UI element in the screen, receive a selection of a screen and display each UI element found on the screen, or both.

14. The computer program further comprises: The computer program product of claim 1 , further comprising: providing one or more containers for the screens, the one or more containers configured to contain a subset of the screens.

15. The computer program further comprises: The computer program product of claim 1 , configured to add the indicated one or more UI elements to an object repository for subsequent reuse.

16. 1. A computer-implemented method for mapping user interface (UI) elements for robotic process automation (RPA), comprising: providing an interface for a UI mapper to indicate UI elements within a screen of an application; receiving, by the UI mapper, one or more indications of the UI elements within the screen; including, by the UI mapper, the indicated one or more UI elements in a UI tree of the application; adding, by the UI mapper or UI object browser, the indicated one or more UI elements to an object repository for later reuse; the UI mapper includes a UI descriptor pane that is configured with one or more applications, one or more respective screens for each application, and one or more respective UI elements for each screen included in the UI tree; A computer-implemented method, wherein when a new application is selected via the UI mapper, a new application and screen is created in the UI tree.

17. the UI mapper is configured to operate in a live mode to provide the interface and to receive the indication of the UI elements on the screen; 20. The computer-implemented method of claim 16, wherein the UI mapper is configured to operate in an offline mode, the UI mapper making selections of different screens.

18. When the UI mapper includes a highlight function and the highlight function is enabled, the method further comprises:

17. The computer-implemented method of claim 16, comprising receiving a selection of a UI element at the UI mapper and searching the screen for the selected UI element, receiving a selection of a screen and displaying each UI element found on the screen, or both.

19. 1. A computer program product for providing a user interface (UI) mapper configured to map UI elements stored on a non-transitory computer readable medium, the computer program product comprising: Providing an interface for instructing UI elements on a screen of an application; receiving one or more indications of the UI elements of the screen; including the indicated one or more UI elements in a UI tree of the application; configured to display a properties pane containing properties of the selected application, screen, or UI element; when a UI element is selected, the property pane includes properties of a target UI element and one or more UI descriptors for identifying the target UI element; The UI mapper is configured to effect removal of an anchor via the property pane.

20. 20. The computer program product of claim 19, wherein the UI mapper is configured to delete the target UI element, edit the one or more UI descriptors, edit a name and description of the target UI element, or combinations thereof, via the property pane.

Images