Interactive UI element mapping to an RPA object repository for RPA development.

Abstract

A system and method are provided for mapping interactive UI (User Interface) elements to an RPA (Robotic Process Automation) object repository. User input is received selecting a window of an application to be displayed on a display device. In response to receiving the user input selecting the window of the application, interactive UI elements within the window of the application are automatically identified. User input is received selecting one or more of the identified interactive UI elements within the window of the application. The one or more selected interactive UI elements are stored in an RPA object repository of the RPA system.

Description

Technical Field

[0001] The present invention generally relates to RPA (Robotic Process Automation), and more particularly, to the mapping of interactive UI (User Interface) elements to an RPA object repository for RPA development.

Background Art

[0002] RPA (Robotic Process Automation) is a form of process automation implemented to automate repetitive and / or labor-intensive tasks, thereby reducing costs and improving efficiency. In RPA, software robots are performed to automate workflows. Such workflows include multiple activities each corresponding to one or more actions. During RPA development, workflows are defined by developers who interact with an RPA designer application to define each activity of the workflow. The activities of the workflow are defined by recording the developer's interaction with interactive UI (User Interface) elements such as, for example, buttons, checkboxes, text fields, etc. Conventionally, to record the interaction, the developer selects an interactive UI element, stores the interactive UI element in an object repository, and then performs an action on the stored interactive UI element. However, the conventional recording of interactions is cumbersome and time-consuming because the developer needs to select individual interactive UI elements and store the interactive UI elements in the object repository before performing an action.

Summary of the Invention

[0003] According to one or more embodiments, a system and method are provided for mapping interactive UI (User Interface) elements to a Robotic Process Automation (RPA) object repository. User input is received to select an application window displayed on a display device. In response to the receipt of user input to select an application window, interactive UI elements within the application window are automatically identified. User input is received to select one or more of the identified interactive UI elements within the application window. One or more selected interactive UI elements are stored in the RPA object repository of the RPA system.

[0004] In one embodiment, identified interactive UI elements are automatically selected. User input is received to deselect one or more of the automatically selected interactive UI elements. The interactive UI elements may include one or more buttons, checkboxes, or text fields.

[0005] In one embodiment, the user input editing property of a stored interactive UI element is received.

[0006] In one embodiment, one or more selected interactive UI elements are compared with existing interactive UI elements stored in the RPA object repository to determine whether one or more selected interactive UI elements are duplicates. Depending on whether it is determined that one or more selected interactive UI elements are not duplicates, one or more selected interactive UI elements are stored. In one embodiment, the comparison is performed by filtering descriptors associated with existing interactive UI elements to generate a match score. Depending on whether it is determined that none of the match scores are the maximum match score, descriptors with the top N match scores are identified, where N is an arbitrary positive integer. A unified target algorithm is applied to the identified descriptors to determine whether they match one or more selected interactive UI elements. Depending on whether the unified target algorithm determines that the identified descriptors do not match one or more selected interactive UI elements, one or more selected interactive UI elements are stored in the RPA object repository.

[0007] In one embodiment, RPA automation of actions performed on one or more of the stored interactive UI elements is recorded.

[0008] These and other advantages of the present invention will become apparent to those skilled in the art by referring to the following detailed description and accompanying drawings. [Brief explanation of the drawing]

[0009] [Figure 1] This is an architecture diagram showing a robotic process automation (RPA) system according to one embodiment of the present invention.

[0010] [Figure 2] This is an architecture diagram showing an example of a deployed RPA system according to one embodiment of the present invention.

[0011] [Figure 3] This is an architecture diagram showing a simplified deployment example of an RPA system according to one embodiment of the present invention.

[0012] [Figure 4] This is an architecture diagram showing a cloud-based RPA system for implementing cloud-based management of robotic process automation robots, according to one embodiment of the present invention.

[0013] [Figure 5] This document describes a method for mapping interactive UI elements to an RPA object repository according to one or more embodiments.

[0014] [Figure 6] This document shows a user interface for an RPA designer application according to one or more embodiments.

[0015] [Figure 7] The user interface of a capture element module according to one or more embodiments is shown.

[0016] [Figure 8] This describes a user interface for selecting an application window from which interactive UI elements are captured, according to one or more embodiments.

[0017] [Figure 9] This document shows a user interface that identifies interactive UI elements according to one or more embodiments.

[0018] [Figure 10] This document shows a user interface for selecting interactive UI elements according to one or more embodiments.

[0019] [Figure 11]A user interface showing interactive UI elements stored in an RPA object repository according to one or more embodiments is shown.

[0020] [Figure 12] A user interface showing a capture element module according to one or more embodiments is shown.

[0021] [Figure 13] It is a block diagram of a computing system that may be used to implement embodiments of the present invention.

Mode for Carrying Out the Invention

[0022] The embodiments described in this specification are described in relation to the drawings, in which like reference numerals represent the same or similar elements. In RPA (Robotic Process Automation), robots are used to automatically execute workflows. RPA may be implemented by an RPA system as shown in FIGS. 1 to 4.

[0023] FIG. 1 is an architecture diagram of an RPA system 100 according to one or more embodiments. As shown in FIG. 1, the RPA system 100 includes a designer 102 for enabling a developer to design an automation process. More specifically, the designer 102 facilitates the development and deployment of RPA processes and robots for executing activities in a process. The designer 102 can provide solutions for application integration and automate business processes for third-party applications, management information technology (IT) tasks, and contact center operations. One commercial example of one embodiment of the designer 102 is UiPath Studio (trademark).

[0024] When designing the automation of a rule-based process, developers control the execution order and the relationships between the custom set of steps developed in the process, which are defined herein as “activities.” Each activity may include actions such as clicking a button, reading a file, or writing to a log panel. In some embodiments, processes may be nested or embedded.

[0025] Several types of processes may include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suitable for linear processes that allow a flow from one activity to another without cluttering the process. Flowcharts may be particularly suitable for more complex business logic that allows for the integration of decisions and the connection of activities in a more diverse manner through multiple branching logical operators. FSMs may be particularly suitable for large-scale workflows. FSMs can use a finite number of states triggered by conditions (i.e., transitions) or activities in their execution. Global exception handlers may be particularly suitable for debugging processes by determining the behavior of a workflow when an execution error is encountered.

[0026] Once a process is developed in Designer 102, the execution of the business process is organized by Conductor 104, which organizes one or more robots 106 to execute the process developed in Designer 102. One commercially available example of one embodiment of Conductor 104 is UiPath Orchestrator®. Conductor 220 facilitates the creation, monitoring, and deployment management of resources in the RPA environment. In one example, Conductor 104 is a web application. Conductor 104 can also function as an integration point with third-party solutions and applications.

[0027] Conductor 104 can manage a fleet of RPA robots 106 by connecting and executing robots 106 from a central point. Conductor 104 can have a variety of functions, including but not limited to provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interoperability. Provisioning may include creating and maintaining connections between robots 106 and Conductor 104 (e.g., a web application). Deployment may include ensuring the correct delivery of package versions to robots 106 assigned for execution. Configuration may include maintaining and delivering robot environments and process configurations. Queuing may include managing queues and queue items. Monitoring may include tracking robot identification data and maintaining user privileges. Logging may include storing and indexing logs to a database (e.g., an SQL database) and / or other storage mechanisms (e.g., ElasticSearch®, which gives the ability to store and quickly query large datasets). Conductor 104 can provide interoperability by acting as a central point of communication for third-party solutions and / or applications.

[0028] Robot 106 is an execution agent that executes processes built in Designer 102. One commercially available example of several embodiments of Robot 106 is UiPath Robots®. Types of Robot 106 include, but are not limited to, attended robots 108 and unattended robots 110. Attended robots 108 are triggered by a user or user event and operate alongside human users on the same computing system. Attended robots 108 can assist human users in accomplishing various tasks and can be triggered directly by human users and / or user events. In the case of attended robots, conductor 104 can provide a centralized process deployment and logging medium. In certain embodiments, attended robots 108 can only be launched from a “robot tray” or from a command prompt in a web application. Unattended robots 110 operate in unattended mode in a virtual environment and can be used to automate many processes, for example, for a large number of backend processes. The unattended robot 110 can be involved in remote execution, monitoring, scheduling, and providing support for work queues. Both attended and unattended robots can automate a wide range of systems and applications, including but not limited to mainframes, web applications, VMs, enterprise applications (e.g., those manufactured by SAP®, Salesforce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0029] In some embodiments, the robot 106 installs the Microsoft Windows® Service Control Manager (SCM) management service during initial setup. As a result, such a robot 106 can open an interactive Windows® session under the local system account and have rights to Windows® services. In some embodiments, the robot 106 can be deployed in user mode, having the same rights as a user, and a given robot 106 is deployed under user mode.

[0030] In some embodiments, the robot 106 is divided into several components, each dedicated to a specific task. These robot components in some embodiments include, but are not limited to, an SCM-managed robot service, a user-mode robot service, an executor, an agent, and a command line. The SCM-managed robot service manages and monitors Windows® sessions and acts as a proxy between the conductor 104 and the execution host (i.e., the computing system on which the robot 106 is performed). These services are trusted with and manage the credentials for the robot 106. The console application is launched by the SCM under the local system. In some embodiments, the user-mode robot service manages and monitors Windows® sessions and acts as a proxy between the conductor 104 and the execution host. The user-mode robot service can be trusted with and manage the credentials for the robot 106. If the SCM-managed robot service is not installed, the Windows® application can be launched automatically. An executor can execute a given job under a Windows® session (for example, an executor can perform a workflow) and can recognize per-monitor dots per inch (DPI) settings. An agent may be a Windows® Presentation Foundation (WPF) application that displays available jobs in a system tray window. An agent may be a client of the service. An agent can request to start or stop a job and change its settings. The command line is a client of the service and is a console application that can request to start a job and wait for the job's output.Dividing robot components can assist developers and support users, and allow computing systems to more easily execute, identify, and track what each robot component is performing. For example, special behavior can be configured for each robot component, such as setting different firewall rules for executors and services. As a further example, in some embodiments, executors can be aware of the DPI settings for each monitor, and as a result, workflows can be executed at any DPI, regardless of the configuration of the computing system on which they were created.

[0031] Figure 2 shows an RPA system 200 according to one or more embodiments. The RPA system 200 may be the RPA system 100 of Figure 1 or a part of the RPA system 100. It should be noted that the "client side," "server side," or both may include any desired number of computing systems without departing from the scope of the present invention.

[0032] As shown on the client side in this embodiment, the computing system 202 includes one or more executors 204, agents 206, and a designer 208. In other embodiments, the designer 208 does not have to run on the same computing system 202. The executors 204 (which may be robotic components as described above) execute processes, and in some embodiments, multiple business processes may run simultaneously. In this example, the agent 206 (e.g., a Windows® service) is a single connection point for managing the executors 204.

[0033] In some embodiments, the robot represents an association between a machine name and a username. The robot can manage multiple executors simultaneously. In a computing system that supports multiple interactive sessions running concurrently (e.g., Windows® Server 2012), multiple robots may run simultaneously in separate Windows® sessions (e.g., in a high-density (HD) environment), each using a unique username.

[0034] Agent 206 also transmits the robot's status (for example, periodically sending a "heartbeat" message indicating that the robot is still functioning) and is involved in downloading the necessary versions of packages to be executed. Communication between Agent 206 and Conductor 212 is initiated by Agent 206 in some embodiments. In an example notification scenario, Agent 206 can open a WebSocket channel that will later be used by Conductor 212 to send commands to the robot (e.g., start, stop, etc.).

[0035] As shown on the server side in this embodiment, the presentation layer comprises a web application 214, an Open Data Protocol (OData) Representation State Transfer (REST) ​​Application Programming Interface (API) endpoint 216, and a notification monitoring API 218. The server-side service layer includes the API implementation / business logic 220. The server-side persistence layer includes a database server 222 and an indexer server 224. The conductor 212 includes the web application 214, the OData REST API endpoint 216, the notification monitoring API 218, and the API implementation / business logic 220.

[0036] In various embodiments, most actions performed by the user on the conductor 212 interface (e.g., via a browser 210) are performed by calling various APIs. Such actions include, but are not limited to, starting jobs on robots, adding / deleting data in queues, and scheduling jobs to run unattended. The web application 214 is the visual layer of the server platform. In this embodiment, the web application 214 uses Hypertext Markup Language (HTML) and JavaScript® (JS). However, any desired markup language, scripting language, or any other format can be used without departing from the scope of the present invention. In this embodiment, the user interacts with web pages from the web application 214 via a browser 210 to perform various actions to control the conductor 212. For example, the user can create robot groups, assign packages to robots, analyze logs per robot and / or per process, and start and stop robots.

[0037] In addition to the web application 214, the conductor 212 also includes a service layer that exposes an OData REST API endpoint 216 (or other endpoints may be implemented without departing from the scope of the present invention). The REST API is consumed by both the web application 214 and the agent 206. In this exemplary configuration, the agent 206 is a supervisor of one or more robots on a client computer.

[0038] The REST API in this embodiment covers configuration, logging, monitoring, and queuing functions. In some embodiments, a configuration REST endpoint can be used to define and configure application users, permissions, robots, assets, releases, and environments. Logging of the REST endpoint may be useful for logging different information, such as errors, explicit messages sent by robots, and other environment-specific information. A deployment REST endpoint may be used by a robot to query the package version to be executed when a start job command is used within the conductor 212. Queuing of the REST endpoint allows involvement in the management of queues and queue items, such as adding data to the queue, retrieving transactions from the queue, and setting the status of transactions. Monitoring of the REST endpoint monitors the web application 214 and agent 206. The notification monitoring API 218 may be a REST endpoint used for registering agent 206, delivering configuration settings to agent 206, and sending / receiving notifications from the server and agent 206. In some embodiments, the notification monitoring API 218 may also use WebSocket communication.

[0039] The server-side persistence tier includes a pair of servers in this exemplary embodiment, namely a database server 222 (e.g., an SQL server) and an indexer server 224. In this embodiment, the database server 222 stores configurations such as robots, robot groups, associated processes, users, roles, and schedules. This information is managed via a web application 214 in some embodiments. The database server 222 can also manage queues and queue entries. In some embodiments, the database server 222 can store messages logged by robots (in addition to or instead of the indexer server 224). The indexer server 224 is optional in some embodiments and stores and indexes information logged by robots. In certain embodiments, the indexer server 224 may be disabled through configuration settings. In some embodiments, the indexer server 224 uses ElasticSearch®, an open-source full-text search engine. Messages logged by the robot (for example, using activities such as writing log messages or lines) may also be sent to the indexer server 224 via the logging REST endpoint, where they are indexed for future use.

[0040] Figure 3 is an architecture diagram showing a simplified deployment example of the RPA system 300 according to one or more embodiments. In some embodiments, the RPA system 300 may be the RPA systems 100 and / or 200 in Figures 1 and 2, respectively, or may include the RPA systems 100 and / or 200. The RPA system 300 includes a plurality of client computing systems 302 that run robots. The computing systems 302 can communicate with the conductor computing system 304 via web applications running thereon. Furthermore, the conductor computing system 304 communicates with a database server 306 and an optional indexer server 308. With respect to Figures 2 and 3, it should be noted that while web applications are used in these embodiments, any suitable client / server software can be used without departing from the scope of the present invention. For example, the conductor can run a server-side application that communicates with a non-web-based client software application on the client computing system.

[0041] In one embodiment, the RPA system 100 in Figure 1, the RPA system 200 in Figure 2, and / or the RPA system 300 in Figure 3 may be implemented for cloud-based management of RPA robots. Such cloud-based management of RPA robots allows RPA to be offered as Software as a Service (SaaS). Therefore, the conductor 104 in Figure 1, the conductor 212 in Figure 2, and / or the conductor 304 in Figure 3 are implemented in the cloud for cloud-based management of RPA robots to perform, for example, creating RPA robots, providing RPA robots, scheduling tasks related to RPA robots, stopping RPA robots, or any other orchestration tasks for managing RPA robots.

[0042] Figure 4 shows an architectural diagram of a cloud RPA system 400 for implementing cloud-based management of RPA robots according to one or more embodiments. The cloud RPA system 400 comprises a cloud computing environment 402 and a local computing environment 404. The local computing environment 404 represents the local network architecture of a user, such as a company or enterprise, or any other one or more entities. The local computing environment 404 includes a local network 406. The cloud computing environment 402 represents a cloud computing network architecture that provides services or processing of workloads remotely from the user in the local computing environment 404. The cloud computing environment 402 comprises various cloud networks, including the Internet 414, a user cloud network 418 representing a cloud network managed (or controlled) by a user and hosted by a cloud platform provider, and a cloud service provider cloud network 420 representing a cloud network managed by a cloud service provider and hosted by a cloud platform provider. A cloud service provider is an entity that provides services (e.g., RPA) via the cloud. A cloud platform provider is an entity that maintains the cloud computing infrastructure. The local network 406 of the local computing environment 404 is connected to the internet 414 of the cloud computing environment 402 in a communicable manner to facilitate communication between the local computing environment 404 and the cloud computing environment 402.

[0043] As shown in Figure 4, the cloud orchestrator 430 is implemented in the cloud computing environment 402 to enable cloud-based management of RPA robots. In particular, the cloud orchestrator 430 is managed by a cloud service provider and hosted within the cloud service provider cloud network 420 within the cloud computing environment 402. In one embodiment, the cloud service provider provides RPA to users in the local computing environment 404.

[0044] The cloud orchestrator 430 manages RPA robots in the cloud computing environment 402. Specifically, the user interacts with the computing device 412 in the local computing environment 404 to send commands to the cloud orchestrator 430 in the cloud computing environment 402 to manage the RPA robots. Alternatively, the user interacts with the computing device 412 in the local computing environment 404 to schedule the cloud orchestrator 430 to automatically send commands on behalf of the user to manage the RPA robots. Typical commands for managing RPA robots include commands to create RPA robots, commands to provision RPA robots, commands to schedule tasks on RPA robots (e.g., scheduling the time to execute a task and the type of robot to execute the task), commands to decommission RPA robots, or any other organizational commands for RPA robots. Upon receiving a command, the cloud orchestrator 430 executes the command, for example, by creating an RPA robot, provisioning an RPA robot, scheduling tasks for an RPA robot, or decommissioning an RPA robot. In one embodiment, the cloud orchestrator 430 may be similar to the conductor 104 in Figure 1, the conductor 212 in Figure 2, or the conductor 304 in Figure 3, but it may also be implemented in a cloud service provider cloud network 420 within a cloud computing environment 402.

[0045] The RPA robots managed by the cloud orchestrator 430 may include a pool of cloud robots deployed and maintained within the cloud computing environment 402. Such cloud robots may include one or more cloud service robots 428-A, ..., 428-X (hereinafter collectively referred to as cloud service robots 428) from the cloud service robot pool 426, and one or more cloud management robots 424-A, ..., 424-Y (hereinafter collectively referred to as cloud management robots 424) from the cloud management robot pool 422. Such cloud robots execute tasks (i.e., processes) in the cloud computing environment 402 and send the results of the tasks to users in the local computing environment 404. In addition to or instead of this, the RPA robots managed by the cloud orchestrator 430 may also include one or more local robots 410-A, ..., 410-Z (hereinafter collectively referred to as local robots 410) from the local robot pool 408.

[0046] The cloud service robot 428 is maintained by the cloud service provider in the cloud network 420 of the cloud service provider to execute RPA tasks in the cloud computing environment 402 for users in the local network environment 404. The cloud service robot 428 is created on demand when a user sends a command from the computing device 412 to the cloud orchestrator 430. Once created, the cloud service robot 428 enters standby mode while waiting to execute a task (or workflow). While in standby mode, the cost of operating the cloud service robot 428 is minimized or reduced. Tasks are scheduled on the cloud service robot 428 when a user sends a command from the computing device 412 to the cloud orchestrator 430. The command for scheduling a task defines the time for the task to be executed and the type of robot to execute the task. The cloud service robot 428 wakes up from standby mode to execute a task and returns to standby mode when the task is complete. Therefore, the cloud service robot 428 performs tasks on the cloud service provider cloud network 420 for users within the local computing environment 404.

[0047] The cloud management robot 424 is maintained by a user in the user cloud network 418 to execute RPA tasks in the cloud computing environment 402 for a user in the local network environment 404. The cloud management robot 424 is functionally similar to the cloud service robot 428 and is also hosted in the cloud computing environment 402. However, the user cloud network 418 where the cloud management robot 424 is hosted is managed by the user, while the cloud service provider cloud network 420 where the cloud service robot 428 is hosted is managed by the cloud service provider and hosted by the cloud platform provider. The cloud orchestrator 430 manages the cloud management robot 424 by establishing a connection between the cloud service provider cloud network 420 and the user cloud network 418. The user cloud network 418 may be established by a user utilizing cloud provider technology to tunnel back to the local network 406. The user can establish a dedicated network connection from the local network 406 to the cloud service provider cloud network 420. The connections generally take the form of, for example, arbitrary-to-arbitrary (e.g., Internet Protocol Virtual Private Network) networks, point-to-point Ethernet networks, or virtual cross-connections via connection providers in colocation facilities. These connections do not traverse the public internet. This provides higher reliability, faster speeds, consistent latency, and greater security than typical connections over the internet. The user cloud network 418 remains fully controlled and managed by the user, thereby providing the user with strict control over their data.

[0048] Once a connection is established between the cloud service provider cloud network 420 and the user cloud network 418, the cloud management robot 424 is created in response to a request from a user interacting with the cloud orchestrator 430 via the computing device 412. The cloud management robot 424 is created on the user cloud network 418. Therefore, the cloud management robot 424 performs tasks on the user cloud network 418 for users in the local computing environment 404. An algorithm can be applied to maximize the utilization of robots in the cloud management robot pool 422 and reduce the user's operational costs.

[0049] The local robot 410 is maintained by a user in local network 406 to perform RPA tasks for a user in local network environment 404. Local network 406 is controlled or managed by the user. The cloud orchestrator 430 maintains a connection to the local robot 410 via a standard HTTPS connection.

[0050] During RPA development, the workflow is defined by a developer (or other user) who interacts with an RPA designer application to define each activity of the workflow. For example, such an RPA designer application may be the designer 102 of RPA system 100 in Figure 1, the designer 208 of RPA system 200 in Figure 2, the designer of RPA system 300 (not shown) in Figure 3, or the designer of cloud RPA system 400 (not shown) in Figure 4. Traditionally, developers design workflows by individually selecting interactive UI (user interface) elements from the application window and recording the automation of the selected interactive UI elements, which can be a cumbersome and time-consuming process. Embodiments described herein facilitate the development of RPA workflows by mapping interactive UI elements to an RPA object repository. In this way, developers can select one or more interactive UI elements from the RPA object repository to record automation without having to select each interactive UI element from the application window. Preferably, embodiments described herein provide seamless recording of automation.

[0051] Figure 5 shows a method 500 for mapping interactive UI elements to an RPA object repository according to one or more embodiments. The steps of method 500 can be performed by one or more suitable computing devices, such as the computing system 1300 in Figure 13.

[0052] In step 502 of Figure 5, user input is received to select an application window to be displayed on the display device. User input is received from the developer or any other appropriate user.

[0053] User input, such as selecting an application window, is received during the RPA workflow design by the developer. During the RPA workflow design, the developer interacts with the RPA Designer application to define each activity in the workflow. Workflow activities can be defined by selecting interactive UI elements from the RPA object repository via the Designer application and recording the actions performed on those interactive UI elements.

[0054] Figure 6 shows a user interface 600 of an RPA designer application according to one or more embodiments. The user interface 600 shows a main view 602 in which the developer defines the RPA workflow and an object repository 604 that stores interactive UI elements. As shown in Figure 6, the object repository 604 is empty. To capture the interactive UI elements to be stored in the object repository 604, the developer selects a recorder symbol 606 that invokes the capture element module.

[0055] Figure 7 shows a user interface 700 for a capture element module according to one or more embodiments. The user interface 700 includes an all-element capture button 702 for capturing all interactive UI elements within a selected window of the application. The developer selects the all-element capture button 702, which activates the scan module to begin the capture process.

[0056] Figure 8 shows a user interface 800 for selecting an application window in which interactive UI elements are to be captured, according to one or more embodiments. The user interface 800 shows a scan module 802 that prompts the developer to select an application window in which interactive UI elements are to be captured. In response to the prompt from the scan module 802, the developer selects window 804. The developer's selection of window 804 may be the user input to select an application window in step 502 of Figure 5.

[0057] In step 504 of Figure 5, in response to user input selecting an application window, interactive UI elements are automatically identified within the application window. Interactive UI elements are user interface elements that users can interact with. Typical interactive UI elements include buttons, checkboxes, and text fields.

[0058] Interactive UI elements may be automatically identified using any appropriate method. In one embodiment, interactive UI elements are automatically identified using a trained machine learning-based model. The trained machine learning-based model takes an image of the application window as input and produces an image of the application window with the identified interactive UI elements as output. The trained machine learning-based model is trained using training data during a previous offline or training phase. The training data includes training images of the application window annotated with interactive UI elements (e.g., by a user).

[0059] Figure 9 shows a user interface 900 that identifies interactive UI elements according to one or more embodiments. The user interface 900 displays a scan module 902 and a window 804 of an application selected by the developer, along with the interactive UI elements 904-A to 904-Y highlighted therein.

[0060] In step 506 of Figure 5, user input is received to select one or more identified interactive UI elements within the application window. In one embodiment, the user input may be a selection of the Select All button or Deselect All button to select each of the identified interactive UI elements or to deselect all identified interactive UI elements. In other embodiments, the user input may, in addition to or instead, be a direct selection or deselection of one or more interactive UI elements within the application window. In one embodiment, all identified interactive UI elements are automatically selected, and user input is received to deselect one or more of the identified interactive UI elements.

[0061] Figure 10 shows a user interface 1000 for selecting interactive UI elements according to one or more embodiments. In the user interface 1000, the developer can select the Deselect All button 1002 in the scan module 902 to deselect all identified UI elements, or select the Select All button 1004 to select all identified UI elements. In addition to or instead, the developer can select or deselect one or more interactive UI elements 904-A to 904-Y in the window 804. As shown in Figure 10, all identified interactive UI elements 904-A to 904-Y are automatically selected, and the developer can then deselect interactive UI elements 904-A to 904-M, thereby resulting in the selection of interactive UI elements 904-N to 904-Y. The developer confirms the selection of interactive UI elements 904-N to 904-Y by selecting the Capture button 1006.

[0062] In step 508 of Figure 5, the selected interactive UI element is stored in the RPA object repository of the RPA system. For example, the RPA system may be RPA system 100 in Figure 1, RPA system 200 in Figure 2, RPA system 300 in Figure 3, or cloud RPA system 400 in Figure 4.

[0063] Figure 11 shows a user interface 1100 that displays interactive UI elements stored in an RPA object repository, according to one or more embodiments. The user interface 1100 includes a capture element module 1102 that displays an RPA object repository 1104 that stores interactive UI elements selected from a window 804.

[0064] In one embodiment, a selected interactive UI element is compared to an existing interactive UI element stored in the RPA object repository to determine whether the selected interactive UI element is a duplicate. To perform the comparison, all descriptors associated with the existing interactive UI elements stored in the RPA object repository are retrieved. The descriptors hold information that uniquely identifies their associated existing interactive UI elements. The descriptors are filtered to determine whether their associated existing interactive UI elements are potential candidates for matching the selected interactive UI element. The filter generates a match score (for example, from 0 to 1, where 0 indicates no match with the selected interactive UI element and 1 indicates a match with the selected interactive UI element). If the filter identifies a descriptor with the highest match score (i.e., indicating a match), the existing interactive UI element associated with that descriptor is returned as a match with the selected interactive UI element. If the filter identifies multiple descriptors with the highest match score, the developer can select one of the descriptors, and any existing interactive UI elements associated with the selected descriptor are returned as matching the selected interactive UI element. If the filter does not identify a descriptor with the highest match score, the top N descriptors with the highest match scores (where N is any positive integer, such as 5) are identified, and the fully unified target algorithm is applied to the identified descriptor. If the unified target algorithm determines that the identified descriptor matches the selected interactive UI element, any existing interactive UI elements associated with that identified descriptor are returned as matching the selected interactive UI element.Otherwise, if the unified target algorithm determines that none of the identified descriptors match the selected interactive UI element, the selected interactive UI element is not considered a duplicate, and the selected interactive UI element is stored in the RPA object repository. In this way, only selected interactive UI elements that do not match existing interactive UI elements are stored in the RPA object repository.

[0065] In one embodiment, user input can be received from a developer to edit and save the properties of interactive UI elements stored in an object repository. Figure 12 shows a user interface 1200 that represents a capture element module according to one or more embodiments. The user interface 1200 includes a property panel 1202 for editing the properties of a selected interactive UI element 1204 stored in an RPA object repository 1206. Typical properties that can be edited in the property panel 1202 include the element name, type, description, selector, and fuzzy selector. In one embodiment, the semantic meaning of the interactive UI element may also be stored in the RPA object repository 1202.

[0066] In one embodiment, interactive UI elements stored in the RPA object repository may be tested by a developer to determine whether all interactive UI elements within the application window have been identified. If one or more interactive UI elements within the window are not identified, the unidentified interactive UI elements are marked as missing or problematic.

[0067] In step 510 of Figure 5, RPA automation is recorded for actions performed on one or more of the memorized interactive UI elements. The actions performed on one or more of the memorized interactive UI elements are executed by the developer. The recorded RPA automation is stored in the RPA object repository and can be used to define activities in the RPA workflow.

[0068] Figure 13 is a block diagram showing a computing system 1300 configured to perform the methods, workflows, and processes described herein, including the method 500 of Figure 5, according to one embodiment of the present invention. In some embodiments, the computing system 1300 may be one or more of the computing systems shown and / or described herein. The computing system 1300 includes a bus 1302 or other communication mechanism for communicating information and a processor 1304 coupled to the bus 1302 for processing information. The processor 1304 may be any type of general-purpose or dedicated processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor 1304 may also have multiple processing cores, and at least some of the cores may be configured to perform specific functions. In some embodiments, multiple parallel processing may be used.

[0069] The computing system 1300 further includes a memory 1306 for storing information and instructions executed by the processor 1304. The memory 1306 may consist of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage such as magnetic or optical disks, or any other type of non-temporary computer-readable medium, or any combination thereof. The non-temporary computer-readable medium may be any available medium accessible by the processor 1304, and may include volatile medium, non-volatile medium, or both. The medium may also be removable, non-removable, or both.

[0070] Furthermore, the computing system 1300 includes communication devices 1308, such as transceivers, for providing access to a communication network via wireless and / or wired connections in accordance with any currently existing or future implemented communication standards and / or protocols.

[0071] The processor 1304 is further coupled via the bus 1302 to a display 1310 suitable for displaying information to the user. The display 1310 may also be configured as a touch display and / or any suitable haptic I / O device.

[0072] The keyboard 1312 and cursor control devices 1314, such as a computer mouse and touchpad, are further coupled to the bus 1302 to enable the user to interface with the computing system. However, in certain embodiments, a physical keyboard and mouse may not be present, and the user can interact with the device only through the display 1310 and / or touchpad (not shown). Any type and combination of input devices can be used as a design choice. In certain embodiments, no physical input devices and / or displays are present. For example, the user can interact with the computing system 1300 remotely through another computing system communicating with it, or the computing system 1300 can operate autonomously.

[0073] Memory 1306 stores software modules that, when executed by processor 1304, provide functionality. The modules include an operating system 1316 for computing system 1300 and one or more further functional modules 1318 configured to execute all or part of the processes or derivatives thereof described herein.

[0074] As those skilled in the art will see, “System” can be embodied without departing from the scope of the present invention as a server, embedded computing system, personal computer, console, personal digital assistant (PDA), mobile phone, tablet computing device, quantum computing system, or any other suitable computing device, or combination of devices. Presenting the above functions as being performed by “System” is not intended in any way to limit the scope of the present invention, but rather to provide an example of many embodiments of the present invention. In fact, the methods, systems, and apparatus disclosed herein can be implemented in localized and distributed forms that are consistent with computing technologies, including cloud computing systems.

[0075] It should be noted that some of the system features described herein are presented as modules to more specifically emphasize the independence of their implementations. For example, a module may be implemented as a custom very large-scale integrated circuit (VLSI) circuit or as a hardware circuit including off-the-shelf semiconductors such as gate arrays, logic chips, transistors, or other discrete components. Alternatively, a module may be implemented as a programmable hardware device such as a field-programmable gate array, programmable array logic, programmable logic device, or graphics processing unit. Furthermore, a module may be at least partially implemented as software for execution by various types of processors. An identified unit of executable code may include, for example, one or more physical or logical blocks of computer instructions that can be organized as objects, procedures, or functions. Nevertheless, the executable files of an identified module do not need to be physically located together, but may include the module when logically combined and may contain different instructions stored in different locations to achieve the stated purpose of the module. Furthermore, the module may be stored in a computer-readable medium which may be, for example, a hard disk drive, a flash device, RAM, tape, and / or any other such non-temporary computer-readable medium used to store data without departing from the scope of the present invention. In fact, a module of executable code may be a single instruction or many instructions, and may be distributed across several different code segments, different programs, and several memory devices. Similarly, operational data may be identified within a module and illustrated herein, embodied in any suitable form, and organized within any suitable type of data structure. Operational data may be collected as a single dataset, or distributed across different locations including different storage devices, or at least partially exist only as electronic signals on a system or network.

[0076] The foregoing merely illustrates the principles of this disclosure. Therefore, it will be apparent that those skilled in the art can conceive of various configurations that embody the principles of this disclosure and that fall within the spirit and scope of this disclosure, even if not explicitly stated or shown herein. Furthermore, all examples and conditional language enumerated herein are primarily for educational purposes to help the reader understand the principles of this disclosure and the concepts to which the inventors contribute to advance the art, and should not be interpreted as limiting the reader to such specifically enumerated examples and conditions. Moreover, all descriptions herein that enumerate the principles, aspects, and embodiments of this disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents. Furthermore, such equivalents are intended to include both currently known equivalents and equivalents to be developed in the future.

Claims

1. A computer implementation method performed by at least one processor, The steps include: receiving user input by at least one processor to select an application window to be displayed on a display device; The steps include: automatically identifying interactive UI (user interface) elements within a window of the application in response to receiving user input to select the window of the application, using the at least one processor and a trained machine learning model; The steps include: receiving user input by at least one processor to select one or more of the identified interactive UI elements within the window of the application; The steps include filtering descriptors associated with existing interactive UI elements to generate a numerical match score representing the likelihood of matching with one or more selected interactive UI elements, A step of identifying descriptors with the top N match scores, in accordance with determining that none of the aforementioned match scores are the maximum match scores, wherein N is an arbitrary positive integer; The steps include: applying a unified target algorithm to the identified descriptor to determine whether the identified descriptor matches one or more selected interactive UI elements; The steps include storing the one or more selected interactive UI elements in the RPA (Robotic Process Automation) object repository by at least one processor, in response to the unified target algorithm determining that the identified descriptor does not match the one or more selected interactive UI elements, The steps include recording RPA automation actions performed on one or more of the stored interactive UI elements by the at least one processor, Computer implementation methods including

2. The step further includes the step of automatically selecting the identified interactive UI elements by the at least one processor, and the step of receiving user input by the at least one processor to select one or more of the identified interactive UI elements in the window of the application, The computer implementation method according to claim 1, further comprising the step of receiving user input by at least one processor to deselect one or more of the automatically selected interactive UI elements.

3. The computer implementation method according to claim 1, further comprising the step of receiving user input editing properties of the stored interactive UI element by at least one processor.

4. The step of storing the one or more selected interactive UI elements in the RPA (Robotic Process Automation) object repository of the RPA system using at least one processor is: The steps include: comparing the one or more selected interactive UI elements with existing interactive UI elements stored in the RPA object repository using at least one processor to determine whether the one or more selected interactive UI elements are duplicates; In response to determining that one or more selected interactive UI elements are not duplicates, the at least one processor stores the one or more selected interactive UI elements. The computer implementation method according to claim 1, including the method described in claim 1.

5. The step of comparing the one or more selected interactive UI elements with existing interactive UI elements stored in the RPA object repository by at least one processor to determine whether the one or more selected interactive UI elements are duplicates is: The computer implementation method according to claim 4, comprising the step of searching for a descriptor associated with an existing interactive UI element stored in the RPA object repository, wherein the retrieved descriptor holds information that uniquely identifies the associated existing interactive UI element.

6. The computer implementation method according to claim 1, wherein the interactive UI element includes one or more of a button, a checkbox, or a text field.

7. Memory for storing computer program instructions, A processor configured to execute the computer program instructions, wherein the computer program instructions are performed by the at least one processor An operation that receives user input to select an application window displayed on the display device, An operation to automatically identify interactive UI (user interface) elements within a window of an application in response to receiving user input to select the window of the application, using a trained machine learning model; An operation to receive user input to select one or more of the identified interactive UI elements within the window of the application, An operation to filter descriptors associated with existing interactive UI elements and generate a numerical match score representing the likelihood of matching with one or more selected interactive UI elements, An operation to identify descriptors with the top N match scores, where N is an arbitrary positive integer, in accordance with the determination that none of the aforementioned match scores are the maximum match scores, An operation to determine whether the identified descriptor matches one or more selected interactive UI elements by applying a unified target algorithm to the identified descriptor, In response to the unified target algorithm determining that the identified descriptor does not match one or more selected interactive UI elements, the operation of storing one or more selected interactive UI elements in the RPA (Robotic Process Automation) object repository, and the operation of storing one or more selected interactive UI elements in the RPA object repository of the RPA system, An operation to record an RPA automation action performed on one or more of the stored interactive UI elements, At least one processor configured to perform the following: A device equipped with the following features.

8. The operation further includes an operation to automatically select the identified interactive UI element, and the operation to receive user input to select one or more of the identified interactive UI elements in the window of the application, The apparatus according to claim 7, further comprising an operation to receive user input to deselect one or more of the automatically selected interactive UI elements.

9. The aforementioned operation is, The apparatus according to claim 7, further comprising an operation to receive user input editing properties of the stored interactive UI element.

10. The operation of storing one or more selected interactive UI elements in the RPA (Robotic Process Automation) object repository of the RPA system is: An operation to determine whether the one or more selected interactive UI elements are duplicates by comparing them with the existing interactive UI elements stored in the RPA object repository, In response to determining that one or more selected interactive UI elements are not duplicates, the operation involves storing the one or more selected interactive UI elements. The apparatus according to claim 7, including the following:

11. The operation of comparing the one or more selected interactive UI elements with the existing interactive UI elements stored in the RPA object repository to determine whether the one or more selected interactive UI elements are duplicates is: The apparatus according to claim 10, comprising an operation to retrieve a descriptor associated with an existing interactive UI element stored in the RPA object repository, wherein the descriptor holds information that uniquely identifies the associated existing interactive UI element.

12. The apparatus according to claim 7, wherein the interactive UI element includes one or more of a button, a checkbox, or a text field.

13. A non-temporary computer-readable medium for storing computer program instructions, wherein, when the computer program instructions are executed on at least one processor, the at least one processor, An operation that receives user input to select an application window displayed on the display device, An operation to automatically identify interactive UI (user interface) elements within a window of an application in response to receiving user input to select the window of the application, using a trained machine learning model; An operation to receive user input to select one or more of the identified interactive UI elements within the window of the application, An operation to filter descriptors associated with existing interactive UI elements and generate a numerical match score representing the likelihood of matching with one or more selected interactive UI elements, An operation to identify descriptors with the top N match scores, where N is an arbitrary positive integer, in accordance with the determination that none of the aforementioned match scores are the maximum match scores, An operation to determine whether the identified descriptor matches one or more selected interactive UI elements by applying a unified target algorithm to the identified descriptor, In response to the unified target algorithm determining that the identified descriptor does not match one or more selected interactive UI elements, the operation of storing one or more selected interactive UI elements in the RPA (Robotic Process Automation) object repository, and the operation of storing one or more selected interactive UI elements in the RPA object repository of the RPA system, An operation to record an RPA automation action performed on one or more of the stored interactive UI elements, A non-temporary computer-readable medium that allows the execution of operations including [specific actions].

14. The operation further includes an operation to automatically select the identified interactive UI element, and the operation to receive user input to select one or more of the identified interactive UI elements in the window of the application, The non-temporary computer-readable medium according to claim 13, comprising an operation to receive user input to deselect one or more of the automatically selected interactive UI elements.

15. The aforementioned operation is, The non-temporary computer-readable medium according to claim 13, further comprising an operation to receive user input editing properties of the stored interactive UI element.

16. The operation of storing one or more selected interactive UI elements in the RPA (Robotic Process Automation) object repository of the RPA system is: An operation to determine whether the one or more selected interactive UI elements are duplicates by comparing them with the existing interactive UI elements stored in the RPA object repository, In response to determining that one or more selected interactive UI elements are not duplicates, the operation involves storing the one or more selected interactive UI elements. A non-temporary computer-readable medium according to claim 13, including the following:

17. The operation of comparing the one or more selected interactive UI elements with the existing interactive UI elements stored in the RPA object repository to determine whether the one or more selected interactive UI elements are duplicates is: The non-temporary computer-readable medium according to claim 16, comprising an operation to retrieve a descriptor associated with an existing interactive UI element stored in the RPA object repository, wherein the descriptor holds information that uniquely identifies the associated existing interactive UI element.

18. The non-temporary computer-readable medium according to claim 13, wherein the interactive UI element includes one or more of a button, a checkbox, or a text field.

Images