Semantic target identification for user interface (UI) automation

Abstract

To provide a computer system and a method which improve identification of a target in automation of a user interface (UI)SOLUTION: The method includes: receiving an encoding of a robotic process automation (RPA) activity and an encoding of a design-time target label; and, in response to that, identifying a runtime instance of a target element within a runtime instance of a UI exposed by the computer system. The runtime instance of the target element is identified according to a similarity between a meaning of the design-time target label and a meaning of a label attached to the runtime instance of the target element. The method further includes executing the RPA activity on the runtime instance of the target element in response to identification of the runtime instance of the target element.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] BACKGROUND OF THE INVENTION The present invention relates to robotic process automation (RPA), and in particular to improving targeting in user interface (UI) automation. [Background technology]

[0002] RPA is an emerging field in information technology that aims to improve productivity by automating repetitive computing tasks, freeing up human operators to perform more intelligent, sophisticated, and / or creative activities. The primary tasks targeted for automation include extracting structured data from documents and web pages, and interacting with user interfaces, such as filling out forms and manipulating spreadsheets, among others.

[0003] Automating interactions with a user interface presents certain technical challenges, such as clearly identifying the target of a robotic activity (e.g., a specific button to click, a specific form field to fill out, etc.). When designing an RPA workflow, target UI elements can be specified via a set of programmatic and / or visual properties for each element. Programmatic properties may include, for example, a set of attribute-value pairs that characterize each element's location within a programmatic representation of each UI, such as a UI tree or a Document Object Model (DOM). Exemplary visual properties include each element's location relative to other elements in the UI, each element's color, and its label.

[0004] However, target UIs (e.g., e-commerce web pages, accounting interfaces, etc.) are typically developed and maintained independently of the RPA robots tasked with interacting with the respective interfaces. As a result, the functionality and / or appearance of the target UI may change without the knowledge of the RPA developer. Various UI elements may be moved, renamed and / or resized, the UI color scheme may change, etc. Such changes may result in the RPA robot being unable to identify the activity target because it no longer has the expected characteristics.

[0005] Therefore, there is great interest in developing robust methods for identifying RPA activity targets, i.e., methods that are relatively insensitive to changes in the design of the target UI.

[0006] (Summary of the Invention) According to one aspect, a computer system includes at least one hardware processor configured to receive an encoding of a robotic process automation (RPA) activity and an encoding of a design-time target label, where the RPA activity mimics a human interaction with a target element of a user interface (UI), and the design-time target label includes a text label attached to the target element within a design-time instance of the UI. The at least one hardware processor is further configured to identify a runtime instance of the target element within a runtime instance of the UI exposed by the computer system, and, in response, perform the RPA activity on the runtime instance of the target element. The runtime instance of the target element is identified according to a similarity between the meaning of the design-time target label and the meaning of the label attached to the runtime instance of the target element.

[0007] According to another aspect, a computer-implemented RPA method includes using at least one hardware processor of a computer system to receive an encoding of an RPA activity and an encoding of a design-time target label, where the RPA activity mimics a human interaction with a target element of a UI, and the design-time target label includes a text label attached to the target element in the design-time instance of the UI. The method further includes using the at least one hardware processor to identify a runtime instance of the target element in a runtime instance of the UI exposed by the computer system and, in response, perform the RPA activity on the runtime instance of the target element. The runtime instance of the target element is identified according to a similarity between the meaning of the design-time target label and the meaning of the label attached to the runtime instance of the target element.

[0008] According to yet another aspect, a non-transitory computer-readable medium stores instructions that, when executed by at least one hardware processor of a computer system, cause the computer system to receive an encoding of an RPA activity and an encoding of a design-time target label, the RPA activity mimicking a human interaction with a target element of a UI, and the design-time target label including a text label attached to the target element in the design-time instance of the UI. The instructions further cause the computer system to identify a runtime instance of the target element in a runtime instance of the UI exposed by the computer system and, in response, perform the RPA activity on the runtime instance of the target element. The runtime instance of the target element is identified according to a similarity between the meaning of the design-time target label and the meaning of the label attached to the runtime instance of the target element. [Brief explanation of the drawings]

[0009] The foregoing aspects and advantages of the present invention will be better understood by reading the following detailed description and by reference to the drawings in which:

[0010] [Figure 1] FIG. 1 illustrates an architecture diagram of a hyper-automation system according to some embodiments of the present invention.

[0011] [Figure 2] 1 illustrates an exemplary RPA system according to some embodiments of the present invention.

[0012] [Figure 3] 1 illustrates an exemplary deployed RPA system implemented in a client-server configuration according to some embodiments of the present invention.

[0013] [Figure 4] 1 illustrates an exemplary user interface (UI) and exemplary UI elements that may be subject to automation according to some embodiments of the present invention.

[0014] [Figure 5] 1 illustrates exemplary design-time and run-time target user interfaces according to some embodiments of the present invention.

[0015] [Figure 6] 1 illustrates an exemplary data exchange according to some embodiments of the present invention.

[0016] [Figure 7] 1 illustrates an exemplary sequence of steps performed by an RPA design application according to some embodiments of the present invention.

[0017] [Figure 8] 1 illustrates an exemplary RPA design interface according to some embodiments of the present invention.

[0018] [Figure 9]1 illustrates an exemplary semantic target selection interface exposed by an RPA design application according to some embodiments of the present invention.

[0019] [Figure 10] 1 illustrates an exemplary sequence of steps performed by an RPA robot during execution according to some embodiments of the present invention.

[0020] [Figure 11] 1 illustrates exemplary components of a label similarity query according to some embodiments of the present invention.

[0021] [Figure 12] 10 illustrates exemplary similarity indicators associated with respective label similarity queries according to some embodiments of the present invention.

[0022] [Figure 13] 1 illustrates exemplary components of a semantic assessor module according to some embodiments of the present invention.

[0023] [Figure 14] 1 illustrates the operation of an exemplary generative language model (GLM) according to some embodiments of the present invention.

[0024] [Figure 15] 1 illustrates an exemplary structure of a GLM according to some embodiments of the present invention.

[0025] [Figure 16] 1 illustrates an exemplary similarity measure d that quantifies the semantic similarity between two strings according to some embodiments of the present invention.

[0026] [Figure 17] 1 illustrates an exemplary hardware configuration for a computer system that may be programmed to perform some of the methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0027] (Detailed Description of the Invention) In the following description, it is understood that all references to connections between structures may be direct operative connections or indirect operative connections via intermediary structures. A set of elements includes one or more elements. Any reference to an element is understood to refer to at least one element. A plurality of elements includes at least two elements. The use of "or" is meant as a non-exclusive "or." Unless otherwise required, described method steps do not necessarily have to be performed in the particular illustrated order. A first element (e.g., data) derived from a second element encompasses the first element being equal to the second element, the first element generated by processing the second element, and optionally other data. Making a judgment or decision according to a parameter encompasses making a judgment or decision according to the parameter and, optionally, making a judgment or decision according to other data. Unless otherwise specified, an indication of a quantity / data may be the quantity / data itself or an indication different from the quantity / data itself. A computer program is a sequence of processor instructions that perform tasks. Computer programs described in some embodiments of the present invention may be independent software entities or subentities (e.g., subroutines, libraries) of other computer programs. The term "database" is used herein to refer to an organized, searchable collection of data. Semantic similarity herein refers to similarity in meaning, not similarity in wording. In other words, two text samples may be semantically similar even though they are expressed differently. A basic example includes synonyms and semantically related words such as "car" and "vehicle." Conversely, two text samples may be semantically different (have different meanings) even though they are expressed slightly differently, such as in the example sentences "I will go through with the ceremony" and "I will go through the ceremony plans."Computer-readable media encompass non-transitory media such as magnetic, optical, and semiconductor storage media (e.g., hard disks, optical disks, flash memory, DRAM), as well as communication links such as conductive cables and fiber optic links. According to some embodiments, the present invention provides, inter alia, computer systems including hardware (e.g., one or more processors) programmed to perform the methods described herein, and computer-readable media encoding instructions for performing the methods described herein.

[0028] The following description illustrates embodiments of the present invention and is not necessarily limiting.

[0029] FIG. 1 is an architectural diagram illustrating a hyperautomation system 10 according to some embodiments of the present invention. As used herein, "hyperautomation" refers to an automation system that brings together robotic process automation components, integrated tools, and technologies that amplify the ability to automate work. In an exemplary robotic process automation scenario, employees of a company use business applications (e.g., word processors, spreadsheet editors, browsers, email applications) to perform repetitive tasks, such as issuing invoices to various clients. To perform each task, the employees perform a series of operations / actions, which are considered work processes herein. Typical operations that constitute part of an invoice issuing work process may include opening a Microsoft Excel spreadsheet, looking up details about the client's company, copying the respective details into an invoice template, filling in invoice fields indicating purchased items, switching to an email application, composing an email message to each client, attaching the newly created invoice to the respective email message, and clicking a "send" button. The various elements of system 10 may work together to automate each work process by mimicking the set of operations performed by each human operator in the course of performing each task. Mimicking a human movement / action is understood herein to encompass reproducing the sequence of computing events that occur when a human operator performs the respective movement / action on a computer, and reproducing the results of the human operator performing the respective movement / action on a computer. For example, mimicking the action of clicking a button on a graphical user interface (GUI) may include causing the operating system to move a mouse pointer to the respective button and generate a mouse click event, or may alternatively include toggling the respective GUI button itself to a clicked state.

[0030] Examples of processes targeted by RPA include payment processing, invoicing, communication with business clients (e.g., distribution of newsletters and / or product offerings), internal communication (e.g., scheduling memos, meetings and / or tasks), audits, and payroll processing, among others.

[0031] RPA may constitute the core of hyperautomation system 10, and in certain embodiments, automation capabilities may be extended by artificial intelligence (AI) / machine learning (ML), process mining, analytics, and / or other advanced tools. As hyperautomation system 10 learns processes, trains AI / ML models, and employs analytics, for example, more knowledge work may be automated, and computing systems within an organization, e.g., both those used by individuals and those operating autonomously, may all be engaged as participants in the hyperautomation process. The hyperautomation system of some embodiments enables users and organizations to discover, understand, and extend automation efficiently and effectively.

[0032] The exemplary hyperautomation system 10 includes RPA client computing systems 12a-c, such as desktop computers, server computers, and smartphones, among others. Any desired client computing system, including, but not limited to, smartwatches, laptop computers, tablet computers, Internet of Things (IoT) devices, etc., may be used without departing from the scope of the present invention. Also, while only three client computing systems 12a-c are shown in FIG. 1 , any suitable number of client computing systems may be used without departing from the scope of the present invention. For example, in some embodiments, tens, hundreds, thousands, or millions of RPA clients may be used. The RPA clients 12a-c may be actively operated by a user or may run automatically without much or any user input.

[0033] Each illustrated RPA client computing system 12 a-c has a respective automation module(s) 14 a-c executing thereon. Exemplary automation module(s) 14 a-c may include, without limitation, an RPA robot, part of an operating system, downloadable application(s) for the respective computing system, any other suitable software and / or hardware, or any combination thereof, without departing from the scope of the present invention.

[0034] In some embodiments, one or more of the module(s) 14a-c may be listeners. The listeners monitor and record data related to user interactions with their respective computing systems and / or the operation of unattended computing systems and transmit the data to the hyperautomation core system 30 via a communications network 15 (e.g., a local area network—LAN, a mobile communications network, a satellite communications network, the Internet, any combination thereof, etc.). The data may include, but is not limited to, which buttons were clicked, where the mouse was moved, text entered into a field, when one window was minimized and another opened, the application associated with the window, etc. In certain embodiments, data from such listener processes may be transmitted periodically as part of a heartbeat message or upon the achievement of a data accumulation condition. One or more RPA servers 32 receive and store data from the listeners in a database, such as the RPA database(s) 34 of FIG. 1.

[0035] Other exemplary automation modules 14a-c may execute logic that actually implements the automation of a selected process. Stated differently, at least one automation module 14a-c may include part of an RPA robot, as described further below. A robot may be attended (i.e., requiring human intervention) or unattended. In some embodiments, multiple modules 14a-c or computing systems may participate in the execution of an automation's logic. Some automations may orchestrate multiple modules 14a-c, perform various background processing, and / or execute application programming interface (API) calls. Some robotic activities may cause a module 14a-c to wait for a selected task to be completed (perhaps by another entity or automation module) before resuming the current workflow.

[0036] In some embodiments, the hyperautomation core system 30 may execute a conductor application on one or more server computer systems, such as RPA server(s) 32. While FIG. 1 shows only one RPA server 32, multiple or numerous servers, either in close proximity to one another or in a distributed architecture, may be employed without departing from the scope of the present invention. For example, one or more RPA server(s) 32 may be provided for conductor functions, AI / ML model serving, certification, governance, and / or any other suitable functionality without departing from the scope of the present invention. In some embodiments, the hyperautomation core system 30 may incorporate or be part of a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, or the like. In certain embodiments, the hyperautomation core system 30 may host multiple software-based servers on one or more computing systems, such as the RPA server(s) 32. In some embodiments, one or more servers of the core hyperautomation system 30, such as the RPA server(s) 32, may be implemented via one or more virtual machines (VMs).

[0037] In some embodiments, one or more automation modules 14a-c can invoke one or more AI / ML models 36 deployed on or accessible by the hyper-automation core 30. The AI / ML models 36 may be trained for any suitable purpose without departing from the scope of the present invention. Two or more AI / ML models 36 may be chained (e.g., serially, in parallel, or a combination thereof) in some embodiments so that they collectively provide collaborative output(s). Exemplary AI / ML models 36 may perform or assist in computer vision (CV), image processing, segmentation, recognition, optical character recognition (OCR), document processing and / or understanding, semantic learning and / or analysis, analytical prediction, process discovery, task mining, testing, automated RPA workflow generation, sequence extraction, clustering detection, speech-to-text translation, any combination thereof, and the like. However, any desired number and / or type(s) of AI / ML models 36 may be used without departing from the scope of the present invention. Using multiple AI / ML models 36, for example, the system may develop a holistic picture of what is happening on a given computing system. For example, one AI / ML model may perform OCR, another may detect buttons, another may compare sequences, etc. Patterns may be determined by the AI / ML models individually or collectively by multiple AI / ML models. In particular embodiments, one or more AI / ML models 36 are deployed locally on at least one RPA client computing system 12a-c.

[0038] The hyperautomation system 10 may provide at least four main groups of functionality: (1) discovery, (2) automation build, (3) management, and (4) engagement. The discovery functionality may discover various business process automation opportunities and provide automated recommendations thereof. Such functionality may be implemented by one or more servers, such as the RPA server 32. The discovery functionality, in some embodiments, may include providing an automation hub, process mining, task mining, and / or task capture.

[0039] An automation hub (e.g., UiPath Automation Hub™) can provide a mechanism for managing automation rollouts with visibility and control. Automation ideas can be crowdsourced from employees, for example, via a submission form. Calculations of the feasibility and return on investment (ROI) for automating these ideas can be provided, documentation for future automations can be collected, and collaboration can be provided to expedite automation discovery and construction.

[0040] Process mining (e.g., via UiPath Automation Cloud™ and / or UiPath AI Center™) refers to the process of collecting and analyzing data from applications (e.g., enterprise resource planning (ERP) applications, customer relationship management (CRM) applications, email applications, call center applications, etc.) to identify what end-to-end processes exist in an organization, how they can be effectively automated, and the impact of automation. This data is collected, for example, by listeners from RPA clients 12a-c and processed by RPA server(s) 32. One or more AI / ML models 36 may be employed for this purpose. This information may be exported to an automation hub to expedite implementation and avoid manual information transfer. The goal of process mining may be to increase business value by automating processes within an organization. Some example goals of process mining include, but are not limited to, increased profits, improved customer satisfaction, regulatory and / or contractual compliance, improved employee efficiency, etc.

[0041] Task mining (e.g., via UiPath Automation Cloud™ and / or UiPath AI Center™) identifies and aggregates workflows (e.g., employee workflows), then applies AI to uncover patterns and variations in routine tasks and score such tasks for ease of automation and potential savings (e.g., time and / or cost savings). One or more AI / ML models 36 may be employed to uncover repetitive task patterns within the data. Repetitive tasks ripe for automation may then be identified. This information may initially be provided by a listener module (e.g., automation modules 14a-c) and analyzed on the hyperautomation core 30's servers. Results from task mining may be exported to process documents or RPA design applications such as UiPath Studio™ to more quickly create and deploy automations.

[0042] Task mining in some embodiments may include taking screenshots with user actions (e.g., mouse click locations, keyboard input, application windows and graphical elements with which the user was interacting, timestamps for the interactions, etc.), collecting statistical data (e.g., performance time, number of actions, text input, etc.), editing and annotating screenshots, specifying the types of actions to be recorded, etc.

[0043] Task capture (via UiPath Automation Cloud™ and / or UiPath AI Center™) automatically documents attended processes as users work, or provides a framework for unattended processes. Such documentation may include process definition documents (PDDs), skeleton workflows, capturing actions for each part of the process, recording user actions and automatically generating comprehensive workflow diagrams with details about each step, tasks desired to be automated, in formats such as Microsoft Word documents, XAML files, etc. Configurable workflows can, in some embodiments, be exported directly to RPA design applications such as UiPath Studio™. Task capture may simplify the requirements gathering process for both subject matter experts describing the process and Center of Excellence (CoE) members delivering production-grade automation.

[0044] The automation construction functionality of the hyper-automation system 10 may be achieved through a computer program, shown in FIG. 1 as an RPA design application 40. Examples may include UiPath Studio™, UiPath StudioX™, and UiPath Web™, among others. Such computer programs may be used to build and test automation for various applications and environments, such as web, mobile, SAP®, and virtual desktops. In some embodiments, the RPA design application 40 enables a human developer to design workflows that effectively automate a target work process. A workflow typically includes a sequence of custom automation steps, considered herein as an RPA activity. Each activity includes at least one action performed by a robot, such as clicking a button, reading a file, or writing to a spreadsheet cell. Activities may be nested and / or embedded. In some embodiments, the RPA design application 40 exposes a design interface and a set of tools that give developers control over the execution order of a workflow and the relationships between the workflow's activities. In some embodiments, predefined activities, drag-and-drop modeling, and a workflow recorder may facilitate automation with minimal coding. Document understanding functionality may be provided by an AI activity for data extraction and interpretation that invokes one or more AI / ML models 36. Such automation can process virtually any document type and format, including tables, web pages, forms, signatures, and handwriting.

[0045] The RPA design application 40 may also be used to seamlessly combine user interface (UI) automation with API automation, for example, to provide API integration with various other applications, technologies, and platforms. A repository (e.g., the UiPath Object Repository™) or marketplace (e.g., the UiPath Marketplace™) for pre-built RPA and AI templates and solutions may be provided to enable developers to more quickly automate a wide variety of processes. Thus, when building automations, the hyperautomation system 10 may provide a user interface, development environment, API integration, pre-built and / or custom-built AI / ML models, development templates, an integrated development environment (IDE), and advanced AI capabilities. The hyperautomation system 10 may further enable the deployment, management, configuration, monitoring, debugging, and maintenance of RPA robots to execute automations designed using the application 40.

[0046] The management functions of the hyper-automation system 10 can provide automation deployment, orchestration, test management, AI capabilities, and optimization across the organization. Other example aspects of the management functions include DevOps activities such as continuous integration and continuous deployment of automation. The management functions can also act as an integration point with third-party solutions and applications for automation applications and / or RPA robots.

[0047] As an example of management functionality, a conductor application or service can facilitate, among other things, the provisioning, deployment, configuration, queuing, monitoring, logging, and interconnection of RPA robots. Examples of such conductor applications / services include UiPath Orchestrator™ (which can be offered as part of UiPath Automation Cloud™ or on-premise, in a virtual machine, or as a cloud-native single-container suite via the UiPath Automation Suite™). An application / service test suite (e.g., UiPath Test Suite™) can further provide test management for monitoring the quality of deployed automation. The test suite can facilitate test planning and execution, requirements fulfillment, and defect traceability. The test suite can include comprehensive test reports.

[0048] Analytics software (e.g., UiPath Insights™) can track, measure, and manage the performance of deployed automation. Analytics software can align automation operations with specific key performance indicators (KPIs) and strategic outcomes for the organization. Analytics software can present results in a dashboard format that is more easily understood by human users.

[0049] AI management capabilities may be provided by an AI center (e.g., UiPath AI Center™), which facilitates the incorporation of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options may make such capabilities accessible to non-data scientists. Deployed automation (e.g., RPA robots) may invoke AI / ML models 36 from the AI ​​center. AI / ML model performance can be monitored. Models 36 may be trained and improved using human-verified data, such as data provided by a data review center as shown in FIG. 1. Human reviewers may provide labeled data (e.g., a training corpus) to the hyperautomation core 30 via a reviewer application 38 running on a computer connected to the network 15. The reviewer can also use the application 38 to verify that predictions by the AI / ML model 36 are accurate and provide corrections if not. This dynamic input may then be saved as training data for retraining the AI / ML model 36, and may be stored in a database, such as the RPA database 34. The AI ​​center may schedule and execute training jobs to train new versions of the AI / ML model 36 using the training data.

[0050] The engagement capabilities of the hyper-automation system 10 engage humans and automation as one team for seamless collaboration on a desired process. Low-code applications can be built (e.g., via UiPath Apps™) to connect to browsers and legacy software. Applications can be quickly created using a web browser, for example, through a rich library of drag-and-drop controls. Applications can be connected to one automation or multiple automations. An action center (e.g., UiPath Action Center™) can provide a mechanism to hand off processes from robots to humans, and vice versa. Humans can provide approvals or escalations, handle exceptions, etc. The RPA robots can then execute the automated functions of a given workflow.

[0051] A local assistant may be provided as a launch pad for users to launch automations (e.g., UiPath Assistant™). This functionality may be provided, for example, in a tray provided by the operating system and may allow users to interact with RPA robots and RPA robot-powered applications on their computing system. The interface may list approved automations / workflows for a given user and allow the user to execute them. These may include ready-to-use automations from an automation marketplace, an automation hub's internal automation store, etc. When automations are running, they may run as local instances in parallel with other processes on the computing system so that the user can use the computing system while the automation performs its actions. In certain embodiments, the assistant is integrated with task capture functionality so that users can document their soon-to-be-automated processes from the assistant's launch pad.

[0052] In another exemplary engagement feature, chatbots (e.g., UiPath Chatbots™), social messaging applications, and / or voice commands may enable users to execute automations, which may simplify access to information, tools, and resources needed to interact with customers or conduct other activities. For example, chatbots may respond to commands formulated in natural language by triggering robots configured to perform actions such as checking order status, posting data to a CRM, etc.

[0053] In some embodiments, some functions of the hyper-automation system 10 may be provided iteratively and / or recursively: processes may be discovered, automations may be built, tested, and deployed, performance may be measured, automation usage may be facilitated to users, feedback may be obtained, AI / ML models may be trained and retrained, and the process itself may be repeated, thereby facilitating a more robust and effective set of automations.

[0054] FIG. 2 illustrates example components and operation of an RPA system 20 according to some embodiments of the present invention. The RPA system 20 may form part of the hyper-automation system 10 of FIG. 1. The RPA system 20 includes an RPA design application 40 that enables developers to build automation, i.e., design and implement RPA workflows. For example, the application 40 may expose a user interface and a set of tools that give developers control over the order of execution of a workflow and the relationships between the activities in the workflow. One commercial example of an RPA design application 40 is UiPath Studio™.

[0055] Some types of RPA workflows may include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly well-suited for linear processes, allowing the flow of one activity from another without cluttering the workflow. Flowcharts may be particularly well-suited for more complex business logic, allowing for the integration of decisions and the connection of activities in more diverse ways through multiple branching logic operators. FSMs may be particularly well-suited for large workflows. FSMs may use a finite number of states during their execution, triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly well-suited for determining workflow behavior when an execution error is encountered or for debugging the process.

[0056] Once a workflow is developed, it may be encoded in a computer-readable form, such as an RPA script or RPA package 50 (FIG. 2). The RPA script includes a specification of the respective workflow, which is understandable (or interpretable) by the RPA robot 22. The RPA script may be formulated according to any data specification format known in the art, such as Extensible Markup Language (XML), JavaScript Object Notation (JSON), or a version of a programming language such as C#, Visual Basic, or Java. Alternatively, the RPA script may be formulated in an RPA-specific version of bytecode or even as a sequence of instructions formulated in a natural language such as English, Spanish, or Japanese. In some embodiments, one or more related RPA scripts are bundled together with other files and / or metadata to form an RPA package 50. For example, in addition to the RPA script, the RPA package 50 may include a specification of resources required to execute the respective workflow(s). Exemplary resources include, among other things, file locations (e.g., paths, URLs), file names, and sets of credentials for accessing a particular machine, computer program, or service. In what is commonly known in the art as a "build," an RPA script may be pre-compiled into a set of executable files, including a main executable file and accompanying libraries, resource specifications, and metadata, to form an RPA package 50. Package 50 may use any data specification format known in the art. For example, some embodiments of package 50 include a NuGet package of .NET assembly files.

[0057] Those skilled in the art will appreciate that RPA design application 40 may include multiple components / modules, which may execute on separate physical machines. In one such example illustrating a cloud computing embodiment of the present invention, RPA design application 40 may execute in a client-server configuration, where one component of application 40 may expose an automation design interface on a developer's computer, and another component of application 40 executing on a remote server may assemble a workflow and form / output RPA package 50. For example, a developer may access the automation design interface via a web browser executing on the developer's computer, while software that processes user input received on the developer's computer actually executes on the server.

[0058] In some embodiments, workflows designed in the RPA design application 40 are deployed to the RPA conductor 24, for example, in the form of an RPA package as described above. In accordance with the above, in some embodiments, the conductor 24 may be part of the hyper-automation core system 30 shown in Figure 1. One commercial example of the conductor 24 is UiPath Orchestrator™.

[0059] The conductor 24 orchestrates one or more RPA robots 22 that execute respective workflows. Such “orchestration” may include creating, monitoring, and deploying computing resources for the robots 22 in environments such as cloud computing systems and / or local computers. Orchestration may further include, among other things, deploying, configuring, queuing, monitoring, logging, and / or providing interconnectivity for the robots 22. Provisioning may include creating and maintaining connections between the robots 22 and the conductor 24. Deployment may include ensuring accurate delivery of software (e.g., RPA packages 50, individual workflow specifications) to the robots 22 for execution. Configuration may include maintaining and distributing robot environment and workflow configurations. Queuing may include providing job queues and queue item management. Monitoring may include keeping track of robot states and maintaining user permissions. Logging may include storing and indexing logs in a database and / or another storage mechanism (e.g., SQL, ElasticSearch®, Redis®). Conductor 24 may also serve as a centralized point of communication for third-party solutions and / or applications. As described further below, in some embodiments, conductor 24 may also provide automation troubleshooting services and assistance.

[0060] RPA robots 22 are execution agents (e.g., computer programs) that implement automation workflows for a variety of systems and applications, including, but not limited to, mainframes, web applications, virtual machines, enterprise applications (e.g., those created by SAP®, Salesforce®, Oracle®, etc.), desktop and laptop applications, mobile device applications, wearable computer applications, etc. One commercial example of a robot 22 is UiPath Robots™.

[0061] In some embodiments, to mimic a human user's interaction with the target application's user interface, the RPA robot 22 interfaces with a set of RPA drivers 25 running on each RPA client / host computer. Such drivers generally represent software modules that perform low-level operations such as moving a cursor on the screen, registering and / or fulfilling mouse, keyboard, and / or touchscreen events, detecting the handheld device's current pose / orientation, detecting current accelerometer readings, taking pictures with a smartphone camera, and taking screenshots of the respective device. Some of these drivers form part of the local operating system. Other RPA drivers 25 may implement various application-specific aspects of a user's interaction with complex target applications such as SAP®, Citrix® virtualization software, Microsoft Excel®, etc. One particular example includes a browser driver, which may be embodied as a set of browser-compatible scripts (e.g., JavaScript®). When injected into a web page currently displayed in a browser, such a browser driver may identify various elements of the respective web page (e.g., buttons, menus, form fields, etc.) and invoke specific functions of each element (e.g., filling in form fields, selecting menu items, toggling checkboxes, etc.). Other exemplary RPA drivers 25 include Microsoft® WinAppDriver, Apple, Inc.'s XCTest driver, and Google, Inc.'s UI Automator driver.

[0062] Robot types can include, among others, attended robots 122, unattended robots 222, development robots (similar to unattended robots but used for development and testing purposes), and non-production robots (similar to attended robots but used for development and testing purposes). Some activities of attended robots 122 are triggered by user events and / or commands, and they operate alongside human operators on the same computing system. In some embodiments, attended robots can be started only from the robot tray or from a command prompt and therefore cannot be fully controlled from the conductor 24 and cannot, for example, run under a locked screen. Unattended robots can run unattended in a remote virtual environment and can be responsible for remote execution, monitoring, scheduling, and providing work queue support.

[0063] In some embodiments implemented in a Windows environment, the robot 22 installs as a Microsoft Windows Service Control Manager (SCM)-managed service by default. As a result, such a robot can open an interactive Windows session under the local system account and have the processor privileges of a Windows service. For example, a console application can be launched by an SCM-managed robot. In some embodiments, the robot 22 can be installed with user-level processor privileges (user mode, ring 3). Such robots have the same permissions as the user under which they are installed. For example, such robots can launch any applications that each user can run. On a computing system that supports multiple simultaneous interactive sessions (e.g., Windows Server 2012), multiple robots can run simultaneously, each running in a separate Windows session using a different user name.

[0064] In some embodiments, the robot 22 is divided into multiple components, each specialized for a particular automation task. Robot components in some embodiments include, but are not limited to, an SCM management robot service, a user-mode robot service, an executor, an agent, and a command line. Depending on the specifics of the platform, the SCM management and / or user-mode robot service manage and monitor Windows sessions and act as a proxy between the conductor 24 and the host machine (i.e., the computing system on which the robot 22 executes). These services are responsible for managing credentials for the robot 22. The command line is a client of the service(s) and is a console application that can be used to launch jobs and display or otherwise process their output.

[0065] An exemplary set of robot executors 26 and RPA agents 28 is shown in FIG. 3. The robot executors 26 are capable of executing a given job under a Windows® session. The executors 26 are configured to receive RPA packages 50 that specify a workflow (e.g., a sequence of robotic activities) and to execute the respective packages, which will substantially perform the respective sequences of RPA activities. In some embodiments, the packages 50 include pre-compiled executable code. In other exemplary embodiments, the robot executor(s) 26 include an interpreter (e.g., a just-in-time interpreter or compiler) configured to convert a received RPA script that includes a workflow specification (e.g., bytecode, XML, JSON, etc.) into runtime code that includes processor instructions for executing the respective workflow. Thus, execution of the RPA package 50 involves the executor(s) 26 translating the workflow specification included in the package 50 and instructing the processor of the respective host machine to load the resulting runtime code into memory and launch the runtime code during execution.

[0066] The RPA agent 28 may manage the operation of the robot executor(s) 26. For example, the RPA agent 28 may select tasks / scripts for performance by the robot executor(s) 26 according to input from a human operator and / or according to a schedule. The agent 28 may start and stop jobs and configure various operating parameters of the executor(s) 22. If the robot 22 includes multiple executors 26, the agent 28 may coordinate their activities and / or inter-process communications. The RPA agent 28 may further manage communications between the RPA robot 22 and the conductor 24, and / or other entities.

[0067] The example RPA system 20 of FIG. 2 forms part of the hyperautomation system 10 (see FIG. 1). As such, the robots 22 can interact with various components and use various aspects of the hyperautomation core system 30, generally shown in FIG. 2 as the hyperautomation service 23. For example, a developer can use an RPA design application 40 to build and test an RPA robot 22 that utilizes an AI / ML model 36. Such an RPA robot 22 can send inputs for execution of the AI / ML model(s) and receive outputs therefrom via the hyperautomation core system 30. The robots 22 may also be listeners, as described above. These listeners can provide information to the core hyperautomation system 30 about what users are doing when using their computing system. This information can then be used by the hyperautomation system 30 for process mining, task mining, task capture, etc. In another example embodiment, the hyperautomation service 23 can expose data labeling functionality to users of the computing system hosting the robots 22 or to another computing system to which the robots 22 provide information. For example, if the robot 22 invokes a computer vision AI / ML model 36 but the respective model does not correctly identify a button on the screen, the user may explicitly provide the correct identification. Such information may be passed to the hyperautomation core system 30 and subsequently used to retrain the respective AI / ML model.

[0068] In some embodiments, selected components of the hyperautomation system 10 and / or the RPA system 20 can be implemented in a client-server configuration. In one such configuration, shown in FIG. 3, the RPA robot 20, including the executor(s) 26 and the RPA agent 28, can be implemented on the client side, e.g., on one of the RPA client computers 12a-c of FIG. 1. The functionality of the conductor 24 and / or other services of the hyperautomation core system 30 can then be implemented on the server side, e.g., on a remote RPA server 32 (FIG. 1). It should be noted that the client side, the server side, or both can include any desired number of computing systems (e.g., physical or virtual machines) without departing from the scope of the present invention. The illustrated RPA system can be cloud-based, on-premise, or a combination thereof, providing enterprise-level, user-level, or device-level automation solutions for automating various work processes.

[0069] The robot 22 may execute several jobs / workflows simultaneously. The RPA agent 28 (e.g., a Windows service) may act as a single client-side point of contact for multiple executors 26. The agent 28 may further manage communication between the robot 22 and the conductor 24. In some embodiments, communication is initiated by the RPA agent 28, which may open a WebSocket channel to the conductor 24. The agent 28 may then use the channel to send notifications about the status of each executor 26 to the conductor 24, for example, as a heartbeat signal. The conductor 24 may then use the channel to send approvals, job requests, and other data, such as RPA packages 50, to the robot 22.

[0070] 3 , the conductor 24 includes a web interface 42 and a set of service modules, including a set of application programming interface (API) endpoints 43 and service API / business logic 44. A user can interact with the conductor 24 via the web interface 42 (e.g., by opening a dedicated web page in a browser 16) and instruct the conductor 24 to perform actions such as scheduling and / or starting jobs on robots 22, creating robot groups / pools, assigning workflows to robots, adding / removing data to / from queues, and analyzing logs per robot or workflow. The interface 42 can be implemented using Hypertext Markup Language (HTML), JavaScript (JS), or any other data format known in the art.

[0071] The conductor 24 may perform actions requested by a user by selectively invoking the service API / business logic 44 via the endpoint 43. Additionally, in some embodiments, the API endpoint 43 is used to communicate between the RPA robot 22 and the conductor 24 for tasks such as configuration, logging, deployment, monitoring, and queuing, among others. The API endpoint 43 may be set up using any data format and / or communication protocol known in the art. For example, the API endpoint 43 may conform to Representational State Transfer (REST) ​​and / or Open Data Protocol (OData).

[0072] The configuration endpoint may be used to define and configure application users, permissions, robots, assets, releases, etc. The logging endpoint may be used to log different information such as errors, explicit messages sent by the robot 22, and other environment-specific information. The robot 22 may use the deployment endpoint to query the version of the RPA package 50 to execute. The queuing endpoint may be responsible for managing queues and queue items, such as adding data to the queue, retrieving transactions from the queue, and setting the state of transactions. The monitoring endpoint may monitor the performance of the web interface 42 and / or the RPA agent 28.

[0073] The service API 44 includes computer programs that are accessed / invoked through configuration of appropriate API access paths, for example, based on whether the conductor 24 and the overall hyperautomation system have an on-premise or cloud-based deployment type. The exemplary API 44 provides a custom method for querying statistics about various entities registered with the conductor 24. Each logical resource may, in some embodiments, be an OData entity. In such entities, components such as robots, processes, queues, etc. may have properties, relationships, and behaviours. The API 44 may be consumed by the web application 42 and / or the RPA agent 28 by obtaining appropriate API access information from the conductor 24 or by registering an external application using the OAuth flow mechanism.

[0074] In some embodiments, the persistence layer of the server-side operations implements a database service. The database server 45 may be configured to selectively store and / or retrieve data to / from the RPA database 34. The database server 45 and the database 34 may employ any data storage protocol and format known in the art, such as Structured Query Language (SQL), ElasticSearch®, and Redis®, among others. Exemplary data stored / retrieved by the server 45 may include configuration parameters of the robots 22 and robot pools, as well as data characterizing workflows performed by the robots 22, data characterizing users, roles, schedules, queues, etc. In some embodiments, such information is managed via the web interface 42. Another exemplary category of data stored and / or retrieved by the database server 45 includes data characterizing the current state of each performing robot, as well as messages logged by the robots during execution. Such data may be transmitted by the robots 22 via the API endpoint 43 and centrally managed by the conductor 24, for example, via the API logic 44.

[0075] The server 45 and database 34 also store / manage process mining, task mining, and / or task capture related data received, for example, from listener modules operating on the client side as described above. In one such example, the listeners may record user actions performed on their local hosts (e.g., clicks, typed characters, locations, applications, active elements, times, etc.) and then convert these into an appropriate format to be provided to and stored in the database 34.

[0076] In some embodiments, a dedicated AI / ML server 46 facilitates the incorporation of AI / ML models 36 into automation. Pre-built AI / ML models, model templates, and various deployment options make such capabilities accessible to operators without advanced AI / ML knowledge or expertise. Deployed robots 22 can invoke AI / ML models 36 by interfacing with the AI / ML server 46. The performance of deployed AI / ML models 36 can be monitored, and each model can be retrained and improved using human-verified data. The AI / ML server 46 can schedule and execute training jobs and manage training corpora. The AI / ML server 46 can also manage data related to AI / ML models 36, algorithms and software packages for various AI / ML functions, including, but not limited to, document understanding techniques and frameworks, intent analysis, natural language processing (NLP), speech analysis and synthesis, and computer vision (image processing, segmentation, and recognition).

[0077] Embodiments of the present invention are directed to automating interactions with user interfaces. FIG. 4 illustrates an exemplary user interface 37 according to some embodiments of the present invention. Generally, a user interface (UI) is a computer interface that enables human-machine interaction, such as an interface configured to receive user input, respond to the respective input, and return computational results to the user. A user interface often includes a visual representation of a target document (e.g., an HTML document, a form, a spreadsheet) and a set of UI control elements that allow a user to manipulate or interact with the respective target document. A common example of a user interface is known as a graphical user interface (GUI), which enables human-machine interaction through a set of graphical UI elements displayed to the user. Exemplary UI elements shown in FIG. 4 include windows 62a-c, a menu 64a, an icon 64b, a button 64c, an input field 64d, and hyperlinked text 64e (also known as a link label or anchor text). Other exemplary UI elements include a label, a text area, a form with multiple input fields, and a toggle, among others. Some UI elements may be nested within other UI elements: for example, a menu may have multiple menu items and / or submenus, a form may have multiple input fields of various types, etc. UI elements may fulfill different functional roles, such as containers for displaying information, input controls, navigation controls, etc.

[0078] In typical UI automation, an RPA robot is configured to mimic a human user's interactions with various elements of the target UI, such as a user clicking a button 64c or filling out an input field 64d of the target UI 37. RPA typically involves two distinct stages. In the first stage, referred to herein as design time, an RPA designer configures the RPA robot to perform the required automation. Designing each automation may include specifying a set of RPA activities to perform and providing data that enables the robot to correctly identify the target of each activity, i.e., the correct input field to fill in, the correct button to click, etc. Target identification is typically performed according to a set of attributes specific to each target. Target characteristics can be programmatic (i.e., extracted from or determined according to source code and / or an internal computer representation of the target document, such as a UI tree or DOM) and / or visual (e.g., on-screen position, image, color, label, etc.). Once the target characteristics are determined, they can be included in a workflow specification. In the next stage of automation, commonly known as runtime, the RPA robot effectively executes the respective workflow, i.e., performs the RPA activities as specified in the workflow specification. To achieve this, the robot must correctly identify the activity targets in the target UI and act on them.

[0079] Importantly, the target UI used at design time (referred to herein as the design-time UI) is typically not the same as the target UI used at runtime (referred to herein as the runtime UI) because the design and execution of the respective workflows may be separated in space and time. Instead, the design-time UI and runtime UI are simply instances of the same target UI, with each target UI defined by the identity of the target document rendered by the respective target UI. Otherwise, both the design-time UI and the runtime UI display a document / resource with the same identifier (e.g., document name, location such as a Universal Resource Identifier-URI or Universal Resource Locator-URL). However, because the target document is maintained independently of the automation itself, the content and / or layout of the target document may change unexpectedly between design time and runtime. Such changes may result in different design-time and runtime instances of the target UI. For example, some target-specific data for various UI elements may change, causing the automation to fail.

[0080] 5, an RPA robot is tasked with automatically filling out a web-based airport check-in form (i.e., a target document herein comprising an HTML form with a designated URL) with passenger data. At design time, an automation designer designs an RPA workflow that includes activities for interacting with a design-time instance of a target UI, herein exemplary design-time UI 37a. For example, an RPA activity in the workflow may include filling in target input field 64e, and another RPA activity in the workflow may include clicking button 64g.

[0081] FIG. 5b further illustrates an exemplary runtime UI 37b that the RPA robot 22 encounters at runtime. The runtime UI 37b is an instance of the same web form used to design the respective automation (e.g., as indicated by the same URL), but differs slightly from the design-time UI 37a. Exemplary changes include changes to the position and / or relative position of some input fields, some menu items, and some input field labels. Furthermore, the type of visual element used to label the input fields has been changed, for example, from simple text in UI 37a to a default input / placeholder value in UI 37b. Such changes may prevent the RPA robot from correctly identifying the activity target, resulting in an automation failure. In the example illustrated in FIG. 5, the robot may be configured to fill in input field 64e but may be unable to find an input field with the respective characteristics in runtime UI 37b. Similarly, a robot configured to click button 64g may be unable to find such a button in the runtime UI. Some embodiments of the present invention directly address such shortcomings.

[0082] FIG. 6 illustrates an exemplary data exchange according to some embodiments of the present invention. At design time, an automation designer designs an automation workflow using an instance of an RPA design application 40. The application 40 can interact with a design-time instance of a target UI, i.e., design-time UI 37a. An RPA package 50 containing the specification of the respective workflow is then delivered to the RPA conductor 24 for further distribution to the RPA clients 12, which generally represent any of the RPA clients 12a-c of FIG. 1. In the airport check-in automation example above, the RPA clients 12 may include desktop computers located in airport terminals or cloud computing platforms operated by the respective airlines. The RPA clients 12 execute instances of the RPA robot 22, which interact with the runtime instance of the target UI (runtime UI 37b) via local instances of the RPA driver(s) 25 and send status reports 55 regarding the progress of the respective automations to the RPA conductor 24. In some embodiments of the present invention, the RPA robot 22 and / or driver(s) 25 may cooperate with a semantic evaluation module 70 to correctly identify activity targets, as described in more detail below. Those skilled in the art will recognize that the system shown in FIG. 6 is merely exemplary and may be modified without changing the scope of the present invention. For example, in alternative embodiments, each automation may be performed locally, i.e., on a computer that also executes the RPA design application 40, without the involvement of the RPA conductor 24.

[0083] FIG. 7 illustrates an exemplary sequence of steps performed by the RPA design application 40 to design an automation workflow for interaction with a target UI, according to some embodiments of the present invention. Step 702 exposes a robot design interface to a user / automation designer. An exemplary robot design interface 47 is shown in FIG. 8, depicting a visual representation of an RPA workflow 48 as a sequence of RPA activities. Those skilled in the art will appreciate that the content and appearance of the interface 47 are exemplary only and not limiting. In the illustrated example, each RPA activity or group of activities in the workflow is represented by an individual activity container 49 a-c, and each such container optionally displays an activity configuration interface for configuring various parameters of the respective RPA activity. The containers 49 a-c may include child windows of the interface 47. In some embodiments, the containers 49 a-c may be nested, i.e., some containers may further include a hierarchy of sub-containers representing individual RPA activities, etc. The robot design interface 47 may further expose various controls that allow a user to add, remove, and rearrange activities in the workflow 48.

[0084] In some embodiments as shown, the robot design interface 47 further displays an activity menu 51 that lists or otherwise allows the user to select RPA activities for inclusion in the workflow 48. The activities may be grouped according to various criteria, such as according to the type of user interaction (e.g., click, tap, gesture, hotkey), according to the type of data (e.g., text-related activity, image-related activity), according to the type of data processing (e.g., navigation, data scraping, form filling), the type of target application (e.g., browser, spreadsheet, word processing), etc. In some embodiments, individual RPA activities may be reached via a hierarchy of submenus.

[0085] In step 704, the RPA design application 40 may receive user input selecting an RPA activity for inclusion in the workflow 48. Step 704 may further include redrawing the workflow 48 to include the newly selected RPA activity, e.g., adding an activity container and placing it at the desired location within the workflow 48.

[0086] Some RPA activities launched via activity menu 51 may include semantic targeting, which refers herein to identifying the target of each activity according to semantic criteria, such as the meaning of a label attached to a target UI element, as described below. In some embodiments, a user may choose between activities that use semantic targeting and activities that do not. For example, menu 51 may include a form-filling activity that identifies target input fields by traditional means (e.g., via programmatic attributes, such as a set of attribute-value pairs that characterize each UI element and are determined according to a DOM) and another form-filling activity that uses semantic targeting to identify each input field. Thus, a user may select between the two based on the observation that each activity may be suitable for a different type of target UI.

[0087] Step 706 may determine whether the selected RPA activity includes semantic targeting. If no, some embodiments may proceed to activity-specific configuration actions beyond the scope of this description. If the selected activity includes semantic targeting (step 704 returns yes), step 708 application 40 may receive user input indicating a target UI element, herein indicating a UI element of UI 37a targeted by the respective RPA, such as a button to be clicked, an input field to be filled out, etc. In some embodiments, the target selection user input includes the user hovering, clicking, or tapping over a desired element in UI 37a. Some embodiments may further display a semantic target selection interface to the user, for example, as an overlay. An exemplary semantic target selection interface described herein is shown in FIG. 9, with the target element selected by the user (in the illustrated example, an input field in UI 37a) highlighted for clarity. The target selection interface may further allow the user to confirm or cancel the selected target element.

[0088] In step 710, some embodiments can determine a design-time target label, herein a text label / string displayed within the design-time UI 37a near the selected target element. Exemplary target labels shown in FIG. 5 include the label 66a associated with the target input field 64a and the label "Save" displayed on the target button 64g. Determining the design-time target label may include using the RPA driver(s) 25 to analyze the UI 37a (e.g., the DOM in the case of a web page as shown in FIG. 5) and identify candidate labels. Exemplary candidate labels include, among other things, fragments of text displayed by the UI, such as titles of sections of the target document, labels of menu items displayed by the UI, default values ​​of input fields, placeholder values ​​of input fields, and text content of alternative text or tooltips displayed when hovering a mouse over the target element. Other exemplary candidate labels, such as element names, may be determined according to the source code and / or internal computer representation (e.g., HTML, DOM) of the respective target document. Some embodiments may then apply a set of heuristics to cull multiple candidate labels, based, for example, on the observation that labels are typically short (e.g., a few words), located closest to the associated target element, and typically aligned with the respective target element (e.g., directly above, above, to the left, etc.).

[0089] Alternative embodiments may use artificial intelligence / machine learning to determine the target label. In one such example, step 710 may include taking snapshots of at least one area of ​​UI 37a that includes a user-selected target element and sending each snapshot to a pre-trained AI / ML module for analysis. Each module may form part of AI module 36 described in connection with FIG. 1 and may combine image processing with optical character recognition to determine the target label according to the received snapshot. In some embodiments, step 710 may further include displaying the detected target label to the user (e.g., as shown in FIG. 9) for confirmation. If automatic label detection fails, the user may be prompted to manually indicate the target label.

[0090] In a further step 712, some embodiments of the robot design application 40 may receive user input that further configures various parameters of the selected RPA activity. In the example of input fields, step 712 may receive user input indicating values ​​to be entered into the respective input fields at runtime. Each value may be explicit (e.g., a user-provided text string) or may reference another data structure, and in some cases may include the output of another RPA activity in the workflow 48.

[0091] The workflow design process can continue with the user selecting and configuring other RPA activities as described above. Once the design of the current workflow is complete (step 714 returns yes), in a further step 716, some embodiments can form an RPA package 50 for the current workflow that includes computer-readable encodings of the selected RPA activities and target-specific data, such as design-time target labels, determined in step 710.

[0092] FIG. 10 illustrates an exemplary sequence of steps performed by the RPA robot 22 at runtime, according to some embodiments of the present invention. In step 1002, the robot 22 may receive an RPA package 50 that specifies a workflow for execution. Depending on the embodiment, the package 50 is received from the RPA conductor 24 or directly from an instance of the RPA design application 40, as described above. In a further step 1004, the robot 22 may publish a runtime instance of a target UI, shown herein as runtime UI 37b (see, e.g., FIG. 5 ). In an exemplary use case scenario in which the target UI includes a web page, the robot 22 may invoke an instance of a browser window and navigate to the target URL indicated in the RPA package 50. Similar operations may publish the runtime UI 37 if the current workflow targets other types of applications (e.g., electronic communication applications, spreadsheet applications, etc.). Some embodiments then cooperate with the RPA driver(s) 25 to analyze the runtime UI 37, e.g., to enumerate UI elements for identification as candidate targets for various RPA activities. In the example of web page automation, step 1004 may include, for example, building and / or analyzing the DOM of each web page.

[0093] Some embodiments may then iterate through all RPA activities of the respective workflow as described in the received RPA package. Step 1006 may select an RPA activity from the workflow. The specification of each activity typically includes a set of target-specific data that enables the robot 22 to correctly identify the target runtime instance of the respective RPA activity. In a further step 1008, the robot 22 may extract design-time labels of the respective target UI elements from the target-specific data. Such labels were included in the RPA package at design time (see, e.g., steps 710 and 716 of FIG. 7).

[0094] In a series of steps 1010-1012, the robot 22 may then identify a set of candidate target elements in the runtime UI 37b according to the design-time targeting data, such as the design-time target label. Step 1010 may include analyzing the runtime UI 37 to identify a set of UI elements having the same type (e.g., input field) as the target of the current RPA activity. For each such candidate UI element, step 1012 may determine a runtime label, i.e., a label associated with the respective UI element in the runtime UI 37b. Exemplary runtime labels include label 66b and the word "OK" in the UI 37b of FIG. 5. Determining the runtime label may include a procedure similar to that used at design time to determine the design-time target label (e.g., see step 710 of FIG. 7).

[0095] Step 1014 then determines whether any of the runtime labels match the design-time target labels. If yes, then in step 1016, the RPA robot 25 identifies the runtime target elements as target candidates whose runtime labels exactly match the labels determined at design time.

[0096] If none of the runtime labels match the design-time target labels, some embodiments may identify a runtime target according to the similarity between the design-time target labels and the runtime labels. As shown in FIG. 6 , some embodiments may send a label similarity query 72 to a semantic evaluator 70 and, in response, receive a similarity indicator 74 from the evaluator 70, the similarity indicator 74 being determined according to the similarity query 72 and quantifying the similarity between the design-time label of the target UI element and the label of the runtime target candidate. In some embodiments, the semantic evaluation module 70 includes a set of computer programs that can be executed locally on the RPA client 12 or on a remote computer system. For example, the semantic evaluator 70 can form part of the AI ​​module 36 ( FIG. 1 ) and / or be executed on the AI / ML server 46 ( FIG. 3 ). In alternative embodiments, some or all of the evaluator 70 may be implemented in dedicated hardware or as a combination of hardware and software. A single instance of the assessment module 70 can perform semantic similarity calculations for multiple RPA clients 12.

[0097] In step 1018, the robot 22 may formulate a label similarity query 72. An exemplary query 72 according to some embodiments is shown in FIG. 11. The query 72 includes an encoding of a design-time target label 66a, i.e., a label associated with a target UI element in the design-time UI 37a and included as target-specific data in the RPA package 50. The query 72 further includes an encoding of at least one runtime label 66b, i.e., a label associated with a target candidate in the runtime UI 37b. Those skilled in the art will appreciate that the form and content of the query 72 may vary without departing from the scope of the present invention. It will also be apparent that in alternative embodiments, the design-time and runtime labels may be submitted in separate query / data packages.

[0098] 12 illustrates an exemplary similarity indicator 74 according to some embodiments of the present invention. The similarity indicator 74 quantifies the semantic similarity between the design-time target label and each runtime label included in the respective label similarity query 72. In the illustrated example, the indicator 74 comprises a numeric value between 0 and 1, with 0 indicating no similarity and 1 indicating an exact match. However, the format and data type of the indicator 74 may vary without departing from the scope of the present invention. An alternative exemplary indicator 74 may be a Boolean value, with yes indicating a match and no indicating no semantic similarity, etc.

[0099] The semantic evaluator 70 can use any method known in the art to evaluate the semantic similarity between design-time and runtime labels. A basic embodiment can use an annotated dictionary and / or thesaurus to determine whether two items are semantically similar (e.g., synonyms). Other embodiments can maintain a searchable database of natural language synsets, i.e., sets of words and phrases that are semantically similar to one another (e.g., last names, surnames, and family names). One example of such a database developed for the English language is WordNet, operated by Princeton University in the United States. Still other embodiments can maintain a real-world collection of UI element labels and their runtime counterparts, collected by instances of the RPA robot 22 interacting with various target UIs, for example. The respective labels can be organized according to element type (e.g., input field labels vs. button labels).

[0100] More sophisticated embodiments of the semantic evaluator 70 can rely on a language model (LM), which is a computational probabilistic model of natural language. Examples include word n-gram models, skip-gram models, and large-scale language models (LLMs), among others. FIG. 13 shows an exemplary semantic evaluator 70 including a generative language model (GLM) 71 communicatively coupled to a semantic distance calculator 73. The GLM 71 is considered “generative” herein in the sense that it is configured to input a sequence of words and, in response, automatically generate another sequence of words that contains a plausible continuation of the input word sequence. The GLM 71 may be implemented using any method known in the art of artificial intelligence. For example, the GLM 71 may include a set of artificial neural networks (e.g., recurrent neural networks, generative pre-trained transformers (GPTs), etc.) trained on a corpus of text formulated in the respective natural language. In some embodiments, GLM71 implements instances of pre-trained, off-the-shelf LLMs, such as GPT-3 from OpenAI, LLaMA from Meta AI, and Mistral from Mistral AI, among others. Details of the structure and operation of such models are beyond the scope of this invention.

[0101] The basic operation of a GLM 71 according to some embodiments of the present invention is illustrated in FIG. 14. The GLM 71 receives a language model prompt 75 containing a sequence of text tokens 76a-d and, in response, outputs a predicted text token 77 determined according to the LM prompt 75, where the token 77 contains a possible continuation of the sequence of tokens 76a-d in the LM prompt 75. The set of calculations performed by the GLM 71 to generate each predicted token (excluding the LM initialization step described below) is commonly known in the art as the inference step. If the architecture of the GLM 71 is based on a neural network, such calculations may include, among other things, matrix multiplications and evaluation of a set of activation functions. Training such a GLM typically involves selecting an input token sequence from a training corpus of actual text samples, comparing the predicted tokens 77 with the actual continuation of the input sequence in each text sample, and adjusting a set of internal parameters of the GLM (e.g., neural synaptic weights) with the aim of correcting erroneous inference. Modern GLMs typically have tunable parameters on the order of hundreds of thousands to billions. Training may employ any machine learning algorithm known in the art, including versions of backpropagation, among others.

[0102] In some embodiments, the GLM 71 includes a sandwich of neural network layers, as shown in FIG. 15, where a first set of layers acts as an encoder that receives input tokens 76 and outputs an internal representation of each token, commonly known in the art as an embedding vector 78, or simply an embedding. The embedding vector 78 includes a set of numbers collectively equal to the projection of the token 76 in an abstract multidimensional space known as the embedding space. The coordinates of the embedding vector depend not only on each token but also on the training corpus, since the parameters of the encoder layers are determined by training. Some embodiments generate a separate vector 78 for each input token of the LM prompt 75. Another set of neural network layers then acts as a decoder, taking such embedding vectors 78 and transforming them to generate predicted tokens 77.

[0103] FIG. 16 illustrates an exemplary embedding space spanned by two abstract axes and two embedding vectors 78a-b representing a design-time target label 66a and a runtime label 66b, respectively. Those skilled in the art will appreciate that this diagram is merely schematic and not limiting, as the embedding space typically has thousands of dimensions. In some embodiments, a semantic distance calculator 73 can calculate a similarity measure that quantifies the semantic similarity between the design-time label and the runtime label according to each embedding vector. In such an embodiment, the calculator 73 can receive the output of the encoder portion of the GLM 71 and evaluate the similarity measure according to the distance between the embedding vector representing the design-time target label and each runtime label of the target candidate. In the example shown in FIG. 16, the similarity measure between the labels 66a-b can be determined according to the distance d separating the vectors 78a-b in the embedding space. The distance can be calculated using any formula known in the art, such as Euclidean distance, Manhattan distance, cosine distance, etc., or a combination thereof.

[0104] In some embodiments, in a series of steps 1020-1022 ( FIG. 10 ), the robot 22 may send a label similarity query 72 to the assessment module 70 and, in response, receive a similarity indicator 74 calculated according to the query 72. In some embodiments, such as shown in FIG. 12 , the similarity indicator 74 may include a set of similarity measures, each determined for a distinct runtime label and indicating the degree of similarity between the design-time target label and the respective runtime label. Such similarity measures may be scaled so that values ​​closer to 1 indicate strong similarity and values ​​closer to 0 indicate no similarity. Step 1024 may identify a runtime target according to the content of the similarity indicators 74. For example, the robot 22 may select, as the runtime target for the current RPA activity, an element of the runtime UI 37 whose runtime label is most similar to the design-time target label according to the similarity indicators 74. Some embodiments may also implement an error prevention strategy by comparing the received similarity indicator(s) with a predetermined threshold and rejecting all target candidates having runtime labels with similarity measures below the threshold. If the runtime label is not sufficiently similar to the design-time target label, some embodiments determine that the differences between the design-time UI 37a and the runtime UI 37b are too great to safely continue, abort the execution of the current workflow, and send a status report 55 notifying of the error.

[0105] Next, step 1026 performs the respective RPA activity (e.g., clicking a specified button, filling in a specified input field, etc.). If the performance is successful (step 1028 returns Yes), the robot 22 may proceed to the next RPA activity in the respective workflow. Once all activities have been performed, step 1030 may send a status report 55, for example, to the RPA conductor 24.

[0106] Some embodiments rely on the observation that semantic targeting described herein is significantly more computationally expensive than traditional targeting, for example, by matching sets of program attribute-value pairs extracted from a DOM / UI tree. High-reliability language models are relatively large and expensive to train and execute. Furthermore, sending label similarity queries to a remote server inherently slows targeting, impacting productivity and user experience. Therefore, to conserve computational resources and improve user experience, some embodiments may integrate semantic matching into a targeting optimization strategy. In a first step, the RPA robot 22 may attempt to identify runtime targets according to traditional methods. If such efforts fail, a second step may use semantic targeting as a fallback.

[0107] FIG. 17 illustrates an exemplary hardware configuration of a computer system 80 programmed to perform some of the methods described herein. The computer system 80 may represent any of the RPA clients 12a-c, as well as the RPA server(s) 32, or any other computer that performs the RPA robot 22 and / or semantic assessor module 70. The illustrated appliance is a personal computer; other computer systems, such as servers, mobile phones, tablet computers, and wearable computing devices, may have slightly different configurations. The processor(s) 82 include physical devices (e.g., microprocessors, multi-core integrated circuits formed on a semiconductor substrate) configured to perform calculations and / or logical operations using sets of signals and / or data. Such signals or data may be encoded and sent to the processor(s) 82 in the form of processor instructions, e.g., machine code. The processor 82 may include an array of central processing units (CPUs) and / or graphics processing units (GPUs).

[0108] The memory unit 83 may include volatile computer-readable media (e.g., dynamic random access memory—DRAM) that store data and / or instruction encodings accessed or generated by the processor(s) 82 in the course of performing operations. The input devices 84 may include, among other things, a computer keyboard, a mouse, a trackpad, and a microphone, including respective hardware interfaces and / or adapters that enable a user to introduce data and / or instructions into the computer system 80. The output devices 85 may include, among other things, display devices such as a monitor and speakers, and hardware interfaces / adapters such as a graphics card, that enable the respective computing devices to communicate data to a user. In some embodiments, the input and output devices 84-85 share a common piece of hardware (e.g., a touchscreen). The storage device 86 includes computer-readable media that enable non-volatile storage, reading, and writing of software instructions and / or data. Exemplary storage devices include magnetic and optical disks and flash memory devices, as well as removable media such as CD and / or DVD disks and drives. Network adapter(s) 87 contain mechanical, electrical, and signaling circuitry for communicating data over a physical link coupled to an electronic communications network (e.g., FIG. 1) and / or other device / computer system. Adapter(s) 87 may be configured to transmit and / or receive data using a variety of communications protocols.

[0109] Controller hub 90 generally represents multiple system, peripheral, and / or chipset buses, and / or all other circuitry that enables communication between processor(s) 82 and the remaining hardware components of computer system 80. For example, controller hub 90 may include a memory controller, an input / output (I / O) controller, and an interrupt controller. Depending on the hardware manufacturer, some such controllers may be incorporated into a single integrated circuit and / or may be integrated with processor(s) 82. In another example, controller hub 90 may include a northbridge that connects processor 82 to memory 83 and / or a southbridge that connects processor 82 to devices 84, 85, 86, and 87.

[0110] The exemplary systems and methods described above facilitate UI automation by improving the automatic identification of activity targets—UI elements manipulated by robotic software. In many RPA applications, target identification poses a significant technical challenge because the functionality and / or appearance of the target UI can suddenly change in ways beyond the control of the robot designer. Figure 5 illustrates examples of changes between design time and runtime commonly seen in real-world applications. Some UI elements may be removed, while others may be rearranged within the target UI. The configuration of menus and the order and labeling of individual menu items may change. Element attributes such as color, font, and label may be altered. In traditional RPA, robots are typically trained to recognize activity targets (e.g., buttons to click, input fields to fill) according to such characteristics, so arbitrary changes to such characteristics can cause the respective automation to fail.

[0111] Some embodiments of the present invention directly address these shortcomings while facilitating robot design. At design time, a robot design interface allows an automation designer to specify target UI elements and, in response, automatically identify text labels associated with each target. The design-time target labels are then included in a computer-readable specification of each workflow and sent to the RPA robot for execution. At runtime, the robot may search for targets with the specified design-time labels. If no such targets are found in the runtime instance of the target UI, some embodiments assemble a set of target candidates that partially match the design-time attributes of each target and automatically determine a runtime text label associated with each such candidate. Some embodiments then identify runtime targets according to the semantic similarity (similar meaning, as opposed to wording) between the target's design-time label and the labels of the runtime target candidates.

[0112] In assessing semantic similarity, some embodiments benefit from recent advances in natural language processing, particularly the emergence of language models based on transformer architectures (e.g., Open AI, Inc.'s GPT). A measure of semantic similarity may be calculated, for example, according to the distance separating design-time and runtime labels in the embedding space constructed by the LM. While language models are typically expensive to train and operate, some embodiments rely on the observation that UI labels are relatively compact, and therefore comparing them semantically does not require the largest or most sophisticated LM. Instead, computer experiments have revealed that successful semantic targeting can also be performed by small, portable LMs that can be run locally on each RPA client. Such small LMs are purposefully developed and trained for semantic similarity measures and then incorporated into software distribution to clients.

[0113] By closely mimicking how humans resolve unexpected changes in familiar interfaces, some embodiments of the present invention successfully prevent a large proportion of target identification failures. A particular advantage of the semantic target identification described herein is that it allows many types of textual content (e.g., actual labels, input field placeholders or default values, and alternative text / tooltips) to be used as labels for semantic similarity assessment. In the example shown in FIG. 5, not only does the wording of the label vary (e.g., from "Last Name" to "Surname"), but the type of element that effectively serves as the label also varies. While design-time label 66a is a plain text element, label 66b is a default or placeholder value for its respective input field. Therefore, the programmatic characteristics of the run-time target are significantly different from those of a design-time target that did not have a default / placeholder value. In an even more extreme example, the runtime UI 37b may display simulated passenger data as placeholder values ​​for the respective input fields (e.g., “Bart” and “Simpson” instead of the illustrated “First Name” and “Surname,” respectively). Some conventional UI automation systems may not recognize such items as labels for the respective fields, and even if they did, they may not be able to match the runtime label “Simpson” or “Surname” with the design-time label “Last Name.” In contrast, in some embodiments of the present invention, semantic targeting involves interpreting both items 66a-b as mere text labels and identifying runtime targets primarily according to the semantic similarity of their respective labels, thereby overcoming the aforementioned obstacles. In the simulated passenger name example, some embodiments of the present invention may determine, for example, that “Simpson” is more semantically similar to “Last Name” than to “First Name” or “Gender.”

[0114] The above disclosure and exemplary embodiments focus only on two types of targets: input fields and buttons (e.g., items 64e and 64f in FIG. 5 , respectively). However, the systems and methods described herein can be used to semantically identify other types of target UI elements as well. Examples include section headers and titles, entire menus, individual menu items (e.g., items displayed by expanding a drop-down menu), and virtually any hyperlinked element in a target UI. For examples of hyperlinked text (e.g., see item 64e in FIG. 4 ), the respective anchor text may be used as a label for semantic similarity assessment. If a target hyperlinked element does not naturally display a label (e.g., as in the case of icon 64b in FIG. 4 ), some embodiments may use alternative text or tooltip content displayed when a mouse hovers over the respective target element.

[0115] In addition to efficiently identifying runtime activity targets, some embodiments significantly simplify robot design. Traditional RPA typically requires specialized knowledge of RPA software and user interfaces (e.g., HTML, JavaScript, SAP, etc.) and significant design experience. To design a successful RPA workflow, automation designers must predict or know from experience how the UI is likely to change in the future and also know how to adjust their targeting strategy depending on the type and appearance of the target UI. For example, some traditional RPA systems allow designers to explicitly select a subset of target attributes to be used at runtime (e.g., attribute-value pairs selected from the DOM of the target web page). Experienced designers know which target attributes are less likely to change in the future and therefore are more robust target identifiers. In contrast, some embodiments of the present invention require designers to only indicate the target elements, thereby lowering the access threshold for developers without specialized skills.

[0116] Some embodiments further improve RPA design by including semantic targeting as an additional, complementary tool in the automation designer's toolbox. In an exemplary robot design interface, semantic targeting activities are included as separate items in a menu of available RPA activities alongside RPA activities that use traditional targeting, allowing the automation designer the freedom to choose between semantic and traditional targeting depending on the type and appearance of the target UI. Alternatively or additionally, semantic targeting can be incorporated into existing targeting methods as a fallback strategy in case traditional methods fail.

[0117] It will be apparent to those skilled in the art that many modifications can be made to the above-described embodiments without departing from the scope of the present invention, which should therefore be determined by the following claims and their legal equivalents.

Claims

1. A computer system comprising at least one hardware processor configured to: receiving an encoding of a robotic process automation (RPA) activity and an encoding of a design-time target label, the RPA activity mimicking a human interaction with a target element of a user interface (UI), the design-time target label including a text label attached to the target element in a design-time instance of the UI; responsively, identifying a runtime instance of the target element within a runtime instance of the UI exposed by the computer system, the runtime instance of the target element being identified according to a similarity between a meaning of the design-time target label and a meaning of a label attached to the runtime instance of the target element; In response to identifying the runtime instance of the target element, performing the RPA activity on the runtime instance of the target element.

2. Identifying the runtime instance of the target element includes: Selecting a candidate target element from the runtime instance of the UI; automatically determining candidate labels that include alternative text labels attached to the candidate target elements within the runtime instance of the UI; sending the design-time target label and candidate labels to a semantic evaluation module; receiving from the semantic evaluation module a similarity measure that quantifies the similarity between the meaning of the design-time target label and the meaning of the candidate label; 2. The computer system of claim 1, further comprising: determining whether the runtime instance of the target element includes the candidate target element according to the similarity measure.

3. Identifying the runtime instance of the target element further comprises: selecting a second candidate target element from the runtime instance of the UI; automatically determining second candidate labels that include further text labels attached to the second candidate target elements within the runtime instance of the UI; sending the second candidate label to the semantic evaluation module; receiving a second similarity measure from the semantic evaluation module that quantifies the similarity between the meaning of the design-time target label and the meaning of the candidate label; 3. The computer system of claim 2, further comprising: determining whether the runtime instance of the target element includes the candidate target element further according to the second similarity measure.

4. 3. The computer system of claim 2, wherein identifying the runtime instance of the target element comprises comparing the similarity measure to a predetermined threshold and determining whether the runtime instance of the target element includes the candidate target element according to a result of the comparison.

5. 3. The computer system of claim 2, wherein the semantic evaluation module is configured to determine the similarity measure using a pre-trained generative language model (GLM).

6. Determining the similarity measure comprises: determining a first embedding vector for the design-time target label and a second embedding vector for the candidate label using the GLM; The computer system of claim 5 , further comprising determining the similarity measure according to a distance between the first and second embedding vectors.

7. the target element includes an input field of the UI; the RPA activity includes filling in the input field; 2. The computer system of claim 1, wherein the label attached to the runtime instance of the target element is determined according to a placeholder value of the input field, the placeholder value being displayed by the runtime instance of the UI.

8. 2. The computer system of claim 1, wherein the target element comprises an item selected from a set consisting of a button of the UI, a menu item of the UI, and a hyperlinked element of the UI, and the RPA activity comprises clicking or tapping the item.

9. the target element comprises a hyperlinked element of the UI; 2. The computer system of claim 1, wherein the at least one hardware processor is configured to determine the label attached to the runtime instance of the target element according to alternative text or a tooltip displayed by the runtime instance of the UI upon hovering a mouse over the runtime instance of the target element.

10. 1. A computer-implemented robotic process automation (RPA) method, comprising using at least one hardware processor configured to perform the following: receiving an encoding of an RPA activity and an encoding of a design-time target label, the RPA activity mimicking a human interaction with a target element of a user interface (UI), the design-time target label including a text label attached to the target element in a design-time instance of the UI; In response, identifying a runtime instance of the target element within a runtime instance of the UI exposed by a computer system, the runtime instance of the target element being identified according to a similarity between a meaning of the design-time target label and a meaning of a label attached to the runtime instance of the target element; In response to identifying the runtime instance of the target element, performing the RPA activity on the runtime instance of the target element.

11. Identifying the runtime instance of the target element includes: Selecting a candidate target element from the runtime instance of the UI; automatically determining candidate labels that include alternative text labels attached to the candidate target elements within the runtime instance of the UI; sending the design-time target label and candidate labels to a semantic evaluation module; receiving from the semantic evaluation module a similarity measure that quantifies the similarity between the meaning of the design-time target label and the meaning of the candidate label; The method of claim 10 , comprising determining whether the runtime instance of the target element includes the candidate target element according to the similarity measure.

12. Identifying the runtime instance of the target element further comprises: selecting a second candidate target element from the runtime instance of the UI; automatically determining second candidate labels that include further text labels attached to the second candidate target elements within the runtime instance of the UI; sending the second candidate label to the semantic evaluation module; receiving a second similarity measure from the semantic evaluation module that quantifies the similarity between the meaning of the design-time target label and the meaning of the candidate label; The method of claim 11 , further comprising: determining whether the runtime instance of the target element includes the candidate target element further according to the second similarity measure.

13. 12. The method of claim 11, wherein identifying the runtime instance of the target element comprises comparing the similarity measure to a predetermined threshold and determining whether the runtime instance of the target element includes the candidate target element according to a result of the comparison.

14. The method of claim 11 , wherein the semantic evaluation module is configured to determine the similarity measure using a pre-trained generative language model (GLM).

15. Determining the similarity measure comprises: determining a first embedding vector for the design-time target label and a second embedding vector for the candidate label using the GLM; The method of claim 14 , comprising determining the similarity measure as a function of a distance between the first and second embedding vectors.

16. the target element includes an input field of the UI; the RPA activity includes filling in the input field; The method of claim 10 , wherein the label attached to the runtime instance of the target element is determined according to a placeholder value of the input field, the placeholder value being displayed by the runtime instance of the UI.

17. 11. The method of claim 10, wherein the target element comprises an item selected from a set consisting of a button in the UI, a menu item in the UI, and a hyperlinked element in the UI, and the RPA activity comprises clicking or tapping on the item.

18. the target element comprises a hyperlinked element of the UI; 11. The method of claim 10, wherein the method using the at least one hardware processor is configured to determine the label attached to the runtime instance of the target element according to alternative text or a tooltip displayed by the runtime instance of the UI upon hovering a mouse over the runtime instance of the target element.

19. A non-transitory computer-readable medium storing instructions that, when executed by at least one hardware processor of a computer system, cause the computer system to: receiving an encoding of a robotic process automation (RPA) activity and an encoding of a design-time target label, the RPA activity mimicking a human interaction with a target element of a user interface (UI), the design-time target label including a text label attached to the target element in a design-time instance of the UI; responsively, identifying a runtime instance of the target element within a runtime instance of the UI exposed by the computer system, the runtime instance of the target element being identified according to a similarity between a meaning of the design-time target label and a meaning of a label attached to the runtime instance of the target element; In response to identifying the runtime instance of the target element, performing the RPA activity on the runtime instance of the target element.

Images