System and method for optimizing robot form entry activity

Abstract

To provide a system and method for optimizing robot form entry activity.SOLUTION: A hyperautomation system 10 provides a robotic process automation RPA robot with a strategy ranking indicator that identifies a plurality of alternative strategies for entering a target form field to improve the autonomy of a form entry RPA robot, and when a strategy fails, the RPA robot reverts to the alternative strategy. The ranking indicator ranks each form entry strategy, an RPA robot selects from the available strategies according to the rank and gives priority to a higher rank strategy. The strategy ranking indicator is dynamically updated to promote successful form entry strategies and demote unsuccessful strategies for use in other instances of the RPA robot.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] BACKGROUND OF THEINVENTION The present invention relates to robotic process automation (RPA), and more particularly to facilitating the development and use of RPA software.

[0002] RPA is an emerging field of information technology that aims to increase productivity by automating repetitive office computing tasks, freeing up human operators to perform more intelligent, sophisticated and / or creative activities. Notable tasks targeted for automation include the extraction of structured data from documents and interactions with user interfaces, e.g. filling in forms.

[0003] However, automating even a seemingly simple software task like form filling can involve substantial technical challenges. Depending on the target application, the type and behavior of form fields can vary greatly, from simple click-and-type to advanced structured inputs that include multiple cascading forms, calendars, calculators, and so on. Thus, to properly design form-filling robots, RPA developers typically need to know in advance what types of form fields each robot will encounter and specifically instruct the robot how to respond to each situation. In addition, RPA developers may need in-depth knowledge of the software to be automated, such as web browsers and related technologies, spreadsheet applications, customer relationship management software (e.g., Salesforce® Platform from Salesforce, Inc.), enterprise resource planning software (e.g., SAP®), and so on.

[0004] Moreover, the software being automated is typically developed and maintained independently of the robot, so the appearance, structure, and / or functionality of the target form may change at any time. Traditional RPA robots have limited adaptability, and such changes can cause the respective automation to fail, resulting in maintenance costs and losses in productivity.

[0005] In light of the above, there is a strong demand for robust RPA robots that can efficiently fill out various forms and flexibly respond to unforeseen circumstances. At the same time, there is strong interest in making both the design and runtime aspects of RPA more user-friendly and intuitive, and in an effort to extend the reach of RPA technology to "citizen developers", i.e. users without advanced programming skills or specialized training.

[0006] (Summary of the invention) According to one aspect, a method optimizes a robotic procedure for populating a target form field displayed by a target user interface (UI). The method includes retrieving, using at least one hardware processor of a computer system, a strategy ranking indicator from a database according to an identifier of the target UI, the strategy ranking indicator identifying a plurality of alternative robotic process automation (RPA) strategies for populating the target form field and further indicating a ranking among the plurality of alternative RPA strategies. Each of the plurality of alternative RPA strategies includes a distinct sequence of RPA activities configured to populate the target form field. The method further includes, in response to selecting, using the at least one hardware processor, a first RPA strategy from the plurality of alternative RPA strategies according to the ranking, performing a first attempt to populate the selected form field according to the first RPA strategy. The method further includes, in response to determining that the first attempt is unsuccessful, selecting, using the at least one hardware processor, a second RPA strategy from the plurality of alternative RPA strategies according to the ranking and performing a second attempt to populate the selected form field according to the second RPA strategy. The method further includes updating, using the at least one hardware processor, a strategy ranking indicator in the database in response to determining that the second attempt is successful, the update being configured to change the ranking such that selection of the second RPA strategy is prioritized over selection of the first RPA strategy.

[0007] According to another aspect, a computer system includes at least one hardware processor configured to optimize a robotic procedure for populating a target form field displayed by a target UI. The at least one hardware processor is configured to retrieve a strategy ranking indicator from a database according to an identifier of the target UI, the strategy ranking indicator identifying a plurality of alternative RPA strategies for populating the target form field and further indicating a ranking among the plurality of alternative RPA strategies. Each of the plurality of alternative RPA strategies includes a distinct sequence of RPA activities configured to populate the target form field. The at least one hardware processor is further configured to perform a first attempt to populate the selected form field according to the first RPA strategy in response to selecting a first RPA strategy from the plurality of alternative RPA strategies according to the ranking. The at least one hardware processor is further configured to perform a second attempt to populate the selected form field according to the second RPA strategy in response to determining that the first attempt is unsuccessful. The at least one hardware processor is further configured to update the strategy ranking indicators in the database in response to determining that the second attempt is successful, the update being configured to change the rankings such that selection of the second RPA strategy is prioritized over selection of the first RPA strategy.

[0008] According to another aspect, a non-transitory computer-readable medium stores instructions that, when executed by at least one hardware processor of the computer system, cause the computer system to optimize a robotic procedure for populating a target form field displayed by a target UI. The instructions cause the computer system to retrieve a strategy ranking indicator from a database according to an identifier of the target UI, the strategy ranking indicator identifying a plurality of alternative RPA strategies for populating the target form field, and further indicating a ranking among the plurality of alternative RPA strategies. Each of the plurality of alternative RPA strategies includes a distinct sequence of RPA activities configured to populate the target form field. The instructions further cause the computer system to perform a first attempt to populate the selected form field according to the first RPA strategy in response to selecting a first RPA strategy from the plurality of alternative RPA strategies according to the ranking. The instructions further cause the computer system, and the at least one hardware processor, in response to determining that the first attempt is unsuccessful, to select a second RPA strategy from the plurality of alternative RPA strategies according to the ranking and perform a second attempt to populate the selected form field according to the second RPA strategy. The instructions are further configured to cause the computer system, in response to determining that the second attempt is successful, to update the strategy ranking indicators in the database, the updating changing the ranking such that selection of the second RPA strategy is prioritized over selection of the first RPA strategy. [Brief description of the drawings]

[0009] The foregoing aspects and advantages of the present invention will be better understood by reading the following detailed description and by reference to the drawings, in which:

[0010] [Figure 1]FIG. 1 illustrates an architecture diagram of a hyper-automation system according to some embodiments of the present invention.

[0011] [Diagram 2] 1 illustrates an exemplary RPA system according to some embodiments of the present invention.

[0012] [Diagram 3] 1 illustrates an exemplary deployed RPA system implemented in a client-server configuration according to some embodiments of the present invention.

[0013] [Figure 4] 1 illustrates an exemplary data exchange between an RPA conductor and an RPA robot(s) according to some embodiments of the present invention.

[0014] [Diagram 5] 1 illustrates an exemplary form targeted for automatic filling according to some embodiments of the present invention.

[0015] [Figure 6] 1 illustrates an exemplary form-filling strategy according to some embodiments of the present invention.

[0016] [Figure 7-A] 1 illustrates an exemplary strategy ranking record according to some embodiments of the present invention.

[0017] [Figure 7-B] 1 illustrates another exemplary strategy ranking record according to some embodiments of the present invention.

[0018] [Figure 8] 1 illustrates an exemplary sequence of steps performed by an RPA robot according to some embodiments of the present invention.

[0019] [Figure 9]1 illustrates an exemplary series of steps performed by an RPA server according to some embodiments of the present invention.

[0020] [Figure 10] 1 illustrates an exemplary hardware configuration for a computer system programmed to perform some of the methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Detailed Description of the Invention In the following description, it is understood that all mentioned connections between structures may be direct operative connections or indirect operative connections through intermediary structures. A set of elements includes one or more elements. Any reference to an element is understood to refer to at least one element. A plurality of elements includes at least two elements. The use of "or" is meant as a non-exclusive "or". Unless specifically required, the described method steps do not necessarily have to be performed in the particular illustrated order. A first element (e.g., data) derived from a second element includes a first element equal to a second element, a first element generated by processing a second element, and optionally other data. Making a decision or judgment according to a parameter includes making a decision or judgment according to a parameter and, optionally, making a decision or judgment according to other data. Unless otherwise specified, an indication of a quantity / data may be the quantity / data itself or may be an indication different from the quantity / data itself. A computer program is a sequence of processor instructions that perform a task. The computer programs described in some embodiments of the present invention may be independent software entities or subentities (e.g., subroutines, libraries) of other computer programs. Fillable forms displayed by the user interfaces described herein may include multiple fields, each field being an individual, separate receptacle for user input. The terms "form field" and "input field" are used interchangeably herein. The term "database" is used herein to denote an organized, searchable collection of data. Computer-readable media encompass non-transitory media, such as magnetic, optical, and semiconductor storage media (e.g., hard disks, optical disks, flash memory, DRAM), as well as communications links, such as conductive cables and fiber optic links.According to some embodiments, the present invention provides, inter alia, computer systems including hardware (e.g., one or more processors) programmed to perform the methods described herein, as well as computer-readable media encoding instructions for performing the methods described herein.

[0022] The following description illustrates embodiments of the present invention and is not necessarily limiting.

[0023] FIG. 1 is an architecture diagram illustrating a hyperautomation system 10 according to some embodiments of the present invention. As used herein, "hyperautomation" refers to an extended automation system that brings together robotic process automation (RPA) components, integrated tools, and technologies that amplify the ability to automate work. As used herein, RPA refers to automatically performing computational tasks by mimicking a series of actions of a human operator using a set of software tools to perform the respective tasks. In an exemplary scenario, a company employee uses business software to issue an invoice to a client, which may include a series of actions such as opening a Microsoft Excel spreadsheet, looking up the client's company details, copying the respective details into an invoice template, filling in the invoice fields indicating the items purchased, switching to an email application, composing an email message to the respective client, attaching the newly created invoice to the respective mail message, and clicking the "send" button. An RPA robot can effectively replicate the same series of actions, thus automating the respective invoicing task. Mimicking a human movement / action is understood herein to encompass reproducing the sequence of computing events that occur when a human operator performs the respective movement / action on a computer, and reproducing the results of a human operator performing the respective movement on a computer. For example, mimicking the action of clicking a button on a graphical user interface (GUI) may include causing the operating system to move a mouse pointer to the respective button and generate a mouse click event, or may alternatively include toggling the respective GUI button itself to a clicked state.

[0024] Examples of processes targeted by RPA include payment processing, invoicing, communication with business clients (e.g., distribution of newsletters and / or product offerings), internal communication (e.g., scheduling of memos, meetings and / or tasks), audits, and payroll, among others.

[0025] RPA may constitute the core of the hyperautomation system 10, and in certain embodiments, automation capabilities may be extended by artificial intelligence (AI) / machine learning (ML), process mining, analytics, and / or other advanced tools. As the hyperautomation system 10 learns processes, trains AI / ML models, and employs analytics, for example, more knowledge work may be automated, and computing systems within an organization, for example, both those used by individuals and those operating autonomously, may all be engaged as participants in the hyperautomation process. The hyperautomation system of some embodiments enables users and organizations to efficiently and effectively discover, understand, optimize, and extend automation.

[0026] The exemplary hyperautomation system 10 includes RPA client computing systems 12a-c, such as desktop computers, tablet computers, and smartphones, among others. Any desired client computing systems, including, but not limited to, smart watches, laptop computers, servers, Internet of Things (IoT) devices, and the like, may be used without departing from the scope of the present invention. Also, while only three client computing systems 12a-c are shown in FIG. 1, any suitable number of client computing systems may be used without departing from the scope of the present invention. For example, in some embodiments, tens, hundreds, thousands, or millions of client computing systems may be used. The client computing systems 12a-c may be actively used by a user or may run automatically without much or any user input.

[0027] Each of the illustrated client computing systems 12a-c has a respective automation module(s) 14a-c executing thereon. Exemplary automation module(s) 14a-c may include, without limitation, an RPA robot, part of an operating system, downloadable application(s) for the respective computing system, any other suitable software and / or hardware, or any combination thereof, without departing from the scope of the present invention.

[0028] In some embodiments, one or more of the module(s) 14a-c may be listeners. The listeners monitor and record data related to user interactions with the respective computing systems and / or operation of unattended computing systems and transmit the data to the hyperautomation core system 30 via a communications network 15 (e.g., a local area network-LAN, a mobile communications network, a satellite communications network, the Internet, any combination thereof, etc.). The data may include, but is not limited to, which button was clicked, where the mouse was moved, text entered into a field, one window was minimized and another opened, the application associated with the window, etc. In certain embodiments, data from such listener processes may be transmitted periodically as part of a heartbeat message or in response to the achievement of a data accumulation condition. One or more RPA servers 32 receive and store the data from the listeners in a database, such as the RPA database(s) 34 of FIG. 1.

[0029] Other exemplary automation modules 14a-c may execute logic that actually implements the automation of a selected process. In other words, at least one automation module 14a-c may include a portion of an RPA robot, as further described below. A robot may be attended (i.e., requiring human intervention) or unattended. In some embodiments, multiple modules 14a-c or computing systems may participate in the execution of the logic of an automation. Some automations may orchestrate multiple modules 14a-c, may perform various background processing, and / or may execute application programming interface (API) calls. Some robotic activities may cause modules 14a-c to wait for a selected task to be completed (possibly by another entity or automation module) before resuming the current workflow.

[0030] In some embodiments, the hyperautomation core system 30 may execute the conductor application on one or more servers, such as the RPA server(s) 32. Although FIG. 1 shows only one RPA server 32, multiple or numerous servers in close proximity to each other or in a distributed architecture may be employed without departing from the scope of the present invention. For example, one or more RPA server(s) 32 may be provided for conductor functions, AI / ML model serving, certification, governance, and / or any other suitable functions without departing from the scope of the present invention. In some embodiments, the hyperautomation core system 30 may incorporate or be part of a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, and / or the like. In certain embodiments, the hyperautomation core system 30 may host multiple software-based servers on one or more computing systems, such as the RPA server(s) 32. In some embodiments, one or more servers of the core hyperautomation system 30, such as the RPA server(s) 32, may be implemented via one or more virtual machines (VMs).

[0031] In some embodiments, one or more automation modules 14a-c may invoke one or more AI / ML models 36 deployed on or accessible by the hyper-automation core 30. The AI / ML models 36 may be trained for any suitable purpose without departing from the scope of the present invention. Two or more AI / ML models 36 may be chained (e.g., serially, in parallel, or a combination thereof) in some embodiments such that they collectively provide collaborative output(s). Exemplary AI / ML models 36 may perform or assist in computer vision (CV), optical character recognition (OCR), document processing and / or understanding, semantic learning and / or analysis, analytical prediction, process discovery, task mining, testing, automated RPA workflow generation, sequence extraction, clustering detection, speech-to-text translation, any combination thereof, and the like. However, any desired number and / or type(s) of AI / ML models 36 may be used without departing from the scope of the present invention. Using multiple AI / ML models 36 may, for example, allow a system to develop a holistic view of what is going on on a given computing system. For example, one AI / ML model can perform OCR, another can detect buttons, another can compare sequences, etc. The patterns may be determined by the AI / ML models individually or collectively by multiple AI / ML models. In certain embodiments, one or more AI / ML models 36 are deployed locally on at least one RPA client computing system 12a-c.

[0032] The hyper-automation system 10 may provide at least four main groups of functionality: (1) discovery, (2) build automation, (3) manage, and (4) engagement. The discovery functionality may discover and provide automated recommendations for various business process automation opportunities. Such functionality may be implemented by one or more servers, such as the RPA server 32. The discovery functionality, in some embodiments, may include providing an automation hub, process mining, task mining, and / or task capture.

[0033] An automation hub (e.g., UiPath Automation Hub™) can provide a mechanism to manage automation rollouts with visibility and control. Automation ideas can be crowdsourced from employees, for example, via a submission form. Calculations of feasibility and return on investment (ROI) for automating these ideas can be provided, documentation for future automations can be collected, and collaboration can be provided to expedite automation discovery-to-build.

[0034] Process mining (e.g., via UiPath Automation Cloud™ and / or UiPath AI Center™) refers to the process of collecting and analyzing data from applications (Enterprise Resource Planning (ERP) applications, Customer Relationship Management (CRM) applications, email applications, call center applications, etc.) to identify what end-to-end processes exist in an organization, how they can be effectively automated, and the impact that automation will have. This data is collected, for example, by listeners from RPA clients 12a-c and processed by RPA server(s) 32. One or more AI / ML models 36 may be employed for this purpose. This information may be exported to an automation hub to speed up implementation and avoid manual information transfer. The goal of process mining may be to increase business value by automating processes in an organization. Some examples of process mining goals include, but are not limited to, increased profits, increased customer satisfaction, regulatory and / or contractual compliance, increased employee efficiency, etc.

[0035] Task mining (e.g., via UiPath Automation Cloud™ and / or UiPath AI Center™) identifies and aggregates workflows (e.g., employee workflows) and then applies AI to uncover patterns and variations in routine tasks and score such tasks for ease of automation and potential savings (e.g., time and / or cost savings). One or more AI / ML models 36 may be employed to uncover repetitive task patterns in the data. Repetitive tasks ripe for automation may then be identified. This information may be initially provided by a listener module (e.g., automation modules 14a-c) and analyzed on the hyperautomation core 30's servers. Results from task mining may be exported to process documentation or RPA design applications such as UiPath Studio™ to create and deploy automations more quickly.

[0036] Task mining in some embodiments may include taking screenshots with user actions (e.g., mouse click locations, keyboard input, application windows and graphical elements with which the user was interacting, timestamps for the interactions, etc.), collecting statistical data (e.g., performance time, number of actions, text input, etc.), editing and annotating the screenshots, specifying the types of actions to be recorded, etc.

[0037] Task capture (through UiPath Automation Cloud™ and / or UiPath AI Center™) automatically documents attended processes as users work on them or provides a framework for unattended processes. Such documentation may include process definition documents (PDDs), skeleton workflows, capturing actions for each part of the process, recording user actions and automatically generating comprehensive workflow diagrams with details about each step, tasks desired to be automated in formats such as Microsoft Word™ documents, XAML files, etc. Buildable workflows can, in some embodiments, be exported directly into RPA design applications such as UiPath Studio™. Task capture may simplify the requirements gathering process for both subject matter experts describing the process and Center of Excellence (CoE) members delivering production grade automation.

[0038] The automation building functionality of the hyper-automation system 10 may be accomplished through a computer program, shown in FIG. 1 as an RPA design application 40. Examples may include UiPath Studio™, UiPath StudioX™, UiPath Web™, among others. Such computer programs may be used to build and test automation for a variety of applications and environments, such as web, mobile, SAP®, and virtual desktop. In some embodiments, the RPA design application 40 enables a human developer to design workflows that effectively automate a target work process. A workflow typically includes a sequence of custom automation steps, considered herein as an RPA activity. Each activity includes at least one action performed by a robot, such as clicking a button, reading a file, writing to a spreadsheet cell, etc. Activities may be nested and / or embedded. In some embodiments, the RPA design application 40 exposes a set of design interfaces and tools that give the developer control over the order of execution of the workflow and the relationships between the activities of the workflow. In some embodiments, predefined activities, drag-and-drop modeling, and a workflow recorder may facilitate automation with minimal coding. Document understanding functionality may be provided by an AI activity for data extraction and interpretation that invokes one or more AI / ML models 36. Such automation may process virtually any document type and format, including tables, web pages, forms, signatures, and handwriting.

[0039] Also, the RPA design application 40 may be used to seamlessly combine user interface (UI) automation with API automation, for example, to provide API integration with various other applications, technologies, and platforms. A repository (e.g., UiPath Object Repository™) or marketplace (e.g., UiPath Marketplace™) for pre-built RPA and AI templates and solutions may be provided to enable developers to more quickly automate a wide variety of processes. Thus, when building automation, the hyper-automation system 10 may provide a user interface, development environment, API integration, pre-built and / or custom-built AI / ML models, development templates, an integrated development environment (IDE), and advanced AI capabilities. The hyper-automation system 10 may further enable deployment, management, configuration, monitoring, debugging, and maintenance of RPA robots to execute automations designed using the application 40.

[0040] The management functions of the hyper-automation system 10 can provide deployment, orchestration, test management, AI capabilities, and optimization of automation across an organization. One example of a management function includes optimizing form-filling strategies and maintaining a centralized form-filling strategy knowledge base, as described herein. Other exemplary management functions include DevOps activities such as continuous integration and continuous deployment of automation. The management function may also act as an integration point with third-party solutions and applications for automation applications and / or RPA robots.

[0041] As an example of management functionality, a conductor application or service may facilitate, among other things, provisioning, deployment, configuration, queuing, monitoring, logging, and interconnection of RPA robots. Examples of such conductor applications / services include UiPath Orchestrator™ (which may be offered as part of the UiPath Automation Cloud™ or on-premise, in a virtual machine, or as a cloud-native single-container suite via the UiPath Automation Suite™). The application / service test suite (e.g., UiPath Test Suite™) may further provide test management to monitor the quality of the deployed automation. The test suite may facilitate test planning and execution, requirements fulfillment, and defect traceability. The test suite may include comprehensive test reports.

[0042] The analytics software (e.g., UiPath Insights™) can track, measure, and manage the performance of deployed automation. The analytics software can align automation actions with specific key performance indicators (KPIs) and strategic outcomes of the organization. The analytics software can present results in a dashboard format for better understanding by human users.

[0043] AI management capabilities may be provided by an AI center (e.g., UiPath AI Center™), which facilitates the incorporation of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options may make such capabilities accessible to non-data scientists. Deployed automation (e.g., RPA robots) may invoke AI / ML models 36 from the AI ​​center. The performance of the AI / ML models may be monitored. The models 36 may be trained and improved using human-verified data, such as data provided by a data review center as shown in FIG. 1. Human reviewers may provide labeled data (e.g., training corpus) to the hyperautomation core 30 via reviewer applications 38 performed on computers connected to the network 15. The reviewers may also use the applications 38 to verify that predictions by the AI / ML models 36 are accurate and provide corrections if not. This dynamic input may then be saved as training data for retraining the AI / ML models 36, or may be stored in a database, such as the RPA database 34. The AI ​​center may schedule and execute training jobs to train new versions of the AI / ML model 36 using the training data.

[0044] The engagement capabilities of the hyper-automation system 10 engage humans and automation as one team for seamless collaboration on a desired process. Low-code applications can be built (e.g., via UiPath Apps™) to connect to browsers and legacy software. Applications can be quickly created using a web browser, for example, through a rich library of drag-and-drop controls. Applications can be connected to one automation or multiple automations. An action center (e.g., UiPath Action Center™) can provide a mechanism to hand off a process from a robot to a human and vice versa. Humans can provide approvals or escalations, handle exceptions, etc. The RPA robot can then execute the automated functions of a given workflow.

[0045] A local assistant may be provided as a launch pad for users to launch automations (e.g., UiPath Assistant™). This functionality may be provided, for example, in a tray provided by the operating system, and may allow users to interact with RPA robots and RPA robot-powered applications on their computing systems. The interface may list the automations / workflows approved for a given user and allow the user to execute them. These may include ready-to-use automations from the automation marketplace, the automation hub's internal automation store, etc. When automations are running, they may run as local instances in parallel with other processes on the computing system so that the user can use the computing system while the automation performs its actions. In certain embodiments, the assistant is integrated with a task capture functionality so that users can document their soon-to-be-automated processes from the assistant's launch pad.

[0046] In another exemplary engagement feature, chatbots (e.g., UiPath Chatbots™), social messaging applications, and / or voice commands may enable users to execute automations, which may simplify access to information, tools, and resources needed to interact with customers or conduct other activities. For example, chatbots may respond to commands formulated in natural language by triggering robots configured to perform actions such as checking order status, posting data to a customer relationship management (CRM) platform, etc.

[0047] In some embodiments, some functions of the hyper-automation system 10 may be provided iteratively and / or recursively: processes may be discovered, automations may be built, tested, and deployed, performance may be measured, use of the automation may be facilitated to users, feedback may be obtained, AI / ML models may be trained and retrained, and the process itself may be repeated, facilitating a more robust and effective set of automations.

[0048] 2 illustrates example components and operations of an RPA system 20 according to some embodiments of the present invention. The RPA system 20 may form part of the hyper-automation system 10 of FIG. 1. The RPA system 20 includes an RPA design application 40 that enables developers to build automation by designing and implementing RPA workflows. For example, the application 40 may expose a set of user interfaces and tools that give developers control over the order of execution of the workflow and the relationships between the activities of the workflow. One commercial example of an RPA design application 40 is UiPath Studio™.

[0049] Some types of RPA workflows may include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suitable for linear processes, allowing flow from one activity to another without cluttering the workflow. Flowcharts may be particularly suitable for more complex business logic, allowing decision integration and connection of activities in more diverse ways via multiple branching logic operators. FSMs may be particularly suitable for large workflows. FSMs may use a finite number of states triggered by conditions (i.e., transitions) or activities during their execution. Global exception handlers may be particularly suitable for determining the behavior of a workflow when an execution error is encountered or for debugging a process.

[0050] Once a workflow is developed, it may be encoded in a computer-readable form, such as an RPA script or RPA package 50 (FIG. 2). The RPA script includes a specification of the respective workflow, which specification is understandable (or interpretable) by the RPA robot 22. The RPA script may be formulated according to any data specification format known in the art, such as, for example, Extensible Markup Language (XML), JavaScript Object Notation (JSON), or a version of a programming language, such as C#, Visual Basic, Java, etc. Alternatively, the RPA script may be formulated in an RPA-specific version of bytecode, or even as a sequence of instructions formulated in a natural language, such as English, Spanish, Japanese, etc. In some embodiments, one or more related RPA scripts are bundled together with other files and / or metadata to form the RPA package 50. For example, in addition to the RPA script, the RPA package 50 may include a specification of resources required to fulfill the respective workflow(s). Exemplary resources include, among others, file (e.g., path, URL) locations, file names, and sets of credentials to access a particular machine, computer program, or service. In what is commonly known in the art as a "build," an RPA script may be pre-compiled into a set of executable files, including a main executable file and associated libraries, resource specifications, and metadata, to form an RPA package 50. Package 50 may use any data specification format known in the art. For example, some embodiments of package 50 include a NuGet package of .NET assembly files.

[0051] Those skilled in the art will appreciate that the RPA design application 40 may include multiple components / modules, which may run on separate physical machines. In one such example illustrating a cloud computing embodiment of the present invention, the RPA design application 40 may run in a client-server configuration, where one component of the application 40 may expose an automation design interface on a developer's computer, and another component of the application 40 running on a remote server may assemble the workflow and form / output the RPA package 50. For example, a developer may access the automation design interface via a web browser running on the developer's computer, while software that processes user input received on the developer's computer actually runs on the server.

[0052] In some embodiments, the workflow designed in the RPA design application 40 is deployed to the RPA conductor 24, for example in the form of an RPA package as described above. In accordance with the above, in some embodiments, the conductor 24 may be part of the hyper-automation core system 30 shown in Figure 1. One commercial example of the conductor 24 is UiPath Orchestrator™.

[0053] The conductor 24 orchestrates one or more RPA robots 22 that execute respective workflows. Such "orchestration" may include the creation, monitoring, and deployment of computing resources for the robots 22 in environments such as a cloud computing system and / or a local computer. Orchestration may further include, among others, the deployment, configuration, queuing, monitoring, logging, and / or providing interconnectivity of the robots 22. Provisioning may include creating and maintaining connections between the robots 22 and the conductor 24. Deployment may include ensuring accurate delivery of software (e.g., RPA packages 50, individual workflow specifications) to the robots 22 for execution. Configuration may include maintaining and distributing the robot environment and workflow configurations. Queuing may include providing management of job queues and queue items. Monitoring may include keeping track of robot states and maintaining user permissions. Logging may include storing and indexing logs in a database and / or another storage mechanism (e.g., SQL, ElasticSearch®, Redis®). Conductor 24 may further act as a centralized point of communications for third party solutions and / or applications.

[0054] The RPA robots 22 are execution agents (e.g., computer programs) that implement automation workflows for a variety of systems and applications, including, but not limited to, mainframes, web applications, virtual machines, enterprise applications (e.g., those created by SAP®, SalesForce®, Oracle®, etc.), desktop and laptop applications, mobile device applications, wearable computer applications, etc. One commercial example of a robot 22 is UiPath Robots™.

[0055] In some embodiments, to mimic a human user's interaction with the target application's user interface, the RPA robot 22 interfaces with a set of RPA drivers 25 running on the respective host. Such drivers generally represent software modules that perform low-level operations such as moving a cursor on a screen, registering and / or executing mouse, keyboard, and / or touchscreen events, detecting the current pose / orientation of a handheld device, detecting current accelerometer readings, taking pictures with a smartphone camera, etc. Some of such drivers form part of the local operating system. Other RPA drivers 25 may implement various application-specific aspects of a user's interaction with complex target applications such as SAP®, Citrix® virtualization software, Microsoft Excel®, etc. One particular example includes a browser driver, which may be embodied as a set of browser-compatible scripts (e.g., JavaScript®). When injected into a web page currently displayed within a browser, such a browser driver may identify various elements (e.g., buttons, menus, form fields, etc.) of the respective web page and invoke specific functions of each element (e.g., filling in form fields, selecting menu items, toggling checkboxes, etc.). Other exemplary RPA drivers 25 include Microsoft® WinAppDriver, Apple, Inc.'s XCTest driver, and Google, Inc.'s UI Automator driver.

[0056] The types of robots can include, among others, attended robots 122, unattended robots 222, development robots (similar to unattended robots but used for development and testing purposes), and non-production robots (similar to attended robots but used for development and testing purposes). Some activities of attended robots 122 are triggered by user events and / or commands and operate alongside a human operator on the same computing system. In some embodiments, attended robots can be started only from the robot tray or from a command prompt and therefore cannot be fully controlled from the conductor 24 and cannot run under a locked screen, for example. Unattended robots can run unattended in a remote virtual environment and can be responsible for providing remote execution, monitoring, scheduling, and work queue support.

[0057] In some embodiments implemented in a Windows environment, the robot 22 installs as a Microsoft Windows Service Control Manager (SCM) managed service by default. As a result, such a robot can open an interactive Windows session under the local system account and have the processor privileges of a Windows service. For example, a console application can be launched by an SCM managed robot. In some embodiments, the robot 22 can be installed with user level processor privileges (user mode, ring 3). Such a robot has the same rights as the user under which the respective robot is installed. For example, such a robot can launch any application that each user can run. In a computing system that supports multiple interactive sessions running simultaneously (such as Windows Server 2012), multiple robots can run simultaneously, each running in a separate Windows session using a different user name.

[0058] In some embodiments, the Robot 22 is split into multiple components, each specialized for a particular automation task. The Robot components in some embodiments include, but are not limited to, an SCM Management Robot Service, a User Mode Robot Service, an Executor, an Agent, and a Command Line. Depending on the platform details, the SCM Management and / or the User Mode Robot Service manage and monitor the Windows session and act as a proxy between the Conductor 24 and the host machine (i.e., the computing system on which the Robot 22 executes). These services are entrusted with managing the credentials of the Robot 22. The Command Line is a client of the service(s) and is a console application that can be used to launch jobs and display or otherwise process their output.

[0059] An exemplary set of robot executors 26 and RPA agents 28 is shown in FIG. 3. The robot executors 26 are capable of executing a given job under a Windows session. The executors 26 are configured to receive RPA packages 50 that specify a workflow (e.g., a sequence of robotic activities) and execute the respective packages, which essentially result in executing the respective sequences of RPA activities. In some embodiments, the packages 50 include pre-compiled executable code. In other exemplary embodiments, the robot executor(s) 26 include an interpreter (e.g., a just-in-time interpreter or compiler) configured to convert a received RPA script that includes a workflow specification (e.g., bytecode, XML, JSON, etc.) into runtime code that includes processor instructions for executing the respective workflow. Thus, execution of the RPA package 50 includes the executor(s) 26 translating the workflow specification included in the package 50 and instructing the processor of the respective host machine to load the resulting runtime code into memory and launch the runtime code during execution.

[0060] The RPA agent 28 may manage the operation of the robot executor(s) 26. For example, the RPA agent 28 may select tasks / scripts for execution by the robot executor(s) 26 according to input from a human operator and / or according to a schedule. The agent 28 may start and stop jobs and configure various operating parameters of the executor(s) 22. If the robot 22 includes multiple executors 26, the agent 28 may coordinate their activities and / or inter-process communications. The RPA agent 28 may further manage communications between the RPA robot 22 and the conductor 24, and / or other entities.

[0061] The exemplary RPA system 20 of FIG. 2 forms part of the hyperautomation system 10 (see FIG. 1). As such, the robots 22 can interact with various components and use various aspects of the hyperautomation core system 30, generally shown in FIG. 2 as hyperautomation services 23. For example, a developer can use an RPA design application 40 to build and test an RPA robot 22 that utilizes an AI / ML model 36. Such an RPA robot 22 can send inputs for execution of the AI / ML model(s) and receive outputs therefrom via the hyperautomation core system 30. The robots 22 can be listeners, as described above. These listeners can provide information to the core hyperautomation system 30 about what users are doing when they use their computing system. This information can then be used by the hyperautomation system 30 for process mining, task mining, task capture, and the like. In another exemplary embodiment, the hyperautomation services 23 can expose data labeling functionality to users of the computing system hosting the robots 22, or to another computing system to which the robots 22 provide information. For example, if the robot 22 invokes a computer vision AI / ML model 36 but the respective model does not correctly identify an on-screen button, the user may explicitly provide the correct identification. Such information may be passed to the hyper-automation core system 30 and then used to retrain the respective AI / ML model.

[0062] In some embodiments, selected components of the hyper-automation system 10 and / or the RPA system 20 may be implemented in a client-server configuration. In one such configuration shown in FIG. 3, the RPA robot 20 including the executor(s) 26 and the RPA agent 28 may be implemented on the client side, e.g., on one of the RPA client computers 12a-c of FIG. 1. The functionality of the conductor 24 and / or other services of the hyper-automation core system 30 may then be implemented on the server side, e.g., on a remote RPA server 32 (FIG. 1). It should be noted that the client side, the server side, or both, may include any desired number of computing systems (e.g., physical or virtual machines) without departing from the scope of the present invention. The illustrated RPA system may be cloud-based, on-premise, or a combination thereof, providing an enterprise-level, user-level, or device-level automation solution for automating various work processes.

[0063] The robot 22 may execute several jobs / workflows simultaneously. The RPA agent 28 (e.g., a Windows® service) may act as a single client-side point of contact for multiple executors 26. The agent 28 may further manage the communication between the robot 22 and the conductor 24. In some embodiments, the communication is initiated by the RPA agent 28, which may open a WebSocket channel to the conductor 24. The agent 28 may then use the channel to send notifications about the status of each executor 26 to the conductor 24, for example as a heartbeat signal. The conductor 24 may then use the channel to send approvals, job requests, as well as other data, such as RPA packages 50, to the robot 22.

[0064] In one embodiment shown in Fig. 3, the conductor 24 includes a web interface 42 and a set of service modules including a set of application programming interface (API) endpoints 43 and service API / business logic 44. A user can interact with the conductor 24 via the web interface 42 (e.g., by opening a dedicated web page on the browser 16) and instruct the conductor 24 to perform actions such as scheduling and / or starting jobs on the robots 22, creating robot groups / pools, assigning workflows to robots, adding / removing data to / from queues, analyzing logs per robot or workflow, etc. The interface 42 can be implemented using Hypertext Markup Language (HTML), JavaScript (JS), or any other data format known in the art.

[0065] The conductor 24 may perform actions requested by a user by selectively invoking the service API / business logic 44 via the endpoints 43. Additionally, in some embodiments, the API endpoints 43 are used to communicate between the RPA robots 22 and the conductor 24 for tasks such as configuration, logging, deployment, monitoring, and queuing, among others. The API endpoints 43 may be set up using any data format and / or communication protocol known in the art. For example, the API endpoints 43 may conform to Representational State Transfer (REST) ​​and / or Open Data Protocol (OData).

[0066] The configuration endpoint may be used to define and configure application users, permissions, robots, assets, releases, etc. The logging endpoint may be used to log different information such as errors, explicit messages sent by the robot 22, and other environment specific information. The robot 22 may use the deployment endpoint to query the version of the RPA package 50 to be executed. The queuing endpoint may be responsible for managing queues and queue items, such as adding data to the queue, getting transactions from the queue, setting the state of transactions, etc. The monitoring endpoint may monitor the performance of the web interface 42 and / or the RPA agent 28.

[0067] The service API 44 includes computer programs that are accessed / invoked through configuration of appropriate API access paths based, for example, on whether the conductor 24 and the overall hyperautomation system have an on-premise or cloud-based deployment type. The exemplary API 44 provides custom methods for querying statistics about various entities registered with the conductor 24. Each logical resource may be an OData entity in some embodiments. In such entities, components such as robots, processes, queues, etc. may have properties, relationships, and behaviors. The API 44 may be consumed by the web application 42 and / or the RPA agent 28 by obtaining appropriate API access information from the conductor 24 or by registering an external application using an OAuth flow mechanism.

[0068] In some embodiments, the persistence layer of the server-side operations implements a database service. The database server 45 may be configured to selectively store and / or retrieve data to / from the RPA database 34. The database server 45 and the database 34 may employ any data storage protocol and format known in the art, such as Structured Query Language (SQL), ElasticSearch®, and Redis®, among others. Exemplary data stored / retrieved by the server 45 may include configuration parameters of the robots 22 and robot pools, as well as data characterizing the workflows performed by the robots 22, data characterizing users, roles, schedules, queues, and the like. In some embodiments, such information is managed via a web interface 42. Another exemplary category of data stored and / or retrieved by the database server 45 includes data characterizing the current state of each performing robot, as well as messages logged by the robots during execution. Such data may be transmitted by the robots 22 via API endpoints 43 and centrally managed by the conductor 24, for example, via API logic 44. Another exemplary category of data stored and / or retrieved by database server 45 includes form entry strategy packages and / or strategy ranking records, as described in more detail below.

[0069] The server 45 and database 34 also store / manage process mining, task mining, and / or task capture related data received, for example, from listener modules performing on the client side as described above. In one such example, the listeners can record user actions performed on their local host (e.g., clicks, characters typed, locations, applications, active elements, times, etc.) and then convert these into an appropriate format to be provided to and stored in the database 34.

[0070] In some embodiments, a dedicated AI / ML server 46 facilitates the incorporation of AI / ML models 36 into automation. Pre-built AI / ML models, model templates, and various deployment options make such capabilities accessible to operators without advanced AI / ML knowledge or expertise. Deployed robots 22 can invoke AI / ML models 36 by interfacing with the AI / ML server 46. Performance of deployed AI / ML models 36 can be monitored, and each model can be retrained and improved using human-validated data. The AI / ML server 46 can schedule and fulfill training jobs and manage training corpora. The AI / ML server 46 can further manage data related to AI / ML models 36, algorithms and software packages for various AI / ML functions, including, but not limited to, document understanding techniques and frameworks, intent analysis, natural language processing (NLP), speech analysis and synthesis, computer vision, and the like.

[0071] 4 illustrates an exemplary form-filling RPA system according to some embodiments of the present invention. In an exemplary embodiment illustrating an airport check-in scenario, the RPA robot 22 can be configured to receive a scan of a passenger's passport and automatically extract structured data therefrom, such as name, nationality, and passport number. The RPA robot automatically enters the passenger data into a web-based airport check-in form. To perform the form-filling activity, the robot 22 can identify each field of the respective form and enter the appropriate information.

[0072] The RPA robot 22 (FIG. 4) may be performed on an RPA client 12, e.g., a desktop computer located at the respective airport terminal, or a remote server computer operated by the respective airline. To perform form-filling activities, the robot 22 interacts with a target user interface (UI) 37 exposed by an RPA target application, such as a web browser in this example. An exemplary target UI 37 described herein is shown in FIG. 5 and includes a form having multiple input fields 58. Those skilled in the art will appreciate that the appearance and content of the illustrated form are not limiting and that similar forms may be displayed by other RPA target applications, such as CRM and ERP platform clients, spreadsheet applications, and document viewer applications (e.g., Adobe® Reader®).

[0073] Depending on the target application and target UI 37, the type and behavior of input fields 58 can vary considerably. The HTML specification alone defines over 20 different input types, including text, radio, checkbox, date, and password, among others. Such field types have type-specific behavior. For example, clicking / tapping on a date field, such as the exemplary field 58 of FIG. 5, displays a calendar widget and expects user input to select a date via the calendar. Other field types allow the user to select input data from a drop-down list of options. Web designers can further turn on various optional features, such as auto-completion and / or adding input validation procedures that restrict the format and / or content of the input data (e.g., ensuring that numbers are within a predefined range, dates are written in a predefined format, the respective text does not contain certain special characters, etc.). The behavior of input fields exposed by some enterprise application UIs can be even more complex. For example, selecting an option from a drop-down menu may display a popup window, further expose another set of input fields, etc. Therefore, an RPA developer looking to design a form-filling robot may need precise prior knowledge of the types and behavior of the input fields the robot will encounter at runtime in order to instruct the robot on the appropriate way to fill in each field.

[0074] Some embodiments of the present invention rely on the observation that, given the wide variety of input types and associated behavioral variations, an RPA robot that relies on only one predefined strategy for filling in a given form field may fail for many reasons. For example, one of the details of the target form may suddenly change, causing the respective input procedure to no longer work. In one such example, a developer of the target UI 37 may change a simple text field into a drop-down field. An RPA robot configured to simply fill in each field may be unexpectedly presented with a set of options to choose from and may not be able to respond appropriately.

[0075] To address such shortcomings, in some embodiments, the RPA robot 22 is provided with multiple alternative strategies for filling in each field, such that when one strategy fails, the robot 22 can try at least another strategy. Each of the multiple form filling strategies may be specific to the type of each input field, i.e., there may be some strategies associated with drop-down fields and other strategies associated with date fields, etc. In some embodiments, the strategies may be ranked to indicate, for example, the order in which they should be tried and / or the likelihood of success in filling in each form / field.

[0076] In some embodiments, form entry strategies are developed independently of other RPA workflows, e.g., in the form of a library that may be included in the RPA package 50 at design time or fetched on demand from the strategy repository 53 (FIG. 4). FIG. 6 illustrates an exemplary form entry strategy 70 according to some embodiments of the invention. The strategy 70 includes an RPA workflow for populating at least one input field of a form and may have input and / or output variables. For example, the strategy 70 may be configured to receive a set of input data 75 and output a success indicator 77 indicating whether the respective strategy was successful in introducing the data 75 into the respective input field.

[0077] In some embodiments, the strategy 70 includes a series of individual RPA activities 72 that collectively perform individual form-filling tasks. The activities 72 may include any RPA activities known in the art. In one example of a strategy for filling drop-down fields, the activities 72 may include, among other things, determining whether each input field displays a drop-down symbol (e.g., arrow, triangle); if yes, automatically clicking each symbol to display each drop-down list; parsing the list to identify items that match the input data 75; and selecting each item from the list. Another example strategy 70 for filling drop-down input fields may include an RPA activity 72 that attempts to fill each field and determines whether each field responds by suggesting an input value; if yes, determining whether the suggested value matches the desired input value; if yes, clicking each suggested value. Some activities 72 may be more complex and / or may include invoking external resources or services. In one such example, the target UI 37 may be configured to receive inputs formulated in the English and / or Latin alphabets. However, some of the input data 75 may be provided in different languages ​​and / or scripts (Arabic, Chinese, etc.). In such situations, some activities 72 may include determining whether the incoming data is in the expected format; if no, sending the respective data to a remote server (e.g., HyperAutomation service 23 in FIG. 2) for image analysis, conversion and / or further manipulation, and receiving the correctly formatted input data in response.

[0078] In some embodiments, an RPA developer can use an instance of the RPA design application 40 to develop a form-entry strategy 70, similar to any other RPA workflow. The strategy 70 can then be exported in computer-readable form as a strategy package 51. The package 51 can use the same data format as a regular RPA package (see above discussion regarding FIG. 2). For example, the package 51 can include an RPA script that includes computer-readable specifications of RPA activities 72, dependencies, and / or other resources required for the RPA robot 20 to execute the strategy 70.

[0079] Strategy packages 51 may be stored in a strategy repository 53, for example, in the form of an indexed data structure that allows selective insertion and retrieval. The repository 53 may be managed by a database server and form part of the RPA database 34 (FIG. 1). Having packages 51 retrievable from a central repository allows form entry strategies 70 to be developed and maintained independently of other RPA workflows, and further allows such strategies to be shared and reused across multiple automations. In some embodiments, form entry strategies 70 may be further included in the RPA design application 40 as building blocks for RPA automation. In one such example, the application 40 may give the developer access to various form entry strategies 70 through a dedicated activity menu that lists multiple available ways to fill in a selected form field.

[0080] To further facilitate sharing and reuse of ready-made strategies 70, some embodiments maintain an association between the target UI 37 and the strategies 70 available for filling out forms displayed by the respective target UI. Such embodiments rely on the observation that some UIs 37, e.g., interfaces exposed by a complex ERP platform, may be repeatedly accessed by the same robot 22 and / or targeted simultaneously by multiple automations and / or robots 22. In one such example, each of multiple workflows may be configured to extract data from separate sources (e.g., separate types of documents, separate software applications, etc.). However, all respective workflows may use the same online form to transfer their respective data to the ERP system. Maintaining an association between each form and a set of strategies 70 for filling out the respective form results in a shared knowledge base of what strategies currently work for each form / target UI. If a particular strategy suddenly fails, each robot may search for a valid strategy among the available options and update the knowledge base accordingly to associate the respective form / field with the newly identified successful strategy. Other robots and workflows targeting the respective form / field can then directly use the new success strategies, improving automation performance and user experience.

[0081] Some embodiments store a knowledge base associating target UIs 37 with form entry strategies 70 in the form of a strategy ranking register 52 and / or a local strategy ranking cache 152 (FIG. 4). The register 52 and cache 152 may include a searchable collection of records having any data format known in the art. The register 52 may form part of the RPA database 34 (FIG. 1) and may be centrally managed by a database server such that its contents are accessible to RPA robots 22 executing on various RPA hosts.

[0082] In the exemplary embodiment shown in FIG. 7-A, the register 52 and / or cache 152 store a plurality of strategy ranking records 60, which are shown as individual rows in a relational database table. Each row / record 60 includes a set of target-specific data, such as a form ID 62, a field ID 59, and a field type indicator 64, which collectively identify a particular input field targeted for auto-filling. The form ID 62 may include, for example, a uniform record identifier (URI) of the target UI 37 (see, for example, FIG. 5, where the form is identified by a uniform record locator / URL). If the target UI 37 is not web-based, the form ID 62 may include other data that may identify the UI 37 and / or the particular form displayed by the UI 37. For example, some embodiments may use an identifier (e.g., process ID) of the computer program displaying the UI 37, an attribute (e.g., name) of the window currently displaying the UI 37, the name of the document containing each form, etc. In some embodiments, form ID 62 may further include a digest of UI 37 that includes a list of field identifiers (field labels) for all fields displayed by UI 37. Such data may increase the likelihood of unambiguous identification of each UI / form in cases where URLs or other identifiers may change.

[0083] The field ID 59 may include any data that identifies the target input field among other elements of the target UI 37. For example, the field ID 59 may include a text label that is displayed next to the respective input field, as shown in FIG. 5 and FIG. 7-A. Another exemplary field ID 59 may include a set of attribute-value pairs extracted from a UI tree representation of the UI 37, which is a UI tree representation managed by the local operating system. Such a set of attribute-value pairs may function as an identifier by indicating a unique location within the respective UI tree. Other field ID data may include an image of each input field, a label or title that is displayed next to each field, and the like.

[0084] The field type indicator 64 indicates the type of each input field, for example, by a number or label that distinguishes each type of input from other types. In the example shown in Figures 5 and 7-A, the label "43" may indicate a date input via an attached calendar widget, while the label "2" may indicate a simple input into a text field. The field type indicator 64 may further distinguish between web-based form fields and inputs displayed by other computer programs, such as spreadsheet applications and various legacy software.

[0085] Target identifiers such as items 59, 62, and 64 collectively identify a target input field (in the present example, a date input labeled "Expiration date" and displayed by a web page located at the URL "gammaair.com / boarding-check.html"). In some embodiments, strategy ranking record 60 further includes an identifier of at least one form-filling strategy 70 that can be used to fill in the field characterized by the respective target identifier. In a preferred embodiment, record 60 includes a strategy ranking indicator 54 that identifies multiple alternative form-filling strategies 70 that are applicable to each input, and further indicates a ranking of each strategy. In some embodiments, the strategy ranking indicates the order in which each strategy should be attempted to successfully fill in each field, for example, with higher ranked strategies being attempted before lower ranked strategies.

[0086] Exemplary rankings are determined according to measures of performance of each strategy 70, such as the likelihood that each strategy will successfully populate each form field, the computational cost associated with each strategy, the complexity of each strategy (e.g., count of individual steps), the amount of change to the state of each field caused by the execution of each strategy, etc. Other exemplary strategy selection and ranking criteria may include device type, where some strategies are promoted on some devices (e.g., smartphones) and demoted on other devices (e.g., desktop computers). The content of the ranking indicators 54 may further vary depending on, among other things, the current location (e.g., geographic location, cloud platform, network provider, etc.) of the RPA client 12 executing each automation, the identity of the customer executing each automation, and the service contract associated with each automation.

[0087] In the exemplary embodiment shown in FIG. 7-A, the ranking indicators 54 each include a strategy identifier h that is uniquely associated with a respective form-filling strategy 70. k An exemplary strategy identifier h k can include an integrity hash, for example a hash calculated on a strategy package 51 that includes specifications of the respective form-filling strategy 70 (see, for example, FIG. 6). Another exemplary strategy identifier h k 7-A includes an index (e.g., a hash index) that points to database records associated with each strategy 70. In FIG. 7-A, the strategy repository 53 includes an index h k with a strategy location indicator 66 (eg, a URL) that indicates the address for obtaining a strategy package 51 that specifies the respective form-filling strategy.

[0088] Those skilled in the art will appreciate that there may be many ways to associate target input fields 58 with form input strategies 70 and rankings, and that such implementation details are not limiting. In alternative embodiments, records 60 may be defined by a set of attribute-value pairs formulated in a version of Extensible Markup Language (XML), Javascript® object notation (JSON), or any other declarative data format. In the example shown in FIG. 7-B, field ID 59 includes a value of the “key” attribute, field type indicator 64 includes a value of the “type” argument, and form ID 62 and ranking indicator 54 include the current values ​​of the “application identifier” and “trace” attributes, respectively. The illustrated form ID 62 further characterizes target UI 37 as a web-based form.

[0089] The embodiment shown in FIG. 7-AB further provides an exemplary ranking of strategies, where indicator 54 includes an ordered sequence of strategy IDs, with the position in the ordered sequence indicating the current rank. 12 A strategy identified by h1 may have a higher rank than a strategy identified by h2, which in turn may have a higher rank than a strategy identified by h8, etc. In FIG. 7-B, "Strategy 8" may be ranked higher than "Strategy 3," etc.

[0090] Those skilled in the art will appreciate that there may be many ways to attach rankings to a collection of items, and that such implementation details are not meant to be limiting. In one exemplary alternative embodiment, ranking indicator 54 may include a vector of numbers rk of size n, where n is the count of available strategies 70. Each element r k may correspond to distinct form-filling strategies 70, while the value may indicate the current rank of the respective strategy.

[0091] FIG. 8 illustrates an exemplary sequence of steps performed by the RPA robot 22 in some embodiments of the present invention. Step 802 may receive a specification of an RPA workflow including activities for populating selected fields of a target form. The workflow specification may be delivered to the robot 22 in the form of an RPA package 50 sent by an instance of the RPA conductor 24 executing on a remote RPA server 32 (see, for example, FIG. 4). In a further step 804, the robot 22 may identify a user interface (e.g., the target UI 37 of FIGS. 4-5) targeted by the respective automation. The target UI may be explicitly specified by the package 50. In an alternative embodiment, the robot 22 may prompt the user to select or otherwise indicate the target UI 37, for example by directing the robot's attention to a particular URL or UI currently displayed on the screen.

[0092] Step 804 may further include configuring at least one RPA driver 25 to interact with the UI 37. In the case of a web-based user interface, some embodiments inject a set of scripts into the target UI 37, each of which, when executed by a browser application displaying the UI 37, collectively implements the driver's functionality. Each driver can effectively perform form-entry activities specified in the current RPA workflow and communicate with the robot 22 in the process. In such an embodiment, configuring the drivers 25 may further include setting up a communication channel between the robot 22 and the respective driver. In the case of a web-based interface, an exemplary communication channel may be established using a web extension mechanism. Such implementation details are beyond the scope of this description.

[0093] The received RPA workflow specification may explicitly instruct the robot 22 to fill in a set of input fields of the target UI 37. In alternative embodiments, the autonomous robot may attempt to fill in all fields displayed by the respective UI, or may independently determine which fields to fill according to predefined criteria or algorithms. For each such target input 58 (i.e., while step 806 returns yes), a sequence of steps 808-810 may determine a target ID 61 including a set of characteristics of the respective input field, such as a form ID 62, an input type 64 and / or a field ID 59, as described above in connection with FIG. 7-AB. Depending on the implementation, such target ID data may be specified in the RPA package 50 or may be determined by the RPA driver on the fly by analyzing the target UI 37. When the UI 37 is displayed by a browser application, the RPA driver may derive the target ID data (e.g., the type of input field, the label displayed next to the input field, etc.) by, for example, parsing the Document Object Model (DOM) of the UI 37. Another example embodiment may capture images of the target UI 37, send each image to an AI image processing module (eg, AI model 36 of FIG. 1), and send target ID data 61 back in response.

[0094] Next, in a sequence of steps 812-814, the RPA robot 22 can look-up a strategy ranking record with the respective target ID data. This disclosure describes only a remote look-up to the centralized strategy ranking register 52 (FIG. 4). However, one skilled in the art knows that a similar look-up may be performed to a local ranking cache 152, if available. In some embodiments, the remote look-up may be performed by searching the strategy ranking records in the local cache 152 and only if a record related to the respective target ID cannot be found locally and / or if the local cache has expired. Step 812 may include formulating a query to the RPA server 32, including the target ID data 61 identifying at least one input field to be displayed by the target UI 37. The format and content of the respective query may depend on the implementation.

[0095] FIG. 9 illustrates an exemplary sequence of steps performed by the RPA server 32 in some embodiments of the present invention. The illustrated steps may be performed by a computer program such as the RPA conductor 24 and / or a software component of the database server 45 (e.g., FIG. 3). The sequence of steps 902-904 may listen for input queries from the RPA client 12. When a query is received, step 906 performs a lookup into the strategy ranking register 52 according to the target ID data included in the respective query. Since the register 52 effectively associates the target ID data with the strategy ranking indicators (e.g., see above with respect to FIG. 7-AB), the lookup may generate ranking indicators 54 corresponding to the input fields having respective characteristics.

[0096] If no record 60 with the respective characteristic currently exists in the register 52 (step 908 can return no, for example, if the target UI 37 is new or has not yet been visited by the robot 22), step 910 can create a new strategy ranking record for the input field characterized by the target ID data 61. In some embodiments, the ranking indicators 54 of the new strategy ranking record can include a set of predefined default values. In other words, the record 60 can be initialized to a set of default form input strategies 70 and rankings. The default form input strategies and rankings can be field type specific, i.e., different for each type of input field. The content of the default ranking indicators can be static (e.g., defined by the RPA developer) or dynamically determined according to the current content of the register 52. For example, some embodiments can identify families of records in the register 52 that share some specific data with the current target input and determine the default ranking indicators according to the family of the respective records. In one such example, the default ranking indicators may include copies of other ranking indicators that correspond to other inputs of the same type (e.g., same type indicator 64) and / or location (e.g., same URL, same Internet domain, etc.) as those included in the respective query.

[0097] In other exemplary embodiments, the default strategies may be selected according to how they change the state of the respective input fields. For example, only strategies that cause a reversible change in the state of the respective inputs may be selected as default strategies, which allows the respective inputs to return to their initial state if the respective strategy fails. On the other hand, the ranking of the defaults may be determined according to the size of the state change generated by each respective strategy, with some embodiments prioritizing strategies that generate relatively smaller changes than others. See step 822 (FIG. 8) for details.

[0098] In step 912, the server 32 may send the ranking indicator 54 to the requesting RPA client. Depending on the circumstances as described above, the indicator 54 may include the result of the database lookup (output of step 906) or a default ranking indicator (output of step 908).

[0099] In step 814 (FIG. 8), the RPA robot 22 may receive the ranking indicators 54 and identify a set of strategies 70 according to the indicators 54. The robot 22 may then cycle through each strategy until it finds a successful strategy. If all strategies indicated by the ranking indicators 54 fail, some embodiments report a failure to enter the respective input field (step 830) and proceed to the next target input in the UI 37.

[0100] Step 818 selects a strategy according to ranking indicator 54. Some embodiments select strategies in order of rank: highest ranked first, lower ranked back. Step 820 may obtain a specification of the selected strategy. Some embodiments include a substantial collection of form filling specifications in the RPA package 50. Alternative embodiments may obtain a strategy package 51 on request from the RPA server 32 (e.g., the RPA conductor application / service 24), the package 51 including the specification of the selected form filling strategy.

[0101] A further step 822 can effectively execute the strategies 70, i.e. execute a series of RPA activities that constitute the respective strategies. Such RPA activities can include, for example, clicking / tapping on respective input fields, typing in respective fields, detecting the expansion of drop-down menus, and parsing menu items to identify at least one item that matches the input data 75. To execute such activities, the robot 22 can cooperate with the RPA driver 25 configured in step 804 (see details above).

[0102] In preparation for performing step 822, some embodiments may return the target input 58 to its initial state, i.e., the state of each input when the respective input was identified and selected for automatic entry. In such embodiments, steps 808-810 described above may further include determining and recording the initial state of each input. The state may be defined by a set of initial values ​​of field-specific attributes, such values ​​indicating, for example, default content of each input, whether a widget associated with each input is currently active / visible, whether a drop-down list is currently displayed, etc. Then, in preparation for performing the new strategy (step 822), the RPA robot 22, in cooperation with the RPA driver 25, may reset the field-specific attributes of the target input 58 to the initial values ​​recorded in steps 808-810.

[0103] In an alternative exemplary embodiment, the specification of the form entry strategies 70 (e.g., strategy package 51) may include an indicator of a set of intermediate stages / reference points during performance of each strategy, and an indicator of a reference state of the target input 58 at each intermediate stage. An exemplary set of intermediate stages includes stages between successive RPA activities 72 (FIG. 6). An exemplary reference state may include a desired / expected state of the input 58 at each intermediate stage within the strategy. In such an embodiment, prior to performing each strategy, the robot 22 may determine whether the current state of the target input 58 matches any of the reference states associated with the respective strategy. If yes, instead of resetting the input 58 to its initial state as described above, some embodiments may begin performance of the strategy 70 from the intermediate stage that matches the current state of the target input 58. In one such example, strategy A includes simply typing input data 75 into the target field 58. However, strategy A may fail to properly populate the respective field if the field 58 unexpectedly opens a drop-down list of options in response to typing. The robot 22 may then select an alternative strategy B which attempts to enter into the target field and then attempts to match the input data 75 to one of the displayed drop-down options. However, in preparation for performing strategy B, the robot 22 may first determine that the failed strategy A left the target input 58 in a state where the drop-down list was already displayed, and therefore may begin performing strategy B from the intermediate stage where the drop-down list is displayed.

[0104] In step 824, the robot 22 may determine whether the current strategy was successful in satisfying the respective target inputs. In some embodiments, the robot 22 measures success according to status reports and / or error messages generated during execution of the current strategy (step 822). See, for example, success indicator 77 in FIG. 6. An alternative embodiment determines that the current strategy is successful when all RPA activities 72 of the strategy 70 have completed execution. Such an embodiment may compare the state of the target inputs 58 upon completion of each activity 72 with a reference / expected state at that point during execution of the strategy 70, and determine that the respective activity has completed successfully when the states match. (See description of reference states above.) Yet another exemplary embodiment may determine that the current strategy is successful only if the target inputs contain the desired input data 75. To determine the current contents of the target fields, the robot 22 may ask the RPA driver(s) 25 to parse a programmatic representation of the target UI 37 (e.g., DOM, UI tree, etc.). In another example, step 824 may include taking a snapshot of the target UI 37 or the target input and determining the current contents of the target input using an image processing AI module.

[0105] In response to the successful entry, in step 826, the robot 22 may recalculate the current strategy rankings to promote strategies for which the form entry was successful, so that each strategy may be prioritized in subsequent attempts to enter each target input. Then, in step 828, some embodiments send a ranking update 56 to the strategy ranking register 52 and / or local cache 152 (see, e.g., FIG. 4).

[0106] The ranking updates 56 may include updated strategy indicators that promote successful strategies with respect to other form entry strategies. In an alternative embodiment, the ranking updates 56 may include at least an identifier of the successful strategy and a target ID that identifies the respective target input. Additional data included in the updates 56 may include a list of failed strategies 70 attempted on the way to the current successful entry and / or the contents of the associated success indicators 77, error messages, etc. The server 32 may then use such data to update the ranking indicators 54 to promote / prioritize successful strategies over other strategies currently associated with the respective target input. The failed strategies may be demoted, e.g., by lowering their rank, so that they are less prioritized in future attempts to enter the respective target input. The server 32 may then update the ranking register 52 to include the updated ranking indicators.

[0107] FIG. 10 illustrates an exemplary hardware configuration of a computer system 80 programmed to perform some of the methods described herein. The computer system 80 may represent any of the RPA clients 12a-c of FIG. 1, the RPA client 12 of FIG. 4, and the RPA server(s) 32. The appliance illustrated is a personal computer; other computer systems, such as servers, mobile phones, tablet computers, wearable computing devices, etc., may be configured slightly differently. The processor(s) 82 include physical devices (e.g., microprocessors, multi-core integrated circuits formed on a semiconductor substrate) configured to perform calculations and / or logical operations with a set of signals and / or data. Such signals or data may be encoded and sent to the processor(s) 82 in the form of processor instructions, e.g., machine code. The processor 82 may include an array of central processing units (CPUs) and / or graphics processing units (GPUs).

[0108] The memory unit 83 may include volatile computer-readable media (e.g., Dynamic Random Access Memory - DRAM) that stores data and / or instruction encodings accessed or generated by the processor(s) 82 in the course of performing operations. The input devices 84 may include, among others, a computer keyboard, a mouse, a trackpad, and a microphone, and include respective hardware interfaces and / or adapters that allow a user to introduce data and / or instructions into the computer system 80. The output devices 85 may include, among others, display devices such as a monitor and speakers, and hardware interfaces / adapters such as a graphics card, and include respective computing devices that allow the respective computing devices to communicate data to a user. In some embodiments, the input and output devices 84-85 share a common piece of hardware (e.g., a touch screen). The storage device 86 includes computer-readable media that allows for non-volatile storage, reading, and writing of software instructions and / or data. Exemplary storage devices include magnetic and optical disks and flash memory devices, as well as removable media such as CD and / or DVD disks and drives. Network adapter(s) 87 contain mechanical, electrical, and signaling circuitry for communicating data over a physical link coupled to an electronic communications network (e.g., FIG. 1) and / or other device / computer system. Adapter(s) 87 may be configured to transmit and / or receive data using a variety of communications protocols.

[0109] Controller hub 90 generally represents multiple system, peripheral, and / or chipset buses, and / or any other circuitry that enables communication between processor(s) 82 and the remaining hardware components of computer system 80. For example, controller hub 90 may include memory controllers, input / output (I / O) controllers, and interrupt controllers. Depending on the hardware manufacturer, some such controllers may be built into a single integrated circuit and / or integrated with processor(s) 82. In another example, controller hub 90 may include a northbridge that connects processor 82 to memory 83 and / or a southbridge that connects processor 82 to devices 84, 85, 86, and 87.

[0110] The above-described exemplary systems and methods facilitate the design and operation of an RPA system. In traditional RPA, the complete specification of the workflow, including the strategy for filling each target field, is determined by the RPA developer at design time. However, in practice, a typical target user interface, such as a web page of a complex enterprise resource planning (ERP) platform, may change frequently and without notice. For example, the designer of the respective web page may change the type of input from simple text to a drop-down menu, or change the content or behavior of a drop-down menu to accommodate new functionality. If the RPA strategy for filling each input is fixed, the robot cannot adapt to such changes in the target UI or other unexpected events. Traditional RPA robots have no alternative form-filling strategies and cannot learn from failures.

[0111] This lack of adaptability is a major obstacle to developing autonomous RPA robots, for example, AI-enabled form-filling robots that can extract structured data from various source documents, interact with various forms, and / or handle unexpected situations.

[0112] In contrast to traditional RPA, some embodiments of the present invention provide the RPA robot with a strategy ranking indicator that identifies multiple strategies for filling target form fields, allowing the RPA robot to autonomously fall back to an alternative strategy when one strategy fails. In some embodiments, a form filling strategy includes a set of RPA activities for interacting with a particular type of form field (e.g., simple text input, drop-down list, date field, etc.). Strategies can be developed independently of other RPA workflows, provided to the robot as a separate library, and / or included in an RPA package that describes a more complex workflow.

[0113] The ranking indicator further ranks each form-filling strategy, allowing the RPA robot to select among available strategies according to its rank. Some embodiments give preference to higher-ranked strategies. The strategy ranking indicator is dynamically updated to promote successful form-filling strategies and / or demote unsuccessful strategies, which can be used by other instances of the RPA robot. Such dynamic updates improve the autonomy and performance of the RPA robot, and further improve the user experience, since each RPA robot can benefit from the experience of other robots that have previously tried to fill the same form. In other words, instead of performing the same tedious trial-and-error procedure, the RPA robot can directly try strategies identified by its predecessors as successful with the respective target input.

[0114] To facilitate sharing of information, some embodiments maintain a centralized strategy ranking database that associates an individual ranking indicator 54 with each individual target UI 37 and further with each form field 58 displayed by the respective UI. For example, a web form is identified by its URL, and each individual field 58 is identified by a set of attribute-value pair characteristics extracted from the DOM of the respective web page. Some embodiments further optimize automated form filling by maintaining a local strategy ranking cache on each RPA client machine that runs an RPA robot.

[0115] The dynamic updating of strategies and rankings can be seen as a process of collective training of RPA robots based on real-world data, where a swarm of independent and autonomous RPA robots collaborate to identify successful strategies for filling various target forms and share such information among them in real time. Successful strategies can be promoted and unsuccessful strategies demoted, all this while maintaining a fallback mechanism where multiple alternative strategies are available for each target form field.

[0116] It will be apparent to those skilled in the art that the above embodiments can be modified in many ways without departing from the scope of the present invention. Therefore, the scope of the present invention should be determined by the following claims and their legal equivalents.

Claims

1. 1. A method of optimizing a Robotic Process Automation (RPA) procedure for populating a target form field displayed by a target user interface (UI), the method comprising: obtaining a strategy ranking indicator from a database according to an identifier of the target UI, wherein the strategy ranking indicator identifies a plurality of alternative RPA strategies for populating the target form field, and further indicates a ranking among the plurality of alternative RPA strategies, each of the plurality of alternative RPA strategies including a distinct sequence of RPA activities configured to populate the target form field; in response to selecting a first RPA strategy from the plurality of alternative RPA strategies according to the ranking, performing a first attempt to fill in the selected form field according to the first RPA strategy; in response to determining that the first attempt was unsuccessful, selecting a second RPA strategy from the plurality of alternative RPA strategies according to the ranking; In response to selecting the second RPA strategy, performing a second attempt to fill in the selected form field according to the second RPA strategy; and updating the strategy ranking indicators in the database in response to determining that the second attempt is successful, the updating being configured to change the rankings such that selection of the second RPA strategy is prioritized over selection of the first RPA strategy.

2. 2. The method of claim 1, wherein the at least one hardware processor is configured to determine an order in which each of the plurality of alternative RPA strategies is selected for use according to the ranking.

3. 2. The method of claim 1, wherein the ranking indicates the likelihood that each RPA strategy will be successful in populating the target form field.

4. 2. The method of claim 1, wherein the strategy ranking indicator comprises an ordered sequence of strategy identifiers, each strategy identifier identifying a distinct RPA strategy of the plurality of alternative RPA strategies, and the ranking is determined according to a position of each strategy identifier within the ordered sequence.

5. The method of claim 4 , wherein updating the strategy ranking indicators comprises rearranging the ordered sequence of strategy identifiers.

6. The method of claim 1 , wherein the target UI comprises a browser interface that displays a target web page, and the identifier of the target UI comprises a uniform record identifier (URI) of the target web page.

7. The method of claim 1 , wherein the database associates the strategy ranking indicator with an identifier of the target UI.

8. The method of claim 1 , further comprising selectively retrieving the strategy ranking indicator from the database according to a field ID that identifies the target form field among a plurality of form fields displayed by the target UI.

9. The method of claim 8 , wherein the field ID comprises a text label displayed adjacent to the target form field.

10. 2. The method of claim 1, further comprising: in response to selecting the first RPA strategy, selectively obtaining a strategy package from a remote repository using the at least one hardware processor, the strategy package including a specification for the first RPA package.

11. 1. A computer system including at least one hardware processor configured to optimize a robotic process automation (RPA) procedure for populating a target form field displayed by a target user interface (UI), the optimizing the RPA procedure comprising: obtaining a strategy ranking indicator from a database according to an identifier of the target UI, wherein the strategy ranking indicator identifies a plurality of alternative RPA strategies for populating the target form field, and further indicates a ranking among the plurality of alternative RPA strategies, each of the plurality of alternative RPA strategies including a distinct sequence of RPA activities configured to populate the target form field; in response to selecting a first RPA strategy from the plurality of alternative RPA strategies according to the ranking, performing a first attempt to fill in the selected form field according to the first RPA strategy; in response to determining that the first attempt was unsuccessful, selecting a second RPA strategy from the plurality of alternative RPA strategies according to the ranking; In response to selecting the second RPA strategy, performing a second attempt to fill in the selected form field according to the second RPA strategy; and updating the strategy ranking indicators in the database in response to determining that the second attempt is successful, the updating being configured to change the rankings such that selection of the second RPA strategy is prioritized over selection of the first RPA strategy.

12. 12. The computer system of claim 11, wherein the at least one hardware processor is configured to determine an order in which each of the plurality of alternative RPA strategies is selected for use according to the ranking.

13. 2. The computer system of claim 1, wherein the ranking indicates the likelihood that each RPA strategy will be successful in populating the target form field.

14. 12. The computer system of claim 11, wherein the strategy ranking indicator comprises an ordered sequence of strategy identifiers, each strategy identifier identifying a distinct RPA strategy of the plurality of alternative RPA strategies, and the ranking is determined according to a position of each strategy identifier within the ordered sequence.

15. The computer system of claim 14 , wherein updating the strategy ranking indicators comprises rearranging the ordered sequence of strategy identifiers.

16. 12. The computer system of claim 11, wherein the target UI comprises a browser interface that displays a target web page, and the identifier of the target UI comprises a uniform record identifier (URI) of the target web page.

17. The computer system of claim 11 , wherein the database associates the strategy ranking indicator with an identifier of the target UI.

18. 12. The computer system of claim 11, wherein the at least one hardware processor is configured to selectively retrieve the strategy ranking indicator from the database further according to a field ID that identifies the target form field among a plurality of form fields displayed by the target UI.

19. 20. The computer system of claim 18, wherein the field ID comprises a text label displayed adjacent to the target form field.

20. 12. The computer system of claim 11, wherein the at least one hardware processor is further configured to selectively retrieve a strategy package from a remote repository in response to selecting the first RPA strategy, the strategy package including a specification of the first RPA package.

21. 1. A non-transitory computer-readable medium storing instructions that, when performed by at least one hardware processor of a computer system, cause the computer system to optimize a Robotic Process Automation (RPA) procedure for populating a target form field displayed by a target user interface (UI), the optimizing the RPA procedure comprising: obtaining a strategy ranking indicator from a database according to an identifier of the target UI, wherein the strategy ranking indicator identifies a plurality of alternative RPA strategies for populating the target form field, and further indicates a ranking among the plurality of alternative RPA strategies, each of the plurality of alternative RPA strategies including a distinct sequence of RPA activities configured to populate the target form field; in response to selecting a first RPA strategy from the plurality of alternative RPA strategies according to the ranking, performing a first attempt to fill in the selected form field according to the first RPA strategy; in response to determining that the first attempt was unsuccessful, selecting a second RPA strategy from the plurality of alternative RPA strategies according to the ranking; In response to selecting the second RPA strategy, performing a second attempt to fill in the selected form field according to the second RPA strategy; and updating the strategy ranking indicators in the database in response to determining that the second attempt is successful, the updating being configured to change the rankings such that selection of the second RPA strategy is prioritized over selection of the first RPA strategy.

Images