Reinforcement Learning in Robotic Process Automation

Abstract

Reinforcement learning can be used to train machine learning (ML) models for robotic process automation (RPA) performed by robots. A policy network can be used that learns to achieve a clear output by providing specific inputs. In other words, the policy network notifies the system if it is approaching a winning state. The policy network can be refined by the robot automatically, or with periodic human assistance, to reach a winning state or achieve a more optimal winning state. Robots can also create other robots that utilize reinforcement learning.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Nonprovisional Patent Application No. 16 / 707,858, filed December 9, 2019, and U.S. Provisional Patent Application No. 62 / 915,385, filed October 15, 2019. The subject matter of these prior applications is incorporated herein by reference in its entirety.

[0002] The present invention relates generally to robotic process automation (RPA), and more specifically to reinforcement learning in RPA. [Background technology]

[0003] Most current machine learning algorithms utilize historical data as a basis for learning. Reinforcement learning, however, strikes a balance between leveraging existing knowledge and exploring the unknown. In exploration, random actions guided by current knowledge are used in an attempt to approach an objective function (i.e., a "winning" state). In other words, such algorithms perform an exploration function if a winning state has not been reached. Such techniques have been used to teach computers to play the games "Breakout" and "Go".

[0004] Reinforcement learning has not been used in the context of RPA, and traditional reinforcement learning techniques require significant amounts of computational power and time, making them unsuitable for many applications. Therefore, improved reinforcement learning solutions could be beneficial. Summary of the Invention

[0005] Certain embodiments of the present invention may provide solutions to problems and needs in the art that have not yet been fully identified, recognized, or solved by current RPA technology. For example, some embodiments of the present invention relate to reinforcement learning in RPA.

[0006] In one embodiment, a computer-implemented method for training an ML model for RPA using reinforcement learning includes training the ML model by running simulations on training data using the ML model. The ML model has a reward function. If the ML model does not achieve convergence defined by the reward function based on one or more criteria, the computer-implemented method includes requesting human assistance, monitoring actions taken by the human on the computing system, and modifying a policy network of the ML model, the reward function, or both based on the actions taken by the human.

[0007] In another embodiment, a computer-implemented method for training an ML model for RPA using reinforcement learning includes running a simulation on training data using the ML model. The ML model has a reward function. If the ML model does not achieve convergence defined by the reward function based on one or more criteria, the computer-implemented method includes monitoring actions taken by a human on the computing system and modifying a policy network of the ML model, the reward function, or both, based on the actions taken by the human. The computer-implemented method includes repeating the steps of running a simulation on training data using the ML model, monitoring actions taken by a human on the computing system, and modifying the policy network, the reward function, or both, until convergence is achieved.

[0008] In yet another embodiment, a computer-implemented method for training an ML model for RPA using reinforcement learning includes training an ML model using a policy network. The ML model has a reward function. The computer-implemented method also includes deploying the trained ML model. The deployment of the trained ML model includes modifying one or more activities in an RPA workflow performed by the RPA robot to invoke the trained ML model. The computer-implemented method further includes invoking the trained ML model at runtime by the RPA robot. [Brief description of the drawings]

[0009] So that the advantages of certain embodiments of the invention may be readily understood, a more particular description of the invention, briefly described above, will be provided by reference to specific embodiments which are illustrated in the accompanying drawings, it being understood that these drawings depict only typical embodiments of the invention and therefore should not be considered as limiting its scope, but the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0010] [Figure 1] FIG. 1 is an architectural diagram illustrating an RPA system according to one embodiment of the present invention.

[0011] [Diagram 2] FIG. 1 is an architectural diagram illustrating a deployed RPA system according to one embodiment of the present invention.

[0012] [Diagram 3] FIG. 2 is an architecture diagram illustrating the relationships between designers, activities, and drivers according to one embodiment of the present invention.

[0013] [Figure 4] FIG. 1 is an architectural diagram illustrating an RPA system according to one embodiment of the present invention.

[0014] [Diagram 5] FIG. 1 is an architectural diagram illustrating a computing system configured to implement reinforcement learning in RPA, according to one embodiment of the present invention.

[0015] [Figure 6] 1 is a flowchart illustrating a process for implementing reinforcement learning in an RPA in accordance with one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Some embodiments relate to reinforcement learning in RPA. In reinforcement learning, there is an exploration phase. Process mining can provide various states of the system (e.g., by deploying listeners on computing systems and collecting information about actions taken by users, robots, or both on those computing systems, such as clicking buttons, opening / closing applications, entering text, etc.). By specifying a start state (e.g., a set of email addresses) and an end state (e.g., ending with all email addresses properly entered into Salesforce), the system can randomly try to find a solution from the start state to the end state, using a reward function that explores intermediate transitions and steps with both short-term and long-term rewards to guide the exploration of the state space. In the case of trading, for example, the system can be asked to keep trading randomly until it is profitable.

[0017] However, the system needs some guidance to achieve this goal: if a solution does not meet certain criteria (e.g., after a certain number of tries, a predefined time, etc.), the system can request human guidance. This process may be repeated until a solution is reached.

[0018] Some embodiments use a policy network that adjusts and defines the reward function. In some embodiments, this may be accomplished by the system monitoring actions taken by humans. If the human modifies the operation by taking an action on the computing system, whether or not the human knows that he or she is interacting with the robot, the policy network can learn from this action and incorporate it into the reward function. This helps the robot get closer to the goal of the reward function. In other words, the policy network informs the robot if it is approaching a winning state. As used herein, a winning state may not be related to a game. Rather, it is any suitable desired outcome, including, but not limited to, correctly recognizing a face or human emotion in an image, processing a document, controlling a vehicle in a desired manner, or any other suitable desired outcome without departing from the scope of the present invention.

[0019] To approach a winning state, the robot can analyze the underlying distribution. If the results are in the same range (e.g., within half a standard deviation, one standard deviation, etc.), this provides information about the robot's performance. The analysis can be a search function, a statistical distribution, etc. For example, if the robot cannot log into Salesforce®, the robot can watch a human do so. In other words, the robot can monitor the Application Programming Interface (API) calls made by the human (e.g., the human clicks in the username field, enters the username, clicks in the password field, enters the password, clicks the submit button).

[0020] Rather than performing a completely random search, some embodiments determine what the policy should be via a policy network. This can significantly reduce the number of simulations required to achieve a winning state, allowing machine learning (ML) models trained via reinforcement learning to be developed and deployed more quickly, trained using fewer computing hardware resources, or both. Indeed, in some embodiments, reinforcement learning may be performed on a user's own computing system, such as a desktop computer, laptop computer, smartphone, etc., which is typically not traditionally possible.

[0021] In the context of interacting with a user interface, for example, a robot can randomly click on a screen to determine what is clickable. Reinforcement learning can then be used to drive the desired automation. For example, given a spreadsheet of email addresses and desired email text, reinforcement learning can be used to discover which clickable button will open an email application, discover where different text should be entered within the email, and discover which button to click to send the email. In this example, a policy network can be used to first determine whether sending an email is possible or desirable. Once this determination is made, the policy network can determine whether the email address and email body text from the spreadsheet can be copied into the email before sending.

[0022] In complex use cases, it is often difficult or impossible to initially determine what the reward function should be. Consider, for example, a case where a video feed from a camera is used to try to infer what a human is doing. For example, if the reward function is not converging quickly enough, or at all, a human can be involved in further refinement of the reward function. A controlled environment can be provided at design time to test and train ML models and refine the reward function. At run time, new or refined ML models can be deployed, and the workflow performed by the robot can be modified to invoke new or refined ML models by modifying the respective activities in the workflow.

[0023] As a non-limiting example, consider the case where it is desirable to train an RPA robot to learn how to find the lowest price per unit on Amazon® using reinforcement learning where there are millions of links. The robot can randomly explore and click on links in a simulated environment, but there are too many options for the robot to follow reasonably and achieve convergence to a winning state. A human can initially review the robot's actions and provide some guidance (e.g., manually show the robot how to enter keywords and where to click). The robot learns by watching these actions to not click mindlessly. Given this input, the robot can train and explore further. The robot then modifies its policy network and moves closer to the desired outcome. The human can continue to provide some guidance periodically or under certain conditions until the robot has enough guidance to achieve convergence.

[0024] Thus, humans can assist the robot at any desired layer in training, personalization, etc. to make the ML model more effective. When and whether to involve a human can depend on the particular application and behavior. For example, if the performance of the ML model is degrading beyond a certain threshold, it may trigger the system to ask a human for next steps and / or examples. In the Amazon® example, if the robot performs 10,000 clicks without getting an answer, human review and guidance may be initiated.

[0025] In some embodiments, robots can utilize reinforcement learning to automatically build policy networks themselves. These policy networks can be built based on observations of human interactions with computing systems in given scenarios (e.g., which APIs are called). In certain embodiments, RPA robots themselves can build other robots that implement reinforcement learning.

[0026] FIG. 1 is an architecture diagram illustrating an RPA system 100, according to one embodiment of the present invention. The RPA system 100 includes a designer 110 that enables developers to design and implement workflows. The designer 110 can provide application integration and solutions for automating third-party applications, management information technology (IT) tasks, and business IT processes. The designer 110 can facilitate the development of automation projects, which are graphical representations of business processes. Simply put, the designer 110 facilitates the development and deployment of workflows and robots.

[0027] Automation projects enable rule-based process automation by giving developers control over the order of execution and relationships between custom sets of steps developed in a workflow, defined herein as "activities." One commercial example of an embodiment of the designer 110 is UiPath Studio™. Each activity can include actions such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0028] Some types of workflows may include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suited to linear processes that allow flow from one activity to another without disrupting the workflow. Flowcharts are particularly suited to more complex business logic, allowing for the integration of decisions and the connection of activities in more diverse ways through multiple branching logical operators. FSMs may be particularly suited to large workflows. FSMs may use a finite number of states in their execution that are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suited to determining the behavior of a workflow when an execution error is encountered and to debugging the process.

[0029] Once a workflow is developed in Designer 110, the execution of the business process is orchestrated by Conductor 120, which orchestrates one or more Robots 130 that execute the workflow developed in Designer 110. One commercial example of an embodiment of Conductor 120 is UiPath Orchestrator™. Conductor 120 facilitates the creation, monitoring, and management of deployment of resources in the environment. Conductor 120 can act as one of the integration or aggregation points with third party solutions and applications.

[0030] The conductor 120 can manage all the robots 130 connecting and executing the robots 130 from a centralized point. The types of robots 130 that can be managed include, but are not limited to, attended robots 132, unattended robots 134, development robots (similar to unattended robots 134 but used for development and testing purposes), and non-production robots (similar to attended robots 132 but used for development and testing purposes). Attended robots 132 are triggered by user events and operate alongside humans on the same computing system. Attended robots 132 can be used with the conductor 120 for centralized process deployment and recording. Attended robots 132 can help human users accomplish various tasks and can be triggered by user events. In some embodiments, processes cannot be started from the conductor 120 for this type of robot and / or cannot be run under a locked screen. In certain embodiments, attended robots 132 can only be launched from the robot tray or command prompt. In some embodiments, the attended robot 132 should operate under human supervision.

[0031] Unattended robots 134 operate unattended in a virtual environment and can automate many processes. Unattended robots 134 can be responsible for providing remote execution, monitoring, scheduling, and work queue support. In some embodiments, debugging of all robot types can be performed in the designer 110. Both attended and unattended robots can automate a variety of systems and applications, including but not limited to mainframes, web applications, VMs, enterprise applications (e.g., those manufactured by SAP®, SalesForce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0032] The conductor 120 may have a variety of functions, including but not limited to provisioning, deployment, versioning, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between the robots 130 and the conductor 120 (e.g., web applications). Deployment may include ensuring correct delivery of package versions to the robots 130 assigned for execution. Versioning may include managing unique instances of some processes or configurations in some embodiments. Configuration may include maintaining and delivering robot environments and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs in a database (e.g., an SQL database) and / or another storage mechanism (e.g., ElasticSearch®, which provides the ability to store and quickly query large data sets). The conductor 120 may provide interconnectivity by acting as a centralized point of communication for third party solutions and / or applications.

[0033] The robot 130 is an execution agent that executes the workflow built in the designer 110. One commercial example of some embodiments of the robot 130 is UiPath Robots™. In some embodiments, the robot 130 installs the Microsoft Windows Service Control Manager (SCM) management service by default. As a result, such a robot 130 can open an interactive Windows session under the local system account and have the rights of a Windows service.

[0034] In some embodiments, the robot 130 can be installed in user mode. For such robots 130, this means that a given robot 130 has the same rights as the user for whom it is installed. This feature may also be available for high density (HD) robots, ensuring full utilization of each machine at its maximum potential. In some embodiments, any type of robot 130 can be configured in an HD environment.

[0035] The robot 130 in some embodiments is divided into several components, each dedicated to a specific automation task. The robot components in some embodiments include, but are not limited to, an SCM managed robot service, a user mode robot service, an executor, an agent, and a command line. The SCM managed robot service manages and monitors the Windows session and acts as a proxy between the conductor 120 and the execution host (i.e., the computing system on which the robot 130 executes). These services are trusted with and manage the credentials of the robot 130. The console application is launched by the SCM under the local system.

[0036] The user-mode robot service in some embodiments manages and monitors the Windows session and acts as a proxy between the conductor 120 and the execution host. The user-mode robot service can trust and manage credentials for the robot 130. If the SCM management robot service is not installed, the Windows application can be launched automatically.

[0037] An Executor can run a given job under a Windows session (i.e., an Executor can execute a workflow). An Executor can be aware of per-monitor dots per inch (DPI) settings. An Agent can be a Windows Presentation Foundation (WPF) application that displays available jobs in a system tray window. An Agent can be a client of a Service. An Agent can request to start or stop a job and change settings. A Command Line is a client of a Service. A Command Line is a console application that can request to start a job and wait for its output.

[0038] Splitting up the components of the robot 130 as described above helps developers, support users, and computing systems more easily execute, identify, and track what each component is doing. In this way, special behaviors can be configured per component, such as setting different firewall rules for executors and services. The executors, in some embodiments, can always be aware of the DPI setting per monitor. As a result, workflows can be executed at any DPI regardless of the configuration of the computing system on which they were created. In some embodiments, projects from the designer 110 can be independent of the browser zoom level. For applications that are not DPI aware or are intentionally marked as not aware, DPI can be disabled in some embodiments.

[0039] FIG. 2 is an architecture diagram illustrating a deployed RPA system 200 according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be or may be part of the RPA system 100 of FIG. 1. It should be noted that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 210 includes an executor 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not be running on the computing system 210. The executor 212 is a running process. As shown in FIG. 2, several business projects may be running simultaneously. The agent 214 (e.g., a Windows service) is a single connection point for all the executors 212 in this embodiment. All messages in this embodiment are logged to the conductor 230, which further processes them via the database server 240, the indexer server 250, or both. As discussed above with respect to FIG. 1, the executor 212 may be a robotic component.

[0040] In some embodiments, a robot represents an association between a machine name and a username. A robot can manage multiple executors simultaneously. In computing systems that support multiple interactive sessions running simultaneously (e.g., Windows Server 2012), multiple robots may run simultaneously, each running in a separate Windows session using a unique username. This is referred to above as an HD Robot.

[0041] The agent 214 is also responsible for transmitting the robot's status (e.g., periodically sending "heartbeat" messages to indicate that the robot is still functioning) and downloading necessary versions of packages to be fulfilled. Communication between the agent 214 and the conductor 230 is always initiated by the agent 214 in some embodiments. In a notification scenario, the agent 214 can open a WebSocket channel that is later used by the conductor 230 to send commands to the robot (e.g., start, stop, etc.).

[0042] The server side includes a presentation layer (web application 232, Open Data Protocol (OData) Representational State Transfer (REST) ​​Application Programming Interface (API) endpoints 234, and notifications and monitoring 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240, indexer server 250). Conductor 230 includes web application 232, OData REST API endpoints 234, notifications and monitoring 236, and API implementation / business logic 238. In some embodiments, most actions that a user performs within the conductor 220 interface (e.g., via browser 220) are performed by calling various APIs. Such actions may include, but are not limited to, starting a job on a robot, adding / removing data in a queue, scheduling a job to run unattended, etc., without departing from the scope of the invention. Web application 232 is the visual layer of the server platform. In this embodiment, web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the invention. A user interacts with web pages from web application 232, in this embodiment via browser 220, to perform various actions to control conductor 230. For example, a user can create robot groups, assign packages to robots, analyze logs per robot and / or per process, start and stop robots, etc.

[0043] In addition to the web application 232, the conductor 230 also includes a services layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the invention. The REST API is consumed by both the web application 232 and the agent 214, which in this embodiment is the administrator of one or more robots on a client computer.

[0044] The REST API in this embodiment covers configuration, logging, monitoring, and queuing functions. The configuration endpoint may be used in some embodiments to define and configure application users, permissions, robots, assets, releases, and environments. For example, a logging REST endpoint may be used to log various information such as errors, explicit messages sent by the robot, and other environment specific information. A deployment REST endpoint may be used by a robot to query the package version that should be committed when a start job command is used in the conductor 230. The queuing REST endpoint may be responsible for queue and queue item management, such as adding data to the queue, getting transactions from the queue, setting the state of transactions, etc.

[0045] The monitoring REST endpoints can monitor the web application 232 and the agents 214. The notification monitoring API 236 can be a REST endpoint used to register the agents 214, deliver configuration settings to the agents 214, and send / receive notifications from the server and the agents 214. The notification monitoring API 236 can also use WebSocket communication in some embodiments.

[0046] The persistence layer includes a pair of servers in this embodiment: a database server 240 (e.g., SQL Server) and an indexer server 250. The database server 240 in this embodiment stores configurations for robots, robot groups, associated processes, users, roles, schedules, etc. This information is managed in some embodiments via a web application 232. The database server 240 can manage queues and queue items. In some embodiments, the database server 240 can store messages logged by the robots (in addition to or instead of the indexer server 250).

[0047] Indexer server 250 is optional in some embodiments and stores and indexes information logged by the robots. In certain embodiments, indexer server 250 can be disabled through a configuration setting. In some embodiments, indexer server 250 uses ElasticSearch®, a full-text search engine from an open source project. Messages logged by robots (e.g., using activities such as log messages or line writes) may be sent via a logging REST endpoint to indexer server 250, where they are indexed for future use.

[0048] FIG. 3 is an architecture diagram illustrating the relationship 300 between a designer 310, activities 320, 330, and a driver 340, according to one embodiment of the present invention. In accordance with the above, a developer uses the designer 310 to develop a workflow to be performed by the robot. The workflow can include user-defined activities 320 and UI automation activities 330. Some embodiments can identify non-textual visual components in an image, referred to herein as computer vision (CV). Some CV activities related to such components can include, but are not limited to, click, type, get text, hover, element presence, refresh range, highlight, and the like. In some embodiments, click can identify an element using, for example, CV, optical character recognition (OCR), fuzzy character matching, and multi-anchors, and click on it. Type can identify an element using the above and type within an element. Get text can identify the location of a particular text using OCR and scan it. Hover can identify an element and hover it. Element Present can check if an element is present on the screen using the techniques described above. In some embodiments, there may be hundreds or thousands of activities that can be implemented in the designer 310. However, any number and / or type of activities may be utilized without departing from the scope of the invention.

[0049] UI automation activities 330 are a subset of special low-level activities that are written into lower level code (e.g., CV activities) and facilitate interactions with the screen. UI automation activities 330 facilitate these interactions via drivers 340 that allow the robot to interact with desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc.

[0050] Drivers 340 can interact with the OS at a low level, looking for hooks, monitoring keys, etc. They can facilitate integration with Chrome®, IE®, Citrix®, SAP®, etc. For example, a "click" activity performs the same role in these different applications via drivers 340.

[0051] FIG. 4 is an architecture diagram illustrating an RPA system 400, according to one embodiment of the present invention. In some embodiments, the RPA system 400 may be or include the RPA systems 100 and / or 200 of FIG. 1 and / or FIG. 2. The RPA system 400 includes multiple client computing systems 410 that execute robots. The computing systems 410 may communicate with a conductor computing system 420 via web applications executed thereon. The conductor computing system 420 may in turn communicate with a database server 430 and an optional indexer server 440.

[0052] 1 and 3, it should be noted that although web applications are used in these embodiments, any suitable client / server software may be used without departing from the scope of the present invention. For example, the conductor may run a server-side application that communicates with a non-web-based client software application on a client computing system.

[0053] FIG. 5 is an architecture diagram illustrating a computing system 500 configured to perform reinforcement learning in RPA, according to an embodiment of the present invention. In some embodiments, the computing system 500 may be one or more of the computing systems illustrated and / or described herein. The computing system 500 includes a bus 505 or other communication mechanism for communicating information, and a processor 510 coupled to the bus 505 for processing information. The processor 510 may be any type of general-purpose or special-purpose processor, including a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor 510 may also have multiple processing cores, at least some of which may be configured to perform a particular function. In some embodiments, multiple parallel processing may be used. In certain embodiments, at least one of the processors 510 may be a neuromorphic circuit that includes processing elements that mimic biological neurons. In some embodiments, the neuromorphic circuit may not require typical components of a von Neumann computing architecture.

[0054] The computing system 500 further includes a memory 515 for storing information and instructions executed by the processor 510. The memory 515 can be comprised of any combination of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage such as magnetic or optical disks, or any other type of non-transitory computer-readable medium, or any combination thereof. The non-transitory computer-readable medium may be any available medium accessible by the processor 510 and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both.

[0055] Additionally, the computing system 500 includes a communications device 520, such as a transceiver for providing access to a communications network via wireless and / or wired connections. In some embodiments, the communications device 520 may be a communications device for a variety of communications networks, including Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiplexing (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile Communications (GSM) communications, General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), cdma2000, Wideband CDMA (W-CDMA), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), High Speed ​​... The communication device 520 may be configured to use any of the following communication standards and / or protocols: High Speed ​​Passive Access (HSPA), Long Term Evolution (LTE), LTE Advanced (LTE-A), 802.11x, Wi-Fi, Zigbee, Ultra Wideband (UWB), 802.16x, 802.15, Home Node B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near Field Communication (NFC), Fifth Generation (5G), New Radio (NR), any combination thereof, and / or any other currently existing or future implemented communication standards and / or protocols without departing from the scope of the present invention. In some embodiments, the communication device 520 may include one or more antennas that are single, array, phased, switched, beamforming, beamsteering, combinations thereof, and / or any other antenna configuration without departing from the scope of the present invention.

[0056] The processor 510 is further coupled via bus 505 to a display 525, such as a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, a Retina® display, an in-plane switching (IPS) display, or any other suitable display for displaying information to a user. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, or the like, using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal techniques, acoustic pulse recognition, frustrated total internal reflection, or the like. Any suitable display device and haptic I / O may be used without departing from the scope of the invention.

[0057] A keyboard 530 and cursor control devices 535, such as a computer mouse, touchpad, etc., are further coupled to the bus 505 to allow a user to interface with the computing system 500. However, in certain embodiments, a physical keyboard and mouse may not be present and the user may interact with the device solely through the display 525 and / or touchpad (not shown). Any type and combination of input devices may be used as a matter of design choice. In certain embodiments, there are no physical input devices and / or displays. For example, a user may interact with the computing system 500 remotely via another computing system in communication with it, or the computing system 500 may operate autonomously.

[0058] The memory 515 stores software modules that provide functionality when executed by the processor 510. The modules include an operating system 540 for the computing system 500. The modules further include a reinforcement learning module 545 configured to perform all or a portion of the processes described herein or derivatives thereof. The computing system 500 may include one or more additional functional modules 550 that include additional functionality.

[0059] Those skilled in the art will appreciate that the "system" may be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device or combination of devices without departing from the scope of the present invention. Presenting the above functions as being performed by a "system" is in no way intended to limit the scope of the present invention, but is intended to provide one example of many embodiments of the present invention. Indeed, the methods, systems, and apparatus disclosed herein may be implemented in localized and distributed forms consistent with computing technologies, including cloud computing systems.

[0060] It should be noted that some of the system features described herein have been presented as modules to more specifically emphasize their implementation independence. For example, a module may be implemented as a hardware circuit including custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, and the like.

[0061] Modules may also be implemented at least partially in software for execution by various types of processors. An identified unit of executable code may include one or more physical or logical blocks of computer instructions, which may be organized, for example, as an object, procedure, or function. Nevertheless, executable files of identified modules need not be physically located together, but may include different instructions stored in different locations that, when logically combined together, comprise a module and achieve the stated purpose of the module. Furthermore, modules may be stored on a computer-readable medium, which may be, for example, a hard disk drive, a flash device, RAM, tape, and / or any other such non-transitory computer-readable medium used to store data without departing from the scope of the present invention.

[0062] In fact, a module of executable code may be a single instruction, or many instructions, and may be distributed across several different code segments, different programs, and several memory devices. Similarly, operational data may be identified and illustrated herein in modules, and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set or distributed in different locations, including different storage devices, and may exist, at least in part, only as electronic signals on a system or network.

[0063] FIG. 6 is a flow chart illustrating a process 600 for implementing reinforcement learning in RPA, according to an embodiment of the present invention. In some embodiments, the training may be performed by a server, a user computing system, an RPA robot, or any combination thereof. The process begins with training the ML model with an initial reward function at 610 by running simulations on the training data using the ML model. If the ML model does not achieve convergence (i.e., the ML model has not reached a winning state) at 620 as defined by a reward function based on one or more criteria (e.g., after a predetermined number of simulations, after a predetermined time, a combination of these (e.g., after a predetermined time unless a predetermined number of attempts have been made in the meantime), etc.), human assistance is requested at 630, and the robot monitors actions taken by the human and modifies the policy network of the ML model, the reward function, or both at 640. For example, the robot can monitor API calls made during the human's interaction with the computing system and learn to mimic them. The process then returns to step 610 to run further simulations on the training data using the modified policy network, the modified reward function, or both.

[0064] The process of requesting human assistance, modifying the policy network, and running further simulations is repeated until convergence is achieved at 620. The trained ML model is then deployed at 650, and the deployed ML model is invoked at runtime by the RPA robot at 660. In some embodiments, the deployment of the trained ML model includes modifying one or more activities in the RPA workflow performed by the RPA robot to invoke the trained ML model. If it is detected at 670 that the performance of the ML model has degraded beyond a predetermined performance threshold (e.g., detection accuracy falls below a predetermined threshold, reward function goals are achieved less frequently without human action, etc.), the process returns to step 610 and performs retraining of the ML model.

[0065] The process steps performed in FIG. 6 may be performed by a computer program encoding instructions for a processor to perform at least a portion of the process described in FIG. 6 according to an embodiment of the present invention. The computer program may be embodied on a non-transitory computer readable medium. The computer readable medium may be, but is not limited to, a hard disk drive, a flash device, RAM, tape, and / or any other such medium or combination of media used to store data. The computer program may include encoded instructions for controlling a processor of a computing system (e.g., processor 510 of computing system 500 of FIG. 5) to perform all or a portion of the process steps described in FIG. 6, which may also be stored on a computer readable medium.

[0066] The computer program may be implemented in hardware, software, or a hybrid implementation. The computer program may be composed of modules designed to operatively communicate with each other and pass information or instructions for display. The computer program may be configured to run on a general purpose computer, an ASIC, or any other suitable device.

[0067] It will be readily understood that the components of the various embodiments of the present invention, as generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the detailed description of the embodiments of the present invention, as represented in the accompanying drawings, is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention.

[0068] The features, structures, or characteristics of the invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, references throughout this specification to "certain embodiments," "some embodiments," or similar language means that the particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. Thus, the appearances of the phrases "in certain embodiments," "in some embodiments," "in other embodiments," or similar language throughout this specification do not necessarily all refer to the same group of embodiments, but rather the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0069] It should be noted that the reference to features, advantages, or similar language throughout this specification does not mean that all of the features and advantages that may be realized in the present invention should be in any single embodiment of the present invention, or that any single embodiment of the present invention is. Rather, language referring to features and advantages is understood to mean that the particular feature, advantage, or characteristic described in connection with one embodiment is included in at least one embodiment of the present invention. Thus, the description of features and advantages and similar language throughout this specification may, but does not necessarily, refer to the same embodiment.

[0070] Furthermore, the described features, advantages, and characteristics of the invention may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the invention may be practiced without one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments of the invention.

[0071] Those skilled in the art will readily appreciate that the invention described above may be practiced with steps in a different order and / or hardware elements in a different configuration than that disclosed. Thus, while the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative constructions will be apparent while remaining within the spirit and scope of the invention. Accordingly, reference should be made to the appended claims to determine the scope of the invention.

Claims

1. 1. A computer-implemented method for training a machine learning (ML) model for robotic process automation (RPA) using reinforcement learning, comprising: training, by an RPA robot, a ML model by running a simulation on training data using the ML model, the ML model having a reward function; if the ML model does not achieve convergence as defined by the reward function based on one or more criteria, requesting human assistance by the RPA robot; monitoring, by the RPA robot, actions taken by a human on a computing system; and modifying, by the RPA robot, a policy network for the ML model, the reward function, or both based on the action taken by the human.

2. and repeating the steps of running the simulation on the training data using the ML model, requesting human assistance, monitoring the actions taken by the human on the computing system, and modifying the policy network, the reward function, or both, until convergence is achieved.

2. The computer-implemented method of claim 1.

3. After convergence is achieved, the method further comprises: deploying the trained ML model; 3. The computer-implemented method of claim 2, further comprising: invoking, by the RPA robot, the trained ML model at runtime.

4. 4. The computer-implemented method of claim 3, wherein deploying the ML model comprises modifying one or more activities in an RPA workflow performed by the RPA robot to invoke the trained ML model.

5. detecting, by the RPA robot, that performance of the ML model has degraded beyond a predetermined performance threshold; retraining the trained ML model until convergence is achieved; The computer-implemented method of claim 3 further comprising:

6. The computer-implemented method of claim 5 , wherein the predetermined performance threshold comprises a detection accuracy or frequency at which convergence is achieved without user action.

7. The computer-implemented method of claim 1 , wherein the one or more criteria include a predetermined number of attempts, a predetermined amount of time, or a combination thereof.

8. 2. The computer-implemented method of claim 1, wherein the monitoring of the actions taken by the human comprises monitoring application programming interface (API) calls made based on the actions taken by the human.

9. 1. A computer-implemented method for training a machine learning (ML) model for robotic process automation (RPA) using reinforcement learning, comprising: running, by an RPA robot, a simulation on training data using the ML model, the ML model having a reward function; if the ML model does not achieve convergence as defined by the reward function based on one or more criteria, monitoring, by the RPA robot, actions taken by a human on a computing system; modifying, by the RPA robot, a policy network for the ML model, the reward function, or both based on the action taken by the human; and repeating, by the RPA robot, the steps of running the simulation on the training data using the ML model, monitoring the actions taken by the human on the computing system, and modifying the policy network, the reward function, or both, until convergence is achieved.

10. After convergence is achieved, the method further comprises: deploying the trained ML model; 10. The computer-implemented method of claim 9, further comprising: invoking, by the RPA robot, the trained ML model at runtime.

11. 11. The computer-implemented method of claim 10, wherein deploying the ML model comprises modifying one or more activities in an RPA workflow performed by the RPA robot to invoke the trained ML model.

12. detecting, by the RPA robot, that performance of the ML model has degraded beyond a predetermined performance threshold; retraining the trained ML model until convergence is achieved; The computer-implemented method of claim 10, further comprising:

13. The computer-implemented method of claim 12 , wherein the predetermined performance threshold comprises a detection accuracy or frequency at which convergence is achieved without user action.

14. 10. The computer-implemented method of claim 9, wherein the one or more criteria include a predetermined number of attempts, a predetermined amount of time, or a combination thereof.

15. 10. The computer-implemented method of claim 9, wherein the monitoring of the actions taken by the human comprises monitoring application programming interface (API) calls made based on the actions taken by the human.

16. 1. A computer-implemented method for training a machine learning (ML) model for robotic process automation (RPA) using reinforcement learning, comprising: training the ML model using a policy network, the ML model having a reward function; deploying the trained ML model, wherein the deployment of the trained ML model comprises modifying one or more activities in an RPA workflow performed by an RPA robot to invoke the trained ML model; and invoking, at runtime, the trained ML model by the RPA robot.

17. detecting, by the RPA robot, that performance of the ML model has degraded beyond a predetermined performance threshold; retraining the ML model that was trained using reinforcement learning; 20. The computer-implemented method of claim 16, further comprising:

18. The training of the ML model comprises: running a simulation on training data using said ML model; if the ML model does not achieve convergence as defined by the reward function based on one or more criteria, requesting human assistance; monitoring actions taken by a human on a computing system; and modifying the policy network for the ML model, the reward function, or both based on the action taken by the human.

Images