Human-in-the-loop robot training for robotic process automation

Abstract

Human-in-the-loop robot training using artificial intelligence (AI) for robotic process automation (RPA) is disclosed. This can be accomplished by a listener robot that monitors the interaction of a user or another robot with a computing system. Based on the user's or robot's interaction with the computing system, the robot can be improved and / or personalized for a user or group of users.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Nonprovisional Patent Application No. 16 / 708,036, filed December 9, 2019, and U.S. Provisional Patent Application No. 62 / 915,429, filed October 15, 2019. The subject matter of these earlier-filed applications is incorporated herein by reference in its entirety.

[0002] The present invention relates generally to Robotic Process Automation (RPA), and more specifically to human-in-the-loop robot training for RPA. [Background technology]

[0003] RPA robots may be deployed to assist users in accomplishing a variety of tasks. Such robots may be designed based on expected general functional needs. However, robots may not always match the needs of a particular user. Therefore, improved techniques for training and personalizing RPA robots may be beneficial. Summary of the Invention

[0004] Certain embodiments of the present invention may provide solutions to problems and needs in the art that have not yet been fully identified, recognized, or solved by current robot training technology. For example, some embodiments of the present invention relate to human-in-the-loop robot training for RPA.

[0005] In one embodiment, the system includes a user computing system including an RPA robot and a listener. The system also includes a server. The listener is configured to monitor user interactions with the RPA robot via the user computing system and log data related to the interactions. The listener is also configured to transmit the logged data related to the user interactions to the server. The server is configured to receive the logged data related to the user interactions and determine whether an improvement should be made to the RPA workflow of the RPA robot based on the logged data. If the server determines that an improvement should be made, and if the improvement can be addressed by inserting an activity or a sequence of activities into the RPA workflow of the RPA robot, the server is configured to insert an activity or a sequence of activities into the RPA workflow of the RPA robot that makes the determined improvement.

[0006] In another embodiment, a computer program is stored on a non-transitory computer readable medium. The program is configured to cause at least one processor to monitor user interactions with the RPA robot via a user computing system and log data associated with the interactions, the logged data including exceptions. The program is also configured to cause the at least one processor to transmit the logged data associated with the user interactions to a server, receive from the server a new version of the RPA robot improved to address the exceptions in the logged data, and deploy the new version of the RPA robot.

[0007] In yet another embodiment, a computer-implemented method includes receiving, by a computing system, logged data related to an interaction between a user and an RPA robot, and determining, by the computing system, whether an improvement should be made to an RPA workflow of the RPA robot based on the logged data. If the computing system determines that an improvement should be made, and if the improvement can be addressed by inserting an activity or sequence of activities into the RPA workflow of the RPA robot, the computer-implemented method also includes inserting, by the computing system, an activity or sequence of activities into the RPA workflow of the RPA robot that makes the determined improvement. [Brief description of the drawings]

[0008] So that the advantages of certain embodiments of the invention may be readily understood, a particular description of the invention, briefly described above, will be provided by reference to specific embodiments that are illustrated in the accompanying drawings, it being understood that these drawings depict only typical embodiments of the invention and therefore should not be considered as limiting its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0009] [Figure 1] FIG. 1 is an architectural diagram illustrating an RPA system according to one embodiment of the present invention.

[0010] [Diagram 2] FIG. 1 is an architectural diagram illustrating a deployed RPA system according to one embodiment of the present invention.

[0011] [Diagram 3] FIG. 2 is an architecture diagram illustrating the relationships between designers, activities, and drivers according to one embodiment of the present invention.

[0012] [Figure 4] FIG. 1 is an architectural diagram illustrating an RPA system according to one embodiment of the present invention.

[0013] [Diagram 5] FIG. 1 is an architectural diagram illustrating a computing system configured to train a robot in an RPA environment, according to one embodiment of the present invention.

[0014] [Figure 6] FIG. 1 is an architectural diagram illustrating a system configured to perform human-in-the-loop robot training using AI for RPA, according to one embodiment of the present invention.

[0015] [Figure 7] FIG. 1 is a flowchart illustrating a process for human-in-the-loop robot training for RPA, according to one embodiment of the present invention.

[0016] Unless otherwise indicated, like reference numerals denote corresponding features consistently throughout the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Some embodiments of the present invention relate to human-in-the-loop robot training for RPA. This may be embodied in some embodiments by a listener robot that monitors the interaction of users with a computing system. The computing system may be a personal computer, a mobile device, a server (e.g., where the RPA robot assists in the operation and maintenance of the server), or any other suitable computing system without departing from the scope of the present invention. For example, consider a case where an employee or a group of employees achieves superior performance (e.g., converting a large number of unconventional leads and generating large sales). The listener robot sends the logged data to the server, which may be stored in a database. When one user's actions are analyzed, a workflow may be generated by analyzing and copying the user's actions, and then a robot that mimics these actions may be generated from the workflow and deployed to other users' computing systems to improve performance.

[0018] When a group of users or robots is being monitored, data from each user or robot may be sent to a server and analyzed using AI to recognize patterns in user behavior processes therein and develop a machine learning (ML) model, which may then be generated and deployed to invoke the RPA workflow.

[0019] In certain embodiments, the listener may monitor the user's interactions with the robot and may generate a new version of the robot that is more personalized for the user based on the user's interactions with the robot. If user-specific, the robot's workflow may be refined to make changes specific to that user. For example, if a user instructs the attended robot to generate a specific email, but then the user manually adds a specific closing statement to the email or changes the font, the system may learn that the user typically does this and create one or more activities in the RPA workflow that make this change automatically. The new local version of the robot may be deployed without the user actively making changes, and the user may then notice that the robot has started to perform these actions automatically.

[0020] Some robots may perform actions based on a user's voice commands or text inputs to achieve human-in-the-loop functionality. However, the initial globally deployed version of the robot may not work well for all users. For example, if the user commanding the robot has an accent, a particular global ML model may not be able to properly recognize the user's speech. However, a local ML model may be trained to monitor and effectively respond to that particular user's interactions with the robot, for example, by learning to understand words spoken in the user's accent based on modifications made to the text by the user. Thus, the local model may be trained by learning from those interactions to personalize the robot for the user.

[0021] Generally, a global ML model is trained using data from a broad pool of users. Consider the case of document processing: a global ML model can be trained on all enterprise documents or a subset of them.

[0022] However, in some cases, a particular person or group of people may be processing certain types of documents that are not effectively processed by an ML model. These documents may be in a particular language, may have different data entities extracted, or may use different workflows. In such cases, a local ML model may be trained to learn and process these particular document types.

[0023] In some embodiments, exceptions may be collected for retraining while the global ML model is in use. If the exceptions tend to be specific to a given user or group of users, a local ML model may be initially created and trained (or retrained) based on these exceptions. However, if the exceptions tend to be experienced by a large number of users, the global ML may also be retrained based on these exceptions.

[0024] Consider the example of a focused inbox. There may be keywords that can be added for the purpose of detecting spam in a global ML model. However, for a particular enterprise, emails that have keywords that indicate spam in the global ML context may not actually be spam for that enterprise or individual. A listener deployed on a user computing system may provide data that indicates that emails that are placed in the spam folder for this reason are frequently moved to the inbox by one or more users. A local ML model may then be trained to not mark such emails as spam for that particular user or group of users.

[0025] In some embodiments, the local ML model may learn certain behaviors of a user, such as user interactions with various applications, user steps taken while using the applications, and so on, thereby automating those steps. The local ML model may be applied first, and if no results are obtained, the global model may be applied. In this manner, the global ML model may be used as a fallback when the local ML model fails to provide results.

[0026] In some embodiments, the robot may review analytics. For example, the robot may observe how many calls a salesperson makes to get results. The robot may search for top employees, observe their patterns, and collect this information for training. The robot may also observe keywords, analyze emotions (if video is filmed and / or audio is captured, certain facial expressions, speech patterns, and volume may indicate emotion), etc. These analytics may help the robot select a suitable application. For example, for sales leads, if the user is only taking leads from one system and not another, the robot may choose to focus on lead conversion software that works well.

[0027] Some embodiments may include behavioral aspects. In these embodiments, if effective expertise is quantified (e.g., using specific keywords), the robot may automatically take steps or make recommendations. Thus, the robot may learn to become an intelligent guide tool for the user.

[0028] FIG. 1 is an architecture diagram illustrating an RPA system 100, according to one embodiment of the present invention. The RPA system 100 includes a designer 110 that may enable developers to design and implement workflows. The designer 110 may provide solutions for application integration, as well as automating third-party applications, managing information technology (IT) tasks, and business IT processes. The designer 110 may facilitate the development of automation projects, which are graphical representations of business processes. In simple terms, the designer 110 facilitates the development and deployment of workflows and robots.

[0029] Automation projects enable rule-based process automation by giving developers control over the order of execution and relationships between custom sets of steps developed in a workflow, defined herein as "activities." One commercial example of an embodiment of the designer 110 is UiPath Studio™. Each activity may include an action, such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0030] Some types of workflows may include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suitable for linear processes that allow flow from one activity to another without disrupting the workflow. Flowcharts are particularly suitable for complex business logic, allowing for the integration of decisions and the connection of activities in diverse ways through multiple branching logical operators. FSMs may be particularly suitable for large workflows. FSMs may use a finite number of states in the execution of a workflow that are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suitable for determining the behavior of a workflow when an execution error is encountered and for debugging the process.

[0031] Once a workflow is developed in Designer 110, the execution of the business process is orchestrated by Conductor 120, which orchestrates one or more Robots 130 that execute the workflow developed in Designer 110. One commercially available example of an embodiment of Conductor 120 is UiPath Orchestrator™. Conductor 120 facilitates the creation, monitoring, and management of deployment of resources in the environment. Conductor 120 can act as an integration point with third party solutions and applications.

[0032] The conductor 120 may manage all the robots 130, connecting and executing them from a central point. Types of robots 130 that may be managed include, but are not limited to, attended robots 132, unattended robots 134, development robots (similar to unattended robots 134 but used for development and testing purposes), and non-production robots (similar to attended robots 132 but used for development and testing purposes). Attended robots 132 are triggered by user events and operate alongside humans on the same computing system. Attended robots 132 may be used with the conductor 120 for centralized process deployment and logging. Attended robots 132 help human users accomplish various tasks and may be triggered by user events. In some embodiments, processes cannot be started from the conductor 120 for this type of robot and / or cannot be run under a locked screen. In certain embodiments, attended robots 132 may only be launched from the robot tray or command prompt. In some embodiments, the attended robot 132 should perform under human supervision.

[0033] Unattended Robots 134 may run unattended in a virtual environment and automate many processes. Unattended Robots 134 may be responsible for providing remote execution, monitoring, scheduling, and work queue support. In some embodiments, debugging of all robot types may be performed in Designer 110. Both attended and unattended robots may automate a variety of systems and applications, including, but not limited to, mainframes, web applications, VMs, enterprise applications (e.g., those manufactured by SAP®, SalesForce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0034] The conductor 120 may have a variety of functions, including, but not limited to, provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between the robots 130 and the conductor 120 (e.g., web applications). Deployment may include ensuring correct delivery of package versions to assigned robots 130 for execution. Configuration may include maintaining and delivering robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include keeping track of robot identification data and maintaining user permissions. Logging may include storing and indexing logs in a database (e.g., an SQL database) and / or another storage mechanism (e.g., ElasticSearch®, which provides the ability to store and quickly query large data sets). The conductor 120 may provide interconnectivity by acting as a centralized point of communication for third-party solutions and / or applications.

[0035] The robot 130 is an execution agent that executes the workflows built in the designer 110. One commercial example of some embodiments of the robot 130 is UiPath Robots™. In some embodiments, the robot 130 installs the Microsoft Windows Service Control Manager (SCM) management service by default. As a result, such a robot 130 may open an interactive Windows session under the local system account and have Windows service rights.

[0036] In some embodiments, a robot 130 may be installed in user mode. For such a robot 130, this means that a given robot 130 has the same rights as the user for whom it is installed. This feature may also be available for high density (HD) robots, ensuring full utilization of each machine at its maximum potential. In some embodiments, any type of robot 130 may be configured in an HD environment.

[0037] The Robot 130 in some embodiments is divided into several components, each of which is dedicated to a particular automation task. The Robot components in some embodiments include, but are not limited to, the SCM managed Robot service, the user mode Robot service, the executor, the agent, and the command line. The SCM managed Robot service manages and monitors the Windows session and acts as a proxy between the conductor 120 and the execution host (i.e., the computing system on which the Robot 130 executes). These services are trusted with and manage the credentials of the Robot 130. The console application is launched by the SCM under the local system.

[0038] The user-mode robot service in some embodiments manages and monitors the Windows session and acts as a proxy between the conductor 120 and the execution host. The user-mode robot service may be trusted with and manage credentials for the robot 130. If the SCM Management Robot Service is not installed, the Windows application may be launched automatically.

[0039] An Executor may run a given job under a Windows session (i.e., an Executor may execute a workflow). An Executor may be aware of per-monitor dots per inch (DPI) settings. An Agent may be a Windows Presentation Foundation (WPF) application that displays available jobs in a system tray window. An Agent may be a client of a service. An Agent may request to start or stop a job and change settings. A Command Line is a client of a service. A Command Line is a console application that can request to start a job and wait for its output.

[0040] Splitting up the components of the robot 130 as described above helps developers, support users, and computing systems to easily implement, identify, and track what each component is doing. In this way, special behaviors can be configured per component, such as setting different firewall rules for executors and services. The executors, in some embodiments, can always be aware of the DPI setting per monitor. As a result, workflows can be executed at any DPI regardless of the configuration of the computing system on which they were created. In some embodiments, projects from the designer 110 can be independent of the browser zoom level. For applications that are not DPI aware or are intentionally marked as not aware, in some embodiments, DPI can be disabled.

[0041] FIG. 2 is an architecture diagram illustrating a deployed RPA system 200 according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be or may be part of the RPA system 100 of FIG. 1. It should be noted that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 210 includes an executor 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not be running on the computing system 210. The executor 212 is a running process. As shown in FIG. 2, several business projects may be running simultaneously. The agent 214 (e.g., a Windows service) is a single connection point for all the executors 212 in this embodiment. All messages in this embodiment are logged to the conductor 230, which further processes what is logged via the database server 240, the indexer server 250, or both. As described above with respect to Figure 1, the executor 212 may be a robotic component.

[0042] In some embodiments, a robot represents an association between a machine name and a username. A robot may manage multiple executors simultaneously. In computing systems that support multiple interactive sessions running simultaneously (e.g., Windows Server 2012), multiple robots may run simultaneously, each running in a separate Windows session using a unique username. This is referred to above as an HD Robot.

[0043] The agent 214 is also responsible for transmitting the robot's status (e.g., periodically sending "heartbeat" messages to indicate that the robot is still functioning) and downloading necessary versions of packages to be fulfilled. Communication between the agent 214 and the conductor 230 is, in some embodiments, always initiated by the agent 214. In a notification scenario, the agent 214 may open a WebSocket channel that is later used by the conductor 230 to send commands to the robot (e.g., start, stop, etc.).

[0044] The server side includes a presentation layer (web application 232, Open Data Protocol (OData) Representational State Transfer (REST) ​​Application Programming Interface (API) endpoint 234, and notification monitor 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240, indexer server 250). Conductor 230 includes web application 232, OData REST API endpoint 234, notification monitor 236, and API implementation / business logic 238. In some embodiments, most actions that a user performs within the conductor 220 interface (e.g., via browser 220) are performed by calling various APIs. Such actions may include, but are not limited to, starting a job on a robot, adding / removing data in a queue, scheduling a job for unattended execution, etc., without departing from the scope of the present invention. Web application 232 is the visual layer of the server platform. In this embodiment, web application 232 uses HyperText Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the invention. A user interacts with web pages from web application 232, in this embodiment, via browser 220, to perform various actions that control conductor 230. For example, a user may create robot groups, assign packages to robots, analyze logs per robot and / or per process, start and stop robots, etc.

[0045] In addition to the web application 232, the conductor 230 also includes a services layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the invention. The REST API is consumed by both the web application 232 and the agent 214, which in this embodiment is the administrator of one or more robots on a client computer.

[0046] The REST API in this embodiment covers configuration, logging, monitoring, and queuing functions. The configuration endpoint may be used in some embodiments to define and configure application users, permissions, robots, assets, releases, and environments. The logging REST endpoint may be used to log various information, such as errors, explicit messages sent by the robot, and other environment specific information. The deployment REST endpoint may be used by the robot to query the package version that should be committed when the start job command is used in the conductor 230. The queuing REST endpoint may be responsible for queue and queue item management, such as adding data to the queue, getting transactions from the queue, setting the status of transactions, etc.

[0047] Monitoring REST endpoints may monitor the web application 232 and the agents 214. The notification monitoring API 236 may be a REST endpoint used to register the agents 214, deliver configuration settings to the agents 214, and send / receive notifications from the server and the agents 214. The notification monitoring API 236 may also use WebSocket communication in some embodiments.

[0048] The persistence layer includes a pair of servers in this embodiment: database server 240 (e.g., SQL server) and indexer server 250. Database server 240 in this embodiment stores configurations of robots, robot groups, associated processes, users, roles, schedules, etc. This information is managed in some embodiments via web application 232. Database server 240 may manage queues and queue items. In some embodiments, database server 240 may store messages logged by robots (in addition to or instead of indexer server 250).

[0049] The indexer server 250 is optional in some embodiments and stores and indexes information logged by the robots. In certain embodiments, the indexer server 250 may be disabled via a configuration setting. In some embodiments, the indexer server 250 uses ElasticSearch, a full-text search engine from an open source project. Messages logged by the robots (e.g., using activities such as log messages or line writes) may be sent via a logging REST endpoint to the indexer server 250 where they are indexed for future use.

[0050] FIG. 3 is an architecture diagram illustrating the relationship 300 between a designer 310, activities 320, 330, and a driver 340, according to one embodiment of the present invention. In accordance with the above, a developer uses the designer 310 to develop a workflow to be performed by the robot. The workflow may include user-defined activities 320 and UI automation activities 330. Some embodiments may identify non-textual visual components within an image, referred to herein as computer vision (CV). Some CV activities related to such components may include, but are not limited to, click, type, get text, hover, element presence, refresh range, highlight, and the like. In some embodiments, click may identify an element and click on it, for example, using CV, optical character recognition (OCR), fuzzy character matching, and multi-anchors. Type may identify an element using the above and type within an element. Get text may identify the location of a particular text using OCR and scan it. Hover may identify an element and hover it. Element presence may check if an element is present on the screen using the techniques described above. In some embodiments, there may be hundreds or thousands of activities that may be implemented in designer 310. However, any number and / or type of activities may be utilized without departing from the scope of the invention.

[0051] UI automation activities 330 are a subset of specialized lower level activities that are written into lower level code (e.g., CV activities) to facilitate interactions with the screen. UI automation activities 330 facilitate these interactions through drivers 340 that allow the robot to interact with desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc.

[0052] Drivers 340 may interact with the OS at a low level, looking for hooks, monitoring keys, etc. They may facilitate integration with Chrome®, IE®, Citrix®, SAP®, etc. For example, a "click" activity performs the same role in these different applications via drivers 340.

[0053] FIG. 4 is an architecture diagram illustrating an RPA system 400, according to one embodiment of the present invention. In some embodiments, the RPA system 400 may be or include the RPA systems 100 and / or 200 of FIG. 1 and / or FIG. 2. The RPA system 400 includes multiple client computing systems 410 that execute robots. The computing systems 410 may communicate with a conductor computing system 420 via web applications executed thereon. The conductor computing system 420 may in turn communicate with a database server 430 and an optional indexer server 440.

[0054] 1 and 3, it should be noted that although web applications are used in these embodiments, any suitable client / server software may be used without departing from the scope of the present invention. For example, the conductor may execute a server-side application that communicates with a non-web-based client software application on a client computing system.

[0055] FIG. 5 is an architecture diagram illustrating a computing system 500 configured to perform human-in-the-loop robot training using AI for RPA, according to an embodiment of the present invention. In some embodiments, the computing system 500 may be one or more of the computing systems illustrated and / or described herein. The computing system 500 includes a bus 505 or other communication mechanism for communicating information and a processor 510 coupled to the bus 505 for processing information. The processor 510 may be any type of general-purpose or special-purpose processor, including a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor 510 may also have multiple processing cores, and at least some of the cores may be configured to perform a particular function. In some embodiments, multiple parallel processing may be used. In certain embodiments, at least one of the processors 510 may be a neuromorphic circuit that includes processing elements that mimic biological neurons. In some embodiments, neuromorphic circuitry may not require typical components of a von Neumann computing architecture.

[0056] The computing system 500 further includes a memory 515 that stores information and instructions executed by the processor 510. The memory 515 may be comprised of any combination of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage such as magnetic or optical disks, or any other type of non-transitory computer-readable medium, or any combination thereof. The non-transitory computer-readable medium may be any available medium that can be accessed by the processor 510 and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both.

[0057] Additionally, the computing system 500 includes a communications device 520, such as a transceiver for providing access to a communications network via wireless and / or wired connections. In some embodiments, the communications device 520 may be a wireless LAN adapter, such as a transceiver for providing access to a communications network via wireless and / or wired connections, such as a transceiver for a wireless network, such as a wireless LAN adapter, ... 8.x, 802.15, Home Node B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near Field Communication (NFC), Fifth Generation (5G), National Radio (NR), any combination thereof, and / or any other currently existing or future implemented communications standard and / or protocol without departing from the scope of the present invention. In some embodiments, communication device 520 may include one or more antennas, which may be singular, array, phased, switched, beamforming, beamsteered, combinations thereof, and / or any other antenna configuration without departing from the scope of the present invention.

[0058] The processor 510 is further coupled via bus 505 to a display 525, which may be a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, a Retina® display, an in-plane switching (IPS) display, or any other suitable display for displaying information to a user. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, or the like, using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal techniques, acoustic pulse recognition, frustrated total internal reflection, or the like. Any suitable display device and haptic I / O may be used without departing from the scope of the invention.

[0059] A keyboard 530 and cursor control device 535, such as a computer mouse, touchpad, etc., are further coupled to bus 505 to allow a user to interface with the computing system. However, in certain embodiments, a physical keyboard and mouse may not be present and the user may interact with the device solely through the display 525 and / or touchpad (not shown). Any type and combination of input devices may be used as a matter of design choice. In certain embodiments, there are no physical input devices and / or displays. For example, a user may interact with computing system 500 remotely via another computing system that communicates with computing system 500, or computing system 500 may operate autonomously.

[0060] The memory 515 stores software modules that, when executed by the processor 510, provide functionality. The modules include an operating system 540 for the computing system 500. The modules further include a training module 545 configured to perform all or a portion of the processes described herein or derivatives thereof. The computing system 500 may include one or more additional functional modules 550 that include additional functionality.

[0061] Those skilled in the art will appreciate that the "system" may be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device or combination of devices without departing from the scope of the present invention. Presenting the above functions as being performed by a "system" is in no way intended to limit the scope of the present invention, but is intended to provide an example of one example of many embodiments of the present invention. Indeed, the methods, systems, and apparatus disclosed herein may be implemented in localized and distributed forms consistent with computing technologies, including cloud computing systems.

[0062] It should be noted that some of the system features described herein have been presented as modules to more specifically emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components, etc. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, etc.

[0063] The modules may also be implemented at least in part in software for execution by various types of processors. An identified unit of executable code may include one or more physical or logical blocks of computer instructions, which may be organized as, for example, an object, procedure, or function. Nevertheless, executables of identified modules need not be physically located together, but may include different instructions stored in different locations that, when logically coupled together, accomplish the stated purpose of the modules, including the modules. Furthermore, the modules may be stored on a computer readable medium, which may be, for example, a hard disk drive, a flash device, RAM, tape, and / or any other such non-transitory computer readable medium used to store data without departing from the scope of the present invention.

[0064] Indeed, the executable code of a module may be a single instruction, or many instructions, and may be distributed across several different code segments, among different programs, and among several memory devices. Similarly, operational data, which is identified and illustrated herein in modules, may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set or distributed across different locations, including different storage devices, and may exist, at least in part, only as electronic signals on a system or network.

[0065] 6 is an architecture diagram illustrating a system 600 configured to perform human-in-the-loop robot training using AI for RPA, according to one embodiment of the present invention. System 600 includes user computing systems such as a desktop computer 602, a tablet 604, and a smartphone 606. However, any desired computing system may be used without departing from the scope of the present invention, including, but not limited to, a smart watch, a laptop computer, an Internet of Things (IoT) device, a vehicle computing system, and the like.

[0066] Each computing system 602, 604, 606 has a listener 610 installed thereon. The listener 610 may be a robot generated via an RPA designer application, part of an operating system, a downloadable application for a personal computer (PC) or smartphone, or any other software and / or hardware without departing from the scope of the present invention. Indeed, in some embodiments, the logic of one or more of the listeners 610 is implemented partially or completely via physical hardware.

[0067] The listener 610 generates logs of user interactions with the robot on each computing system 602, 604, 606. The listener 610 then transmits the log data to the server 630 via a network 620 (e.g., a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, any combination thereof, etc.). In some embodiments, the server 630 may run a conductor application and the data may be transmitted periodically as part of a heartbeat message. In certain embodiments, the log data may be transmitted to the server 630 after a predetermined amount of log data has been collected, after a predetermined period of time has elapsed, or both. The server 630 stores the log data received from the listener 610 in a database 640.

[0068] When instructed by a human user (e.g., an RPA engineer or data scientist), when a predefined amount of log data has been collected, when a predefined amount of time has passed since the last analysis, etc., the server 630 accesses the log data collected by the listener 610 from the database 640 and runs the log data through multiple AI layers 632. The AI ​​layers 632 process the log data and train local ML models based on the user's interactions with the robots. The server 630 can then automatically generate workflows that invoke the trained local ML models, generate robots (or replacement robots) that implement the workflows, and push the generated robots to the user computing systems 602, 604, 606 to be executed.

[0069] Alternatively, in certain embodiments, the proposed process from the AI ​​layer 632 may be presented to an RPA engineer via a designer application 652 on the computing system 650. The RPA engineer may then review the workflow, make any desired changes, and then deploy the workflow via the robot to the computing systems 602, 604, 606, or have the robot deployed. For example, the deployment may be done via a conductor application running on the server 630 or another server, which may push a robot implementing the process to the user computing systems 602, 604, 606. In some embodiments, this workflow deployment may be accomplished via an automation manager function of the designer application, and the RPA engineer may simply click a button to implement the process on the robot.

[0070] listener

[0071] A listener 610 may be employed client-side at the driver level (e.g., driver 340 in FIG. 3) to extract data from whitelisted applications to extract data regarding actions taken by users on the computing systems 602, 604, 606 when interacting with the robot. For example, the listener 610 may record where on the screen the user clicked and in which application, keystrokes of the buttons clicked, instances of user switching between applications, changes of focus, emails sent and what the emails are related to, etc. Additionally or alternatively, the listener 610 may collect data regarding robots operating on the computing systems 602, 604, 606. In some embodiments, robots implementing workflows, performing various tasks, may act as listeners for their own operations. Such data may be used to generate high fidelity logs of user interactions with the computing systems 602, 604, 606 and / or the operations of robots executing thereon.

[0072] In addition to or as an alternative to generating log data for process extraction, some embodiments may provide insight into what a user is actually doing. For example, the listener 610 may determine when a user actually used an application, the percentage of time the user was using a given application, which features within the application the user used and which features the user did not use, etc. This information may be provided to managers to make informed decisions such as whether to renew an application license, whether to not renew a feature license or downgrade to a less expensive version that is missing features, whether the user may be properly trained since the user is not using an application that other employees often find productive, whether the user is spending a lot of time performing non-work activities (e.g., checking personal email or surfing the web) or whether the user is away from his desk (e.g., not interacting with the computing system), etc.

[0073] In some embodiments, detection updates may be pushed to the listeners to improve the driver-level user interaction and / or robot motion detection and capture process. In certain embodiments, the listeners 610 may employ AI in their detection. In certain embodiments, robots implementing processes from the automation workflow may be automatically pushed to the computing systems 602, 604, 606 via their respective listeners 610.

[0074] FIG. 7 is a flow chart illustrating a process 700 for implementing human-in-the-loop robot training using AI for RPA, according to one embodiment of the present invention. The process begins at 705 with monitoring interactions between a user of a computing system and an RPA robot by a listener. The listener may be a separate listener robot, an RPA robot with which the user is interacting, another listener software application, etc. At 710, data related to user interactions is logged over time (e.g., stored in a log file) and transmitted to a server.

[0075] The logged data may include exceptions noted by the user, such as when the robot cannot find the license plate in the image, and the robot may ask the user to indicate where in the image the license plate is. This may be provided by the user via bounding box, text, coordinates, on-screen location, etc. Exceptions may also be user corrections made without the RPA robot requesting the correction. These corrections may be due to errors and / or user settings.

[0076] The server then determines at 715 that improvements should be made based on the exceptions noted by the users. This may be determined based on a predetermined number of exceptions of the same type that have been received by one or more users due to the passage of a predetermined amount of time (e.g., a day, a week, a month, etc.), based on exception frequency, any combination thereof (e.g., 50 exceptions of the same type are received in a short period of time, providing at least 75% frequency for the same task, run every week), etc. For example, the server may make this determination by analyzing logged data and noting that users often make a particular change above a predetermined threshold (e.g., at least 50% of the time).

[0077] At 720, the server first determines whether the user is capable of improvement through a particular activity or sequence of activities. For example, the server may implement the correction by including an activity that looks up a particular value (e.g., look up a field in an invoice and add a particular number) or by including a sequence of activities (e.g., look up a template email body text from a database and insert a particular text into the email that the user would like). In other words, the improvement may be made by mimicking the user's actions in the exception. If this is the case, at 725, an improvement activity or sequence of activities is inserted into the workflow. In some embodiments, this may include replacing one or more existing activities and / or improving the logic of the workflow itself. Then, at 730, a robot implementing the workflow is generated and deployed to the user's computing system.

[0078] In certain embodiments, the user's own computing system may perform steps 715, 720, 725 and 730. For example, if the user has made certain changes for personalization, this may not require the computing power of a server to analyze and improve the robot. Thus, the user's computing system itself may determine whether improvements should be made, improve the workflow, generate a new version of the local robot, and deploy the new version of the robot in place of the previous version.

[0079] However, in some cases, the refinement is too complex to add / refine only workflow activities. For example, an ML model to recognize speech may need to be trained first, or an existing ML model may need to be retrained to recognize the user's accent. In this case, at 735, the server trains an ML model based on the logged data using machine learning techniques (e.g., deep learning neural networks) to create a local ML model (or a new version thereof). At 740, the workflow is refined to invoke this local ML model, and then at 730, a robot implementing this workflow is generated and deployed.

[0080] In some embodiments, the process of FIG. 7 may be performed for many users, organization-wide and / or globally. When logged data is collected from multiple, many, or all users, certain patterns may emerge. For example, analysis may reveal that a group of users in a particular location have similar accents and notice similar exceptions, that a significant number of all users modify a particular field on an invoice, that a significant number of all users have a particular setting, etc. In some embodiments, this may be determined by a predefined threshold (e.g., a predefined number or percentage of users with exceptions, a predefined frequency of exceptions, the number of times an exception occurs over a period of time, etc.). If this is the case at 745, then at 750 a "community" ML model for the group of users or company, or a global ML model for all users, may be trained using this logged data. Then, at 740, an appropriate workflow is refined, and at 730, a robot implementing this workflow is generated and deployed.

[0081] The frequency of retraining may be based on the amount of logged data received. A large group of users may generate enough data to retrain the ML model frequently (e.g., daily). However, a small group of users may take a lot of time to generate enough logged data to be useful for retraining purposes.

[0082] The process steps performed in FIG. 7 may be implemented by a computer program encoding instructions for a processor to perform at least a portion of the process described in FIG. 7 according to an embodiment of the present invention. The computer program may be stored on a non-transitory computer readable medium. The computer readable medium may be, but is not limited to, a hard disk drive, a flash device, RAM, tape, and / or any other such medium or combination of media used to store data. The computer program may include encoded instructions for controlling a processor of a computing system (e.g., processor 510 of computing system 500 of FIG. 5) to implement all or a portion of the process steps described in FIG. 7, which may be stored on a computer readable medium.

[0083] The computer program may be implemented in hardware, software, or a hybrid implementation. The computer program may be composed of modules designed to operatively communicate with each other and pass information or instructions for display. The computer program may be configured to run on a general purpose computer, an ASIC, or any other suitable device.

[0084] It will be readily understood that the components of the various embodiments of the present invention, as generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the detailed description of the embodiments of the present invention, as represented in the accompanying drawings, is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention.

[0085] The features, structures, or characteristics of the invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, references throughout this specification to "certain embodiments," "some embodiments," or similar language means that the particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Thus, the appearances of "certain embodiments," "some embodiments," "other embodiments," or similar language throughout this specification do not necessarily all refer to the same group of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0086] It should be noted that references to features, advantages, or similar language throughout this specification do not imply that all of the features and advantages that may be realized in the present invention should be in any single embodiment of the present invention, or that any single embodiment of the present invention is. Rather, language referring to features and advantages is understood to mean that the particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, descriptions of features and advantages and similar language throughout this specification may, but do not necessarily, refer to the same embodiment.

[0087] Furthermore, the described features, advantages, and characteristics of the invention may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the invention may be practiced without one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments of the invention.

[0088] Those skilled in the art will readily appreciate that the invention as described above may be implemented with steps in a different order and / or with hardware elements in a different configuration than that disclosed. Thus, while the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain improvements, modifications, and alternative constructions will be apparent while remaining within the spirit and scope of the invention. Accordingly, reference should be made to the appended claims to determine the scope of the invention.

Claims

1. A user computing system comprising a robotic process automation (RPA) robot and a listener; A server; A system comprising: The listener, monitoring user interactions with the RPA robot via the user computing system and logging data relating to the user interactions; and transmitting the logged data relating to the user interactions to the server; The server, receiving the logged data relating to the user interactions; Determining whether an improvement should be made to the RPA workflow of the RPA robot based on the logged data; and If the server determines that the improvement should be made, If the improvement can be addressed by inserting an activity or a sequence of activities into the RPA workflow of the RPA robot, inserting the activity or the sequence of activities into the RPA workflow of the RPA robot that makes the determined improvement. system.

2. The server, generating a new version of the RPA robot using the improved RPA workflow; and and deploying the new version of the RPA robot to the user computing system. The system of claim 1 .

3. said user computing system comprising: receiving a new version of the RPA robot from the server; and deploying the new version of the RPA robot; The system of claim 1 .

4. 2. The system of claim 1, wherein the logged data includes exceptions noted by the user via the user computing system during operation of the RPA robot.

5. 5. The system of claim 4, wherein the exception relates to an error by the RPA robot, a user setting, or both.

6. 2. The system of claim 1, wherein the determination of whether to make an improvement is based on a predetermined number of exceptions of the same type being received due to the passage of a predetermined amount of time, based on a frequency of exceptions, or any combination thereof.

7. If the improvement is not addressable by inserting the activity or sequence of activities into the RPA workflow, the server: training a local machine learning (ML) model based on the logged data; and and refining the RPA workflow to invoke the trained ML model. The system of claim 1 .

8. The server, Collecting logged data relating to interactions between other users of other computing systems and the respective RPA robots; if an exception for the user is similar to an exception in the collected logged data for a group of other users that is a subset of all of the other users; training a community ML model for the subset of users; Refine the RPA workflow to invoke the community ML model; and if an exception for the user is similar to an exception in the collected logged data for the group of other users and exceeds a global retraining threshold; Train a global ML model for all users, and modifying the RPA workflow to invoke the global ML model. The system of claim 1 .

9. The system of claim 1 , wherein the logged data is sent by the listener to the server as part of a heartbeat message to a conductor application running on the server.

10. A computer program stored on a non-transitory computer readable medium, the program comprising: monitoring user interactions with a robotic process automation (RPA) robot via a user computing system and logging data relating to said user interactions and including exceptions; transmitting the logged data relating to the user interactions to a server; receiving from the server a new version of the RPA robot that is improved to address the exceptions in the logged data; and deploying the new version of the RPA robot; Computer program.

11. 11. The computer program product of claim 10, wherein the exception relates to an error by the RPA robot, a user setting, or both.

12. The computer program product of claim 10 , wherein the logged data is sent to the server as part of a heartbeat message to a conductor application running on the server.

13. 11. The computer program product of claim 10, wherein the logged data is transmitted to the server after a predetermined amount of data has been collected, after a predetermined period of time has elapsed, or both.

14. receiving, by a computing system, logged data relating to an interaction between a user and a robotic process automation (RPA) robot; determining, by the computing system, whether improvements should be made to the RPA workflow of the RPA robot based on the logged data; If the computing system determines that the improvement should be made, If the improvement can be addressed by inserting an activity or sequence of activities into the RPA workflow of the RPA robot, inserting, by the computing system, the activity or sequence of activities into the RPA workflow of the RPA robot that makes the determined improvement. Computer-implemented method.

15. generating, by the computing system, a new version of the RPA robot using the improved RPA workflow; and deploying, by the computing system, the new version of the RPA robot.

15. The computer-implemented method of claim 14.

16. 15. The computer-implemented method of claim 14, wherein the logged data includes exceptions noted by the user during operation of the RPA robot.

17. 20. The computer-implemented method of claim 16, wherein the exception relates to an error by the RPA robot, a user setting, or both.

18. 15. The computer-implemented method of claim 14, wherein the determination of whether to make an improvement is based on a predetermined number of exceptions of the same type being received due to a predetermined amount of time, based on a frequency of exceptions, or any combination thereof.

19. If the improvement is not addressable by inserting the activity or sequence of activities into the RPA workflow, the method further comprises: training, by the computing system, a local machine learning (ML) model based on the logged data; and refining, by the computing system, the RPA workflow to invoke the trained ML model.

15. The computer-implemented method of claim 14.

20. collecting, by the computing system, logged data relating to interactions between other users and each RPA robot; if an exception for the user is similar to an exception in the collected logged data for a group of other users that is a subset of all of the other users; training, by the computing system, a community ML model for the subset of users; refining, by the computing system, the RPA workflow to invoke the community ML model; if an exception for the user is similar to an exception in the collected logged data for the group of other users and exceeds a global retraining threshold; training, by the computing system, a global ML model for all users; and modifying the RPA workflow to invoke the global ML model.

15. The computer-implemented method of claim 14.

Images