Intersection Automation of Robotic Process Automation (RPA) Robots

Abstract

Intersession automation for robotic process automation (RPA) is disclosed. A robot or another application or process running in a user session can interact with the application while one or more attended RPA robots in one or more child sessions perform actions and fetch data that the user session robot will then use to interact with the user session's application. The attended RPA robots in the client session can share data via an inter-process communication (IPC) protocol by storing the data in a persistent data store, such as a spreadsheet, an object-oriented database, a plain text file, another data store or file, etc. The user session robot, or another application or process running in the parent session, can read this information and respond accordingly.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Non - Provisional Application No. 15 / 930,906, filed on May 13, 2020, which is a Continuation - in - Part (CIP) of U.S. Non - Provisional Application No. 16 / 793,064, filed on Feb. 18, 2020, and claims the benefit thereof. The subject matter of these previously filed applications is hereby incorporated herein by reference in its entirety.

[0002] The present invention generally relates to robotic process automation (RPA), and more specifically, to the automation of interactions for RPA robots.

Background Art

[0003] Attended automation RPA robots typically execute on a computing system operated by a user, in the same session as that user. An attended RPA robot can cooperate with the user, for example, to accomplish a particular task in response to the user's command. However, in an attended automation scenario, the RPA robot may "take over" the user's computing system. The user may wish to perform other activities while the robot is interacting with the computing system, but the user cannot do so. In other words, the robot controls the application in the same way as the user does (e.g., by simulating mouse clicks and keyboard inputs) via the user interface (UI).

[0004] There are various technologies for creating a complete or partial copy of an operating system or an application running thereon. Emulators have existed for decades and can provide developers with the ability to test and debug applications. For example, an emulator can provide developers with the ability to test and debug mobile applications that use an operating system that does not directly support the execution of development tools. Since neither Android (registered trademark) nor iOS (registered trademark) can natively execute development tools on these mobile operating systems, they provide emulators that can be run from a development machine to test and debug Android (registered trademark) or iOS (registered trademark) applications.

[0005] Using a simulator, a developer can host a window on his or her local machine, thereby enabling the developer to test and debug the behavior of an application that is difficult or impossible to run on a development machine. For example, in a simulator, a developer can click a button to rotate the simulator, which notifies the application running within the simulator that the device has been rotated for the purpose of testing and debugging the application's behavior in response to these events. Another common example is multi-touch. Since many developer machines do not support touch, using a simulator allows developers to test and debug how an application responds to multiple touch points. Android (registered trademark) and iOS (registered trademark) emulators also provide simulation capabilities. Additionally, Microsoft (registered trademark) provides a simulator for Universal Windows Platform (UWP) applications.

[0006] A virtual machine hosts a second operating system on a machine and can be monitored through an opened window. It runs a completely different operating system and shares hardware with the host machine. The "guest" machine must have a unique copy of the application installed and does not share common resources or files with the user machine.

[0007] Docker (registered trademark) containers are conceptually a hybrid form of virtual machines. All applications that need to be executed are packaged into immutable packages that are directly executed on the host operating system. The package is not a complete copy of another operating system, but by default, it does not share or access any applications or resources on the host machine. Therefore, from the perspective of the user experience, Docker (registered trademark) containers seem similar to virtual machines, but technically, the containers are not executed on a completely different operating system.

[0008] However, traditional emulators, simulators, virtual machines (VMs), and hybrid VMs that provide operating system (OS) level virtualization (such as Docker (registered trademark) containers) do not address the problems that occur with attended automation robots operating on the same computing system as the user. Therefore, the user basically becomes a bystander of his or her computing system, can monitor the operation of the robot, and cannot interact with other applications on the machine that require user interaction. Therefore, an improved approach may be beneficial. Summary of the Invention

[0009] Certain embodiments of the present invention may provide solutions to problems and needs in the art that have not yet been fully identified, evaluated, or solved by current RPA technology. For example, some embodiments of the present invention relate to intersession automation for RPA robots.

[0010] In an embodiment, a computer-implemented method includes starting a user session process in a user session and starting a first client session RPA robot in a first client session. The computer-implemented method also includes performing, by the first client session RPA robot, a workflow of the first client session RPA robot in the first client session, and enabling, by the first client session RPA robot, the user session process to utilize a result of the performance of the workflow of the first client session RPA robot. The computer-implemented method further includes interacting, by the user session process, with an application in the user session using the result of the performance of the workflow of the first client session RPA robot.

[0011] In another embodiment, a computer-implemented method includes performing, by a client session RPA robot, a workflow of the client session RPA robot in a client session, and enabling, by the client session RPA robot, a user session RPA robot to utilize a result of the performance of the workflow of the client session RPA robot. The computer-implemented method also includes interacting, by the user session RPA robot, with an application on a computing system using the result of the performance of the workflow of the client session RPA robot.

[0012] In yet another embodiment, the computer-implemented method includes performing, by a client session RPA robot, a workflow of the client session RPA robot in a client session, and making, by the client session RPA robot, the result of the performance of the workflow of the client session RPA robot available to a user session RPA robot. The computer-implemented method also includes interacting, by the user session RPA robot, with an application on a computing system using the result of the performance of the workflow of the client session RPA robot. The user session RPA robot and the client session RPA robot collectively complete the performance of a larger workflow that includes the workflow of the client session RPA robot.

Brief Description of the Drawings

[0013] For a better understanding of the advantages of certain embodiments of the present invention, a more specific description of the invention briefly described above is presented with reference to the specific embodiments illustrated in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the invention and are not considered to limit its scope, but the invention will be described and explained in further detail and specificity by use of the following accompanying drawings.

[0014]

Figure 1

[0015]

Figure 2

[0016]

Figure 3

[0017]

Figure 4

[0018]

Figure 5

[0019]

Figure 6

[0020]

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 7E

Figure 7F

Figure 7G

Figure 7H

Figure 7I

Figure 7J

[0021]

Figure 8

[0022]

Figure 9

[0023]

Figure 10

[0024]

Figure 11

[0025] Unless otherwise specified, similar reference characters indicate corresponding features consistently throughout the accompanying drawings.

DETAILED DESCRIPTION OF THE INVENTION

[0026] (Detailed Description of the Embodiment) Some embodiments relate to attended RPA session automation. For example, a form (e.g., a web page, a spreadsheet, another application having an inputtable field, etc.) may be displayed in a user session (also referred to herein as the main session or parent session). A robot or another application or process executing in the user session can interact with the form, but one or more attended RPA robots in one or more child sessions (also referred to herein as client sessions, robot sessions, or second sessions) can perform operations and fetch data that the user session robot will then use to input into the form of the user session. An attended RPA robot in a client session can share data via an inter-process communication (IPC) protocol by storing the data in a persistent data store including but not limited to a spreadsheet, an object-oriented database, a plain text file, another data store or file, etc. A user session robot, or another application or process executing in the parent session, can read this information and respond accordingly (e.g., by inputting the information into a form). Thus, the operations of the attended automation robot(s) should not prevent the user from using other applications or instances while the client session RPA robot(s) is executing a workflow, but the input provided by the user session RPA robot is visualized to the user of the user session when the user session robot fills it into the application.

[0027] In some embodiments, the process may operate as follows. The user may click a button or otherwise initiate in a child session of the user session (e.g., a parent session or main session that is launched when the user boots his or her computing system) an RPA robot. In certain embodiments, the robot session may already be running or may be launched separately by the user from, for example, a robot tray. The robot tray in some embodiments may be a Windows® Presentation Foundation (WPF) application that displays jobs available in the system tray window. The user can have the robot execute its workflow in the child session, after which the robot can interact with one or more applications in the main session. The robot functionality may be initiated by the user clicking a button in the main session that invokes the robot in some embodiments. In certain embodiments, the user session robot may initiate a client session robot.

[0028] In some embodiments, if at least one application on which the workflow operates is not launched, the client session robot may stop operating and provide a message. In certain embodiments, the user session robot or other user session application or process may launch the application(s) if they are not open. If the operating system permits such an operation, the client session robot may launch the application in the parent session. When the client session robot makes data available for use by the user session robot, the data may be displayed in the main session when the user session robot inputs it.

[0029] In some embodiments, a user can cause a robot in a client session to perform a workflow, e.g., go to a website, collect some information, and make it available for input into a spreadsheet viewable in the main session by a user session robot or another application or process. In certain embodiments, an application such as Salesforce® is open in the main session. Next, the user or user session robot, in the client session, receives, e.g., a client ID from the main session, goes to a website (e.g., HubSpot®), and performs an automation to collect information related to the client's interaction with the website. However, an attended RPA robot may interact with any suitable application(s) and / or obtain data from any suitable source (e.g., a database, another application, etc.) without departing from the scope of the present invention.

[0030] In some embodiments, the RPA workflow of a user session robot executing in the main session can cause a client session robot to execute a second workflow in the client session. Objects can be passed back and forth between workflows in some embodiments via IPC, e.g., data can be stored by a client session robot accessible to the user session robot, etc. This can be beneficial when a portion of the workflow takes a relatively long time to complete. If a user is prevented from interacting with his or her computing system, the user may become unproductive if the user session robot performs all of the workflow activities. However, by operating the robot from another session, some embodiments allow a user to access other applications / instances while the robot is performing its task.

[0031] Applications of some embodiments include, but are not limited to, emulators, simulators, VMs, and hybrid VMs (such as Docker® containers) that provide OS-level virtualization. Some embodiments create and host one or more robot sessions as windows that include the UI of an application controlled by respective attended automation robots. In certain embodiments, the robot session window may include only the application(s) with which the client session robot is interacting. As used herein, "window" can apply to a window representing the UI shown within the main UI, a second screen of a second display of a computing system, a virtual desktop, an isolated environment (i.e., a window (referred to as "host") that renders the UI of all applications (referred to as "children") launched within the environment and executes them in the context of the host session), etc., without departing from the scope of the present invention.

[0032] By executing multiple sessions, a user and a user session robot can interact in a first session (e.g., a parent session) while a robot (or robots) can operate in those session(s). In this way, the user can interact with an application not being used by the user session and the client session robot (e.g., the user can use Outlook® while the user session robot is moving data from Excel® to a web browser), or, if the application can support this functionality, the user may be able to interact with the same application as the user session robot and / or the client session robot (e.g., the user can interact with a different instance while the user session robot or the client session robot is interacting with one instance of a web browser).

[0033] Both the user and the robot(s) are installing the same application and interacting with the file system in some embodiments. Changes made through the application with the robot(s) and the user are made as if a single user had made them in some embodiments, rather than the user and the robot working with different versions of the application and the file system respectively. In other words, the application in such embodiments is the user's local Excel®, Outlook®, etc. Also, the local file system can be utilized without additional configuration. This is different from, for example, Docker® containers, which require additional configuration steps to provide access to the host operating system's file system for the application running within the container.

[0034] In some embodiments, any desired number of sessions for any number of robots can be created and used without departing from the scope of the present invention. For example, the user and the user session robot or another application or process may operate in a first session, the first client session robot may operate in a second session, the second client session robot may operate in a third session, and so on. In certain embodiments, multiple robots operate in a single session and can potentially alternate for interaction with one or more common applications.

[0035] The function for creating a session may be implemented, for example, via a Windows (registered trademark) Terminal Services child session, which allows the user to create a session back on their own machine without logging out. The newly created session is displayed as a child window and can contain and launch applications that exist in the user's session. In other words, the separation between the user and the robot occurs at the UI level. For example, if a file is deleted, this occurs in all sessions running on the computing system.

[0036] In some embodiments, a hybrid virtual machine such as Docker (registered trademark) may be used. The hybrid virtual machine enables the execution of applications and robots within the container using a remote UI. Such embodiments may separate the robots contained therein and grant permissions in units of permissions. The robot needs to "open holes" to things outside the container. Therefore, the container is essentially its own sandbox with configurable permissions on how to "escape" (e.g., open ports, mount folders, etc.). In certain embodiments, a full virtual machine may be used.

[0037] Certain embodiments may be employed in robotic process automation (RPA). FIG. 1 is an architecture diagram showing an RPA system 100 according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that enables developers to design and implement workflows. The designer 110 provides solutions for application integration and automates third-party applications, administrative information technology (IT) tasks, and business IT processes. The designer 110 may facilitate the development of automation projects that are graphical representations of business processes. Put simply, the designer 110 facilitates the development and deployment of workflows and robots.

[0038] Automation projects enable the automation of rule - based processes by giving developers control over the execution order and relationships between a custom set of steps developed in a workflow defined herein as "activities". A commercial example of an embodiment of Designer 110 is UiPath Studio (trademark). Each activity can include actions such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows can be nested or embedded.

[0039] Workflow types can include, but are not limited to, sequences, flowcharts, FSMs, and / or global exception handlers, etc. Sequences may be particularly suitable for linear processes that enable the flow from one activity to another without cluttering the workflow. Flowcharts may be particularly suitable for more complex business logic, enabling the integration of decision - making and the connection of activities in more diverse ways through multiple branching logic operators. FSMs may be particularly suitable for large - scale workflows. FSMs can use a finite number of states triggered by conditions (i.e., transitions) or activities during their execution. Global exception handlers may be particularly suitable for determining the behavior of a workflow when encountering execution errors or for debugging the process.

[0040] When a workflow is developed within Designer 110, the execution of the business process is coordinated by Conductor 120, which coordinates one or more robots 130 that execute the workflows developed within Designer 110. A commercial example of an embodiment of Conductor 120 is UiPath Orchestrator (trademark). Conductor 120 facilitates the management of the generation, monitoring, and deployment of resources in an environment. Conductor 120 can operate as an integration point, or one of the integration points, with third - party solutions and applications.

[0041] The conductor 120 can manage all the robots 130 and connect and execute the robots 130 from a central point. The types of robots 130 that can be managed include, but are not limited to, attended robots 132, unattended robots 134, development robots (similar to unattended robots 134 but used for development and testing purposes), and non-production robots (similar to attended robots 132 but used for development and testing purposes). The attended robot 132 may be triggered by user events or scheduled to occur automatically and can operate alongside humans on the same computing system. The attended robot 132 can be used with the conductor 120 for centralized process deployment and logging media. The attended robot 132 may assist a human user in achieving various tasks and may be triggered by user events. In some embodiments, the process cannot start from the conductor 120 on this type of robot and / or they cannot execute under a locked screen. In certain embodiments, the attended robot 132 can only be launched from a robot tray or from a command prompt. The attended robot 132 preferably operates under human supervision in some embodiments.

[0042] The unattended robot 134 can operate unmanned on a virtual environment or a physical machine and can automate many processes. The unattended robot 134 can be responsible for providing remote execution, monitoring, scheduling, and work queue support. Debugging for all robot types can be performed from the designer 110 in some embodiments. Both attended and unattended robots can automate various systems and applications including, but not limited to, mainframes, web applications, VMs, enterprise applications (e.g., those generated by SAP®, SalesForce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0043] The conductor 120 can have various capabilities including, but not limited to, provisioning, deployment, versioning, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning can include creating and maintaining a connection between the robot 130 and the conductor 120 (e.g., a web application). Deployment can include ensuring the correct delivery of the package version to the robot 130 assigned for execution. Versioning may, in some embodiments, include managing unique instances of some processes or configurations. Configuration can include maintaining and delivering the robot environment and process configurations. Queuing can include providing management of queues and queue items. Monitoring can include tracking specific data of the robot and maintaining user permissions. Logging can include saving and indexing logs to a database (e.g., an SQL database) and / or another storage mechanism (e.g., ElasticSearch (registered trademark) that provides the ability to store large datasets and execute queries quickly). The conductor 120 can provide interconnectivity by operating as a central point of communication for third - party solutions and / or applications.

[0044] The robot 130 is an execution agent that executes the workflow constructed by the designer 110. One commercial example of some embodiments of the robot(s) 130 is UiPath Robots (trademark). In some embodiments, the robot 130, by default, installs the Microsoft Windows (registered trademark) Service Control Manager (SCM) management service. As a result, such a robot 130 can open an interactive Windows (registered trademark) session under a local system account and can have the rights of a Windows (registered trademark) service.

[0045] In some embodiments, the robot 130 can be installed in user mode. For such a robot 130, it means having the same rights as the user in which a given robot 130 is installed. This feature may also be available for high-density (HD) robots that ensure maximum utilization of each machine. In some embodiments, any type of robot 130 can be configured in an HD environment.

[0046] The robot 130 in some embodiments is divided into a plurality of components, each specialized for a specific automation task. The robot components in some embodiments include, but are not limited to, SCM management robot service, user mode robot service, executor, agent, and command line. The SCM management robot service manages and monitors Windows® sessions and operates as a proxy between the conductor 120 and the execution host (i.e., the computing system on which the robot 130 is executed). These services are entrusted with managing the qualification information of the robot 130. The console application is launched by the SCM under the local system.

[0047] The user mode robot service in some embodiments manages and monitors Windows® sessions and operates as a proxy between the conductor 120 and the execution host. The user mode robot service may be entrusted with managing the qualification information of the robot 130. If the SCM management robot service is not installed, a Windows® application can be automatically launched.

[0048] The executor can perform a job given under a Windows (registered trademark) session (i.e., can perform a workflow. The executor can be aware of the dots per inch (DPI) setting per monitor. The agent can be a Windows (registered trademark) Presentation Foundation (WPF) application that displays jobs available in the system tray window. The agent can be a client of the service. The agent can request the start or stop of a job and the change of settings. The command line is a client of the service. The command line is a console application that can request the start of a job and wait for its output.

[0049] As described above, the fact that the components of the robot 130 are divided helps developers, support users, and computing systems to more easily execute, identify, and track what each component is doing. In this way, special behaviors can be configured for each component, such as setting different firewall rules for the executor and the service. The executor can always, in some embodiments, recognize the DPI setting per monitor. As a result, the workflow can be performed at any DPI, regardless of the configuration of the computing system on which the workflow was created. Also, in some embodiments, projects from the designer 110 can be made independent of the browser zoom level. In the case of an application marked as not recognizing or intentionally not recognizing DPI, DPI can be disabled in some embodiments.

[0050] Figure 2 is an architectural diagram showing the deployed RPA system 200 according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be, or may be a part of, the RPA system 100 of FIG. 1. It should be noted that either the client side, the server side, or both can include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 210 includes an executor 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not be executed on the computing system 210. The executor 212 is executing a process. As shown in FIG. 2, a plurality of business projects can be executed simultaneously. The agent 214 (e.g., Windows® service) is, in this embodiment, a single connection point for all executors 212. All messages in this embodiment are recorded in the conductor 230, which further processes them via the database server 240, the indexer server 250, or both. As described above with respect to FIG. 1, the executor 212 can be a robot component.

[0051] In some embodiments, the robot represents an association between a machine name and a user name. The robot can manage multiple executors simultaneously. In a computing system (such as Windows® Server 2012) that supports multiple interactive sessions running simultaneously, multiple robots can be executed simultaneously, each running in a separate Windows® session using a unique user name. This is called the HD robot described above.

[0052] Agent 214 is also responsible for sending the state of the robot (e.g., periodically sending a "heartbeat" message indicating that the robot is still functioning) and downloading the required version of the package to be executed. The communication between Agent 214 and Conducter 230 is, in some embodiments, always initiated by Agent 214. In a notification scenario, Agent 214 may open a WebSocket channel that is later used by Conducter 230 to send commands (e.g., start, stop, etc.) to the robot.

[0053] On the server side, there are a presentation layer (web application 232, Open Data Protocol (OData) Representational State Transfer (REST) Application Programming Interface (API) endpoint 234, notification and monitoring 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240, indexer server 250). The conductor 230 includes the web application 232, the OData REST API endpoint 234, notification and monitoring 236, and the API implementation / business logic 238. In some embodiments, most actions that a user performs at the interface of the conductor 230 (e.g., via the browser 220) are executed by calling various APIs. Such operations may include, but are not limited to, launching jobs on a robot, adding / removing data in a queue, scheduling jobs to be executed unattended, etc., without departing from the scope of the present invention. The web application 232 is the visual layer of the server platform. In this embodiment, the web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, script language, or any other format may be used without departing from the scope of the present invention. The user interacts with the web page from the web application 232 via the browser 220 in this embodiment to perform various operations for controlling the conductor 230. For example, the user may create a robot group, assign packages to robots, analyze logs for each robot and / or for each process, start and stop robots, etc.

[0054] In addition to the web application 232, the conductor 230 also includes a service layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the present invention. The REST API is consumed by both the web application 232 and the agent 214. The agent 214 is, in this embodiment, a supervisor for one or more robots on a client computer.

[0055] The REST API of this embodiment covers configuration, logging, monitoring, and queuing functions. Configuration endpoints may, in some embodiments, be used to define and configure the users, permissions, robots, assets, releases, and environments of the application. Logging REST endpoints may be used to log various information, such as errors, explicit messages sent by robots, and other environment-specific information. Deployment REST endpoints may be used by robots to query the version of the package to be executed when a job start command is used in the conductor 230. Queuing REST endpoints may be responsible for the management of queues and queue items, such as adding data to a queue, retrieving transactions from a queue, and setting the status of a transaction.

[0056] Monitoring of the REST endpoints may monitor the web application 232 and the agent 214. The notification and monitoring API 236 may be a REST endpoint used for the registration of the agent 214, the delivery of configuration settings to the agent 214, and the sending and receiving of notifications from the server and the agent 214. The notification and monitoring API 236 may use WebSocket communication in some embodiments.

[0057] In this embodiment, the persistent layer includes a pair of server-database servers 240 (e.g., SQL servers) and an indexer server 250. The database server 240 in this embodiment stores configurations such as robots, robot groups, related processes, users, roles, schedules, etc. In some embodiments, this information is managed via a web application 232. The database server 240 may manage queues and queue items. In some embodiments, the database server 240 may store (in addition to or instead of the indexer server 250) messages recorded by the robots.

[0058] Although optional in some embodiments, the indexer server 250 stores information recorded by the robots and creates indexes. In certain embodiments, the indexer server 250 may be disabled via configuration settings. In some embodiments, the indexer server 250 uses ElasticSearch (registered trademark), an open-source project full-text search engine. Messages recorded by the robots (e.g., using activities such as log messages or line writes) may be sent to the indexer server 250 via logging REST endpoint(s), where they are indexed for future use.

[0059] Figure 3 is an architecture diagram showing the relationship 300 between the designer 310, activities 320, 330, and driver 340 according to an embodiment of the present invention. As described above, the developer uses the designer 310 to develop a workflow to be performed by the robot. The workflow may include user-defined activities 320 and UI automation activities 330. In some embodiments, non-text visual components in an image can be identified, which is referred to herein as computer vision (CV). Some CV activities related to such components can include, but are not limited to, click, type, get text, hover, detect presence of an element, update scope, highlight, etc. In some embodiments, click can identify an element using, for example, CV, optical character recognition (OCR), fuzzy text matching, and multi-anchor, and click on it. Type may identify an element using the above and types within the element. Getting text can identify the location of specific text and scan it using OCR. Hover can identify an element and hover over it. Detecting the presence or absence of an element can confirm whether the presence or absence of an element on the screen using the techniques described above. In some embodiments, there may be hundreds or even thousands of activities that can be implemented in the designer 310. However, any number and / or type of activities can be utilized without departing from the scope of the present invention.

[0060] The UI automation activity 330 is a subset of special low-level activities described in low-level code (e.g., CV activities) that facilitate interaction with applications through the UI layer. In certain embodiments, the UI automation activity 300 may simulate user input, for example, via window messages. The UI automation activity 330 facilitates these interactions via a driver 340 that enables the robot to interact with the desired software. For example, the driver 340 may include an OS driver 342, a browser driver 344, a VM driver 346, an enterprise application driver 348, and the like.

[0061] The driver 340 may interact with the OS at a low level, such as by looking for hooks and monitoring keys. They may also facilitate integration with Chrome (registered trademark), IE (registered trademark), Citrix (registered trademark), SAP (registered trademark), etc. For example, a "click" activity performs the same role in these different applications via the driver 340.

[0062] Figure 4 is an architecture diagram showing an RPA system 400 according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be or include the RPA system 100 and / or 200 of FIGS. 1 and / or 2. The RPA system 400 includes a plurality of client computing systems 410 that execute robots. The computing system 410 can communicate with a conductor computing system 420 via a web application executed thereon. The conductor computing system 420 can in turn communicate with a database server 430 and any indexer server 440.

[0063] With respect to FIGS. 1 and 3, although web applications are used in these embodiments, it should be noted that any suitable client and / or server software can be used without departing from the scope of the present invention. For example, a conductor may execute a server-side application that communicates with a non-web-based client software application on a client computing system.

[0064] FIG. 5 is an architecture diagram showing a computing system 500 configured to facilitate intersession automation for an RPA robot according to an embodiment of the present invention. In some embodiments, the computing system 500 may be one or more computing systems depicted and / or described herein. The computing system 500 includes a bus 505 or other communication mechanism for communicating information, and one or more processors 510 coupled to the bus 505 for processing information. The processor(s) 510 can be any type of general or special-purpose processor, including a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor(s) 510 may also have multiple processing cores, and at least some of the cores may be configured to perform specific functions. In some embodiments, multiple parallel processing may be used. In certain embodiments, at least one processor(s) 510 can be a neuromorphic circuit that includes processing elements that mimic biological neurons. In some embodiments, the neuromorphic circuit may not require the typical components of a von Neumann computing architecture.

[0065] Computing system 500 further includes a memory 515 for storing information and instructions to be executed by one or more processors 510. Memory 515 can be composed of a random access memory (RAM), a read-only memory (ROM), a flash memory, a cache, a static storage device such as a magnetic disk or an optical disk, or other types of non-transitory computer-readable media, or any combination thereof. The non-transitory computer-readable media can be any available media accessible by one or more processors 510, and can include volatile media, non-volatile media, or both. Also, the media can be removable, non-removable, or both.

[0066] Furthermore, computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communication network via a wireless and / or wired connection. In some embodiments, the communication device 520 may be configured to use Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiplexing (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile (GSM) communication, General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), cdma2000, Wideband CDMA (W-CDMA), High-Speed Downlink Packet Access (HSDPA), High-Speed Uplink Packet Access (HSUPA), High-Speed Packet Access (HSPA), Long Term Evolution (LTE), LTE Advanced (LTE-A), 802.11x, Wi-Fi, Zigbee, Ultra-WideBand (UWB), 802.16x, 802.15, Home Node-B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near-Field Communications (NFC), 5th Generation (5G), New Radio (NR), any combination thereof, and / or be configured to use any other currently existing or future-implemented communication standard and / or protocol without departing from the scope of the present invention.In some embodiments, the communication device 520 may include one or more antennas that, without departing from the scope of the present invention, are a single antenna, an array antenna, a phased antenna, a switched antenna, a beamforming antenna, a beam steering antenna, combinations thereof, and / or any other antenna configuration.

[0067] The processor(s) 510 is further coupled via bus 505 to a display 525 such as a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, a Retina (registered trademark) display, an IPS (In-Plane Switching) display, or any other suitable display for presenting information to a user. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, etc., using a resistive method, a capacitive method, a surface acoustic wave (SAW) capacitive method, an infrared method, an optical imaging method, a dispersion signal method, an acoustic pulse recognition method, a frustrated total internal reflection method, etc. Without departing from the scope of the present invention, any suitable display device and haptic I / O may be used.

[0068] Keyboard 530 and cursor control devices 535, such as computer mice, touch pads, etc., are further coupled to bus 505 to enable a user to interface with computing system 500. However, in certain embodiments, there may be no physical keyboard and mouse, and the user can interact with the device only via display 525 and / or a touch pad (not shown). The type and combination of any input device can be used as a matter of design choice. In certain embodiments, there is no physical input device and / or display. For example, the user may interact with computing system 500 remotely via another computing system communicating with computing system 500, or computing system 500 may operate autonomously.

[0069] Memory 515 stores software modules that provide functionality when executed by processor(s) 510. The modules include operating system 540 for computing system 500. The modules further include an intercession automation module 545 configured to execute all or part of the processes described herein or derivatives thereof. Computing system 500 may include one or more additional functional modules 550 including additional functionality.

[0070] One skilled in the art would understand that the "system" can be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device, or a combination of devices, without departing from the scope of the present invention. Presenting the functions described above as being performed by the "system" is not intended to limit the scope of the present invention in any way, but rather to provide an example of many embodiments of the present invention. In fact, the methods, systems, and devices disclosed herein may be implemented in a localized form and a distributed form that is consistent with computing technologies including cloud computing systems.

[0071] It should be noted that some of the system features described herein are presented as modules in order to emphasize implementation independence more. For example, a module can be implemented as a hardware circuit including custom very large scale integration (VLSI) circuits or gate arrays, logic chips, transistors, or other off-the-shelf semiconductors such as individual components. Also, a module can be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, and the like.

[0072] The module can also be at least partially implemented in software for execution by various types of processors. For example, a specified unit of executable code can include one or more physical or logical blocks of computer instructions that may be organized, for example, as objects, procedures, or functions. Nevertheless, a specified module that is executable need not be physically located together and can include modules when logically combined and can include separate instructions stored in different locations to achieve the purpose stated for the module. Further, the module can be stored in a computer-readable medium such as, for example, a hard disk drive, flash device, RAM, tape, and / or any other non-transitory computer-readable medium used to store data without departing from the scope of the present invention.

[0073] In fact, a module of executable code can be a single instruction, a number of instructions, or even be distributed across multiple different code segments, between different programs, and between multiple memory devices. Similarly, the operational data can be specified and shown within the module and can be embodied and organized in any suitable form within any suitable type of data structure. The operational data can be collected as a single data set or can be distributed in different locations across different storage devices and can exist, at least in part, simply as electronic signals on a system or network.

[0074] As described above, in attended automation, the robot works with the user on the same computing system. Since robots in RPA often interact with the computing system in similar ways (e.g., generating mouse click and key press events and simulating these events via an API (e.g., using window messages), etc.), some embodiments create one or more robot sessions where attended automation is hosted and executed. Unlike existing RPA systems, the user can benefit from the ability to interact with their computing system while the robot(s) execute attended automation in the robot session(s). The user can continue to monitor what the robot is doing and interact with the robot through the host automation window(s) for the same robot session(s) of the same embodiment. Thereby, the RPA robot effectively evolves into a true digital assistant that can execute work in parallel with the user, as it simply executes tasks faster and more reliably than a human user, and can provide a greater productivity improvement. In some embodiments, the user and the robot session(s) may execute on a remote machine controlled by the user's computing system.

[0075] In certain embodiments, the RPA robot can run on the user's computing system and drive a remote computing system via a remote runtime (e.g., via UiPath Remote Runtime™). UiPath Remote Runtime™ is a component that facilitates communication between remote applications or desktops such as Citrix Virtual Apps and Desktops™ and dedicated UiPath® extensions (e.g., UiPath® extension for Citrix® or UiPath® extension for Windows® Remote Desktop). UiPath Remote Runtime™ collects information about the UI elements targeted by the remote application and sends this information to the corresponding extension to enable native generation of selectors in UI Explorer™. In certain embodiments, the robot may use Microsoft's AppV technology that "virtually delivers" client applications to a machine.

[0076] As described above, in some embodiments, both the user and the robot interact with the same application instances and file system. FIG. 6 shows some of the applications and file system 660 of the user computing system 600 accessed by the user in user session 610 and the RPA robot in robot session 620, according to an embodiment of the present invention. As can be seen from FIG. 6, the web browser application 630, the spreadsheet application 640, and the email application 650 are accessible by both the user session 610 and the attended automation robot session 620. The user session 610 and the attended automation robot session 620 can interact with the web browser application 630 and the email application 650 simultaneously.

[0077] However, if the robot is interacting with the same file of the Excel (registered trademark) spreadsheet application 640, the user cannot interact with this application (for example, the user may only be able to open a "read-only" view, or the user may not be able to open the entire file). However, when automation uses the Office 365 API, for example, using a spreadsheet hosted on the user's OneDrive (registered trademark) and the user is accessing the spreadsheet via a web interface, such a restriction may not exist. For example, Google Sheets (registered trademark) is fully web-delivered and supports concurrent access. The user may receive a message indicating that the spreadsheet application 640 is locked and being accessed by another "user" (i.e., the robot). Both the user session 610 and the attended automation robot session 620 can also interact with the file system 660. Therefore, the changes made by the robot and the user in the application via their respective sessions will be made as if they were made by a single user, rather than the robot and the user working with different versions of the application and the file system respectively.

[0078] Figures 7A - 7J are screenshots showing examples of completing forms in a user session using a running RPA robot in a client session, and a user session robot or another application or process running in a user session, according to an embodiment of the present invention. In Figure 7A, the user session window 700 is shown, where the user can interact with the UI application and the robot is not currently performing any tasks. A robot tray icon 710 is displayed in the lower right portion of the user session window 700.

[0079] In FIG. 7B, the user launches a web browser and visits the invoice creation web page 720. In FIG. 7C, the user pulls up the robot tray 712 (e.g., by clicking on the robot tray icon 710) and selects the user session robot option 714 to be executed on his or her computing system in the user session. After selecting the user session robot option 714, the user session robot window 730 is displayed. Refer to FIG. 7D. In this embodiment, the user session robot only displays a status message. However, the user session robot may have any desired workflow function without departing from the scope of the present invention. In some embodiments, the user session robot automatically launches the client session robot or, when the option to execute the user session robot is selected, automatically launches both robots in each session.

[0080] In some embodiments, the window for the user session robot is not displayed and can be executed in the background. In certain embodiments, another application or process executes the function of the user session robot. This function may include, but is not limited to, communication with one or more client session robots, obtaining data provided by the client session robot(s), and interaction with the application running in the user session to input the obtained information. In some embodiments, instead of using the user session robot, another application, or another process, the application into which the information obtained by the client session robot is to be input is input by the application itself. This can be done in some embodiments without simulating mouse clicks, key presses, etc.

[0081] In FIG. 7E, the user pulls up the robot tray 712 and selects the client session robot option 716 to perform on his or her computing system in the client session. As shown in FIG. 7F, after selecting the client session robot option 716, a client session window 740 for the robot session is displayed as a child window on the screen. However, in some embodiments, the client session window 740 is not displayed and the client session robot runs in the client session without being visible to the user. In this embodiment, a client session window 742 for the client session robot automatically launches within the client session window 740 and includes a button 744 for fetching form data of the web page 720 within the user session.

[0082] In some embodiments, the robot session window is not displayed and the client session robot can automatically launch, operate, and close without being visible to the user. In certain embodiments, the client session robot can close its session after completing its workflow. In some embodiments, instead of launching from the robot tray, the client session is initiated and the client session robot can launch and operate without using the robot tray 712 (e.g., due to the user clicking a button in the main session application).

[0083] Looking at FIG. 7G, after the user clicks button 744, the client session robot starts fetching form data for web page 720. The data fetched by the client session robot may be provided to the user session robot via the IPC, stored in a spreadsheet, object-oriented database, plain text file, etc., or otherwise communicated to or made available to the user session robot. After the client session robot completes its workflow, a message is displayed in the client session robot window 742. Refer to FIG. 7H. Next, the user session robot starts using the form data fetched by the client session robot and enters it into the fields of web page 720. The text entered by the user session robot is visible to the user when web page 720 is not covered or minimized by another window. While the user session robot is completing the form, the user can interact with other applications and continue productivity.

[0084] Looking at FIG. 7I, as can be seen in the background, the user session robot has completed the form fields of web page 720. Thereafter, the user may close the client session window 740 and the user session robot window 730, the user session robot or the client session robot may automatically close the client session, or the client session window 740 may remain open. Next, the user may submit the completed form. Refer to FIG. 7J.

[0085] FIG. 8 is a flowchart showing a process 800 for intersession automation according to an embodiment of the present invention. The process begins at 810 by launching a user session window. This may be, for example, the main window associated with the operating system running on the user's computing system. Next, a user session process (e.g., an RPA robot, an application, another process, etc.) is started within the user session at 820. A client session is launched at 830, and a client session robot is launched within the client session at 840. In some embodiments, the client session may be launched in response to, for example, the client session robot being started or otherwise launched. Thereafter, the client session robot performs its workflow at 850. The data and / or actions performed by the client session robot may be provided to or made available to the user session process in some embodiments. Next, the user session process accesses the data generated by the client session robot at 860 and interacts with the appropriate application(s) (e.g., a web page, a spreadsheet, an ERP application, a sales application, etc.) based on that data at 870. In some embodiments, the application(s) to be interacted with may be the user session process itself. After the client session robot has completed its performance, the client session may be automatically terminated at 880 in some embodiments.

[0086] In some embodiments, a client session can be created via the operating system's child session API. Windows® Terminal Services Child Sessions or other child session APIs provided by the operating system can be used in some embodiments to create a second session without departing from the scope of the present invention. A robot tray application (e.g., UiPath® Robot Agent Desktop) or another application configured to launch a robot(s) can then use the operating system's process creation API with appropriate arguments to start a robot process in that child session. The robot tray application or other appropriate application can then communicate with the client robot process using an appropriate protocol (e.g., one built on a named pipe).

[0087] Communication with the client robot between the two sessions can be achieved using IPC protocols in some embodiments. These protocols may facilitate communication over a network, pipe, Component Object Model (COM), Remote Procedure Call (RPC), socket, etc. Appropriate session creation mechanisms and IPC protocols can be used similarly on other operating systems if supported. When the user clicks a button on the robot tray or causes such a function to be initiated in another application (e.g., by clicking a button), the robot tray application or other appropriate application can send that command to the client session robot process using an IPC protocol. The client session robot can similarly send status notifications (e.g., indicating that the robot is starting, running, paused, etc.) back to the robot tray application or other appropriate process via the IPC protocol.

[0088] Figure 9 is a flowchart showing a process 900 of inter - session automation where one robot executes in a user session and the other robot executes in a client session. The process begins at 910 with the start of the execution of the workflow of the user - session robot. When the user - session robot reaches an activity within its workflow that requests execution from the client robot at 920, the user - session robot causes the client - session robot to execute that workflow at 930 and waits for its completion.

[0089] The client - session robot completes that workflow at 940 and notifies the user - session robot. Next, the user - session robot uses the result of the client - session robot execution at 950 to complete the remainder of the user - session robot workflow. In some embodiments, the user - session robot and the client - session robot together complete a single logical workflow.

[0090] In certain embodiments, multiple client session robots may be used and potentially have at least some client session robots in different client sessions. Thereby, the user session robot may pass a portion of the workflow activity to the client session robot to complete. FIG. 10 shows scenario 1000 in which a portion of the workflow is completed by the user session robot and another portion of the workflow is completed by the client session robot according to an embodiment of the present invention. User session 1010 includes user session robot 1012 and application 1014. The first client session 1020 includes client session robot 1022, the second client session 1030 includes two client session robots 1032, 1034, and the third client session 1040 includes client session robot 1042. User session robot 1012 may communicate with and control the operation of client session robots 1022, 1032, 1034, 1042, for example, via IPC. Note that in some embodiments, while the client session robot(s) complete their respective workflows, the user session robot continues to perform its workflow.

[0091] FIG. 11 is a flowchart 1100 showing the execution of a multi-robot population workflow between user session robot U1 and a pair of client session robots C1 and C2 according to an embodiment of the present invention. C1 and C2 may be in the same client session or in different client sessions. U1 starts the execution of its workflow and reaches an activity that requests C1 to complete the workflow. U1 causes C1 to execute the workflow and the calling activity waits. During this time, U1 may perform other tasks in some embodiments.

[0092] When C1 completes its workflow, U1 resumes execution until it reaches an activity that requests the completion of the workflow from C2. U1 causes C2 to execute the workflow, and the calling activity waits. Again, U1 may perform other tasks while waiting for C2 in some embodiments.

[0093] When C2 completes its workflow, U1 resumes execution until it reaches an activity that requests the completion of the workflow from C1 again. This may be the same workflow activity that was previously executed or a different set of workflows or activities. U1 causes C1 to execute the workflow, and the calling activity waits. Again, U1 may perform other tasks while waiting for C1 in some embodiments. After C1's workflow ends, U1 resumes execution until U1's workflow ends.

[0094] The process steps executed in FIGS. 8, 9, and 11 may be executed by a computer program that encodes instructions to a processor(s) to perform at least a portion of the process(es) described in FIGS. 8, 9, and 11 according to embodiments of the present invention. The computer program may be stored on a non-transitory computer-readable medium. The computer-readable medium may be, but is not limited to, a hard disk drive, a flash device, RAM, a tape, and / or any other such medium or combination of media used to store data. The computer program may include encoded instructions for controlling a processor(s) of a computing system (e.g., the processor(s) 510 of the computing system 500 in FIG. 5) to implement all or a portion of the process steps described in FIGS. 8, 9, and 11, and this may also be stored on a computer-readable medium.

[0095] A computer program can be implemented in hardware, software, or a hybrid implementation. The computer program can be composed of modules that communicate operably with each other and is designed to send information or instructions to a display. The computer program can be configured to operate on a general-purpose computer, an ASIC, or any other suitable device.

[0096] It will be readily understood that the components of the various embodiments of the present invention may be arranged and designed in a variety of different configurations as generally described and illustrated herein. Accordingly, the detailed description of the embodiments of the present invention as represented in the accompanying figures is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the present invention.

[0097] The features, structures, or characteristics of the present invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, references throughout this specification to "certain embodiments", "some embodiments", or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the present invention. Accordingly, the appearances throughout this specification of "in certain embodiments", "in some embodiments", "in other embodiments", or similar language are not necessarily all referring to the same group of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0098] References throughout this specification to features, advantages, or similar language do not imply that all of the features and advantages that can be realized in the present invention should be in any single embodiment of the invention or in any embodiment of the invention. Rather, language referring to features and advantages is understood to mean that a particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Accordingly, discussions of features and advantages throughout this specification, as well as similar language, can refer to the same embodiment, but not necessarily to the same embodiment.

[0099] Furthermore, the described features, advantages, and characteristics of the present invention can be combined in any suitable manner in one or more embodiments. Those skilled in the relevant art will recognize that the present invention can be practiced without one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in particular embodiments that may not be present in all embodiments of the present invention.

[0100] Those of ordinary skill in the art will readily understand that the present invention as described above can be implemented using steps in a different order and / or using hardware elements in a configuration different from that disclosed. Accordingly, although the present invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain changes, modifications, and alternative configurations will become apparent while remaining within the spirit and scope of the invention. Therefore, the appended claims should be referred to in order to determine the scope of the present invention.

Claims

1. Start a user session process in a user session, Start a first client session robotic process automation (RPA) robot in a first client session, Start a second client session RPA robot in the first client session or a second client session, Have the first client session RPA robot perform the workflow of the first client session RPA robot in the first client session, Have the second client session RPA robot perform the workflow of the second client session RPA robot in the first client session or the second client session, Have the first client session RPA robot enable the user session process to utilize the result of the performance of the workflow of the first client session RPA robot, Have the second client session RPA robot enable the user session process to utilize the result of the performance of the workflow of the second client session RPA robot, Include interacting with an application in the user session using the result of the performance of the workflow of the first client session RPA robot by the user session process, The user session process is a computer-implemented method in which the first client session RPA robot executes other tasks while waiting for the performance of the workflow of the first client session RPA robot to be completed.

2. The interaction with the application in the user session includes inputting information into a form, The result of the execution of the RPA workflow includes data used by the user session process to input the information into the form, the computer-implemented method according to claim 1.

3. The computer-implemented method according to claim 1, wherein the user session process includes an RPA robot.

4. The computer-implemented method according to claim 1, wherein the first client session RPA robot can be viewed within a window associated with the first client session.

5. The computer-implemented method according to claim 1, wherein the first client session RPA robot is invisible.

6. The computer-implemented method according to claim 1, wherein the user session process waits for the first client session RPA robot to complete the execution of the workflow of the first client session RPA robot.

7. The computer-implemented method according to claim 1, wherein the first client session and the first client session RPA robot are started by the user session process.

8. The computer-implemented method according to claim 1, further including automatically closing the first client session RPA robot and the first client session after the first client session RPA robot completes the execution of the workflow of the first client session RPA robot.

9. The user session process includes an RPA robot, and the user session RPA robot and the first client session RPA robot collectively complete the execution of a larger workflow including the workflow of the first client session RPA robot, the computer-implemented method according to claim 1. **Claim 10**: Instructing the client session RPA robot to perform the workflow of the client session RPA robot via inter-process communication (IPC) by a user session robotic process automation (RPA) robot, performing, by the client session RPA robot, the workflow of the client session RPA robot in the client session, enabling the user session RPA robot to utilize the result of the performance of the workflow of the client session RPA robot by the client session RPA robot, receiving, by the user session RPA robot, the result of the performance of the workflow of the client session RPA robot, interacting, by the user session RPA robot, with an application on a computing system using the result of the performance of the workflow of the client session RPA robot, including a computer-implemented method, wherein the user session RPA robot performs other tasks while waiting for the client session RPA robot to complete the performance of the workflow of the client session RPA robot. **Claim 11** the interaction with the application on the computing system includes inputting information into a form, the result of the performance of the RPA workflow includes data used by the user session RPA robot to input the information into the form, the computer-implemented method according to claim 10. **Claim 12** the user session RPA robot waits for the client session RPA robot to complete the performance of the workflow of the client session RPA robot, the computer-implemented method according to claim 10.

13. The computer-implemented method according to claim 10, wherein the user session RPA robot and the client session RPA robot collectively complete the execution of a larger workflow including the workflow of the client session RPA robot.

14. Instructing, by a user session robotic process automation (RPA) robot, a client session RPA robot to execute a workflow of the client session RPA robot via inter-process communication (IPC), executing, by the client session RPA robot, the workflow of the client session RPA robot in a client session, enabling, by the client session RPA robot, a result of the execution of the workflow of the client session RPA robot to be available to the user session RPA robot, receiving, by the user session RPA robot, the result of the execution of the workflow of the client session RPA robot, interacting, by the user session RPA robot, with an application on a computing system using the result of the execution of the workflow of the client session RPA robot, including the user session RPA robot and the client session RPA robot collectively complete the execution of a larger workflow including the workflow of the client session RPA robot, The computer-implemented method, wherein the user session RPA robot executes other tasks while waiting for the client session RPA robot to complete the execution of the workflow of the client session RPA robot.

15. The interaction with the application in the user session includes inputting information into a form, The computer-implemented method of claim 14, wherein the result of the execution of the RPA workflow includes data used by the user session RPA robot to input the information into the form.

Images