Inter-session automation for Robotic Process Automation (RPA) robots

Abstract

Inter-session automation for Robotic Process Automation (RPA) is disclosed. A robot or another application or process running in a user session can interact with the application, but one or more managed RPA robots in one or more sub-sessions perform operations and retrieve data, which the user session robot then uses to interact with the application in the user session. Managed RPA robots in the client session can share data via inter-process communication (IPC) protocols by storing the data in a persistent data warehouse such as spreadsheets, object-oriented databases, plain text files, another data warehouse, or files. The user session robot or another application or process running in the parent session can then read this information and respond accordingly.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. non-provisional patent application No. 15 / 930,906, filed May 13, 2020, which is a continuation in part (CIP) of U.S. non-provisional patent application No. 16 / 793,064, filed February 18, 2020, and claims the benefit thereto. The technical solutions of these earlier applications are incorporated herein by reference in their entirety. Technical Field

[0003] This invention relates generally to robotic process automation (RPA), and more specifically to inter-session automation for RPA robots. Background Technology

[0004] Manned automated RPA robots typically run on a user-operated computing system in the same session as the user. For example, a manned RPA robot can work alongside the user to complete certain tasks under the user's commands. However, in manned automation scenarios, the RPA robot may "take over" the user's computing system. The user may want to perform other activities while the robot interacts with the computing system, but the user is prevented from doing so. In other words, the robot controls the application through a user interface (UI) in the same way as the user (e.g., simulating mouse clicks and keyboard input).

[0005] Various technologies exist for creating complete or partial copies of an operating system or applications running on it. Emulators have existed for decades and provide developers with the ability to test and debug applications. For example, emulators allow developers to test and debug mobile applications using operating systems that do not support running development tools directly. and Both provide the ability to run from the development machine for testing and debugging. or The application's emulator, because the developer tools cannot run natively on those mobile operating systems.

[0006] Emulators allow developers to host windows on their local machine, enabling them to test and debug the behavior of applications that are difficult or impossible to execute on their development machine. For example, an emulator allows a developer to click a button to rotate the emulator, which tells the application running inside the emulator that the device has been rotated for the purpose of testing and debugging the application's behavior in response to these events. Another common example is multi-touch. Many developer machines do not support touch, so emulators allow developers to test and debug how their applications respond to multiple touch points. and The simulator also provides simulation capabilities. Additionally, They provide emulators for their Universal Windows Platform (UWP) applications.

[0007] A virtual machine hosts a second operating system on the host machine and can be opened and monitored through a window. It runs a completely different operating system and shares hardware with the host machine. The "guest" machine must have its own copies of applications installed and does not share public resources or files with the user machine.

[0008] Containers are conceptually a hybrid form of virtual machines. All applications that need to be executed are encapsulated in an immutable package that runs directly on the host operating system. This package is not a complete copy of another operating system, but by default it does not share or have access to any applications or resources on the hosting machine. Therefore, from a user experience perspective, Containers feel similar to virtual machines, but technically, containers do not run on a completely separate operating system.

[0009] However, conventional emulators, simulators, virtual machines (VMs), and those providing operating system (OS) layer virtualization (e.g., Hybrid VMs (containers) do not address the problems that arise when an automated robot, operated by a human user, is running on the same computing system. Therefore, the user essentially becomes a bystander to their own computing system, watching the robot work without being able to interact with other applications on the machine that require user interaction. Thus, an improved approach could be beneficial. Summary of the Invention

[0010] Certain embodiments of the present invention can provide solutions to problems and needs in the art that have not yet been fully identified, recognized, or resolved by current RPA technologies. For example, some embodiments of the present invention relate to inter-session automation for RPA robots.

[0011] In one embodiment, the computer-implemented method includes initiating a user session procedure in a user session and initiating a first client session RPA robot in a first client session. The computer-implemented method further includes executing a workflow of the first client session RPA robot in the first client session by the first client session RPA robot, and making the results of the execution of the workflow of the first client session RPA robot available to the user session procedure by the first client session RPA robot. The computer-implemented method further includes the user session procedure interacting with an application in the user session using the results of the execution of the workflow of the first client session RPA robot.

[0012] In another embodiment, a computer-implemented method includes executing a client-session RPA robot's workflow in a client session by a client-session RPA robot, and making the results of the client-session RPA robot's workflow execution available to a user-session RPA robot. The computer-implemented method also includes having a user-session RPA robot interact with an application on a computing system using the results of the client-session RPA robot's workflow execution.

[0013] In another embodiment, a computer-implemented method includes a client-session RPA robot executing a workflow of a client-session RPB robot within a client session, and the client-session RPA robot making the results of the execution of the client-session RPA robot's workflow available to a user-session RPA robot. The computer-implemented method also includes the user-session RPA robot interacting with an application on a computing system using the results of the execution of the client-session RPA robot's workflow. The user-session RPA robot and the client-session RPA robot jointly complete the execution of a larger workflow that includes the workflow of the client-session RPB robot. Attached Figure Description

[0014] To facilitate the understanding of the advantages of certain embodiments of the invention, a more specific description of the invention, which has been briefly described above, will be presented by reference to specific embodiments illustrated in the accompanying drawings. While it should be understood that these drawings merely depict exemplary embodiments of the invention and are therefore not intended to limit its scope, the invention is described and explained using additional specificity and detail in the use of the drawings, in which:

[0015] Figure 1 This is an architectural diagram illustrating a robotic process automation (RPA) system according to an embodiment of the present invention.

[0016] Figure 2 This is an architectural diagram illustrating an RPA system deployed according to an embodiment of the present invention.

[0017] Figure 3 This is an architecture diagram illustrating the relationship between the designer, activities, and drivers according to an embodiment of the present invention.

[0018] Figure 4 This is an architectural diagram of an RPA system according to an embodiment of the present invention.

[0019] Figure 5 This is an illustration of the architecture of a computing system configured to facilitate inter-session automation for RPA robots according to an embodiment of the present invention.

[0020] Figure 6The illustrations depict some applications and file systems of a user computing system accessed by a user in a user session and by an RPA robot in a robot session, according to embodiments of the present invention.

[0021] Figure 7A -J is a screenshot illustrating an example of using an RPA bot running in a client session and a user session bot or another application or process running in a user session to complete a form in a user session, according to an embodiment of the present invention.

[0022] Figure 8 This is a flowchart illustrating the process of automating inter-session communication according to an embodiment of the present invention.

[0023] Figure 9 This is a flowchart illustrating an inter-session automation process according to an embodiment of the present invention, in which one robot runs in a user session and another robot runs in a client session.

[0024] Figure 10 The illustration depicts a scenario according to an embodiment of the present invention, in which a portion of the workflow is completed by a user chatbot while other portions of the workflow are completed by a client chatbot.

[0025] Figure 11 This is a flowchart illustrating the execution of a multi-robot aggregation workflow between a user session robot U1 and a pair of client session robots C1 and C2 according to an embodiment of the present invention.

[0026] Unless otherwise indicated, similar reference numerals throughout the figure. Figure 1 Corresponding features. Detailed Implementation

[0027] Some embodiments relate to inter-session automation for managed RPA. For example, a form (e.g., a webpage, spreadsheet, another application with populateable fields, etc.) may be displayed in a user session (also referred to herein as the main session or parent session). A robot or another application or process running in the user session may interact with the form, but one or more managed RPA robots in one or more sub-sessions (also referred to herein as client sessions, robot sessions, or auxiliary sessions) may perform actions and retrieve data, which the user session robot will then use to populate the form in the user session. The managed RPA robots in the client session may share data via inter-process communication (IPC) protocols by storing the data in a persistent data warehouse, including but not limited to spreadsheets, object-oriented databases, plain text files, another data warehouse, or files. The user session robot or another application or process running in the parent session can then read this information and respond accordingly (e.g., fill out the form using the information). In this way, while (multiple) client session RPA bots are executing their workflows, the operation of (multiple) managed automation bots does not prevent users from using other applications or instances, but when a user session bot populates an application with input provided by the user session RPA bot, that input is visible to the user in the user session.

[0028] In some embodiments, the process may operate as follows: A user may click a button or otherwise launch the RPA robot in a sub-session of a user session (e.g., a parent or main session that starts when the user starts his or her computing system). In some embodiments, for example, the robot session may already be running, or it may be launched separately by the user from the robot tray. In some embodiments, the robot tray may be a window that displays available jobs in the system tray. This demonstrates a Workflow Framework (WPF) application. Users can enable bots to execute their workflows in sub-sessions, and then the bots can interact with one or more applications in the main session. In some embodiments, bot functionality can be initiated by the user clicking a button to invoke the bot in the main session. In some embodiments, a user-session bot can initiate a client-side session bot.

[0029] In some embodiments, if at least one application on which the workflow operates has not yet been started, the client session bot may cease operation and provide messages. In some embodiments, if the application(s) are not open, the user session bot or other user session applications or processes may start them. If the operating system allows such operation, the client session bot may start the application in the parent session. When the client session bot makes data available to the user session bot, the data becomes visible in the main session as the user session bot populates the data.

[0030] In some embodiments, a user can enable a bot to execute a workflow in a client session that redirects to a website, collects information, and makes that information available to the user's session bot or another application or process to access and input data into, for example, a spreadsheet visible in the main session. In some embodiments, applications (such as...) It is opened in the main session. Then, the user or user session bot runs an automation in the client session, which receives the client ID from the main session and redirects to the website (e.g., It also collects information related to client-website interactions. However, without departing from the scope of the invention, an attended RPA robot can interact with any suitable application(s) and / or obtain data from any suitable source (e.g., a database, another application, etc.).

[0031] In some embodiments, an RPA workflow of a user session bot running in a main session enables a client session bot to run auxiliary workflows in a client session. In some embodiments, objects can be passed back and forth between workflows via IPC; for example, data can be stored by a client session bot accessible to the user session bot, etc. This can be beneficial in situations where parts of the workflow take a relatively long time to complete. If the user session bot performs all workflow activities while preventing the user from interacting with his or her computing system, the user may not be productive. However, by operating the bot from a different session, some embodiments allow the user to access other applications / instances while the bot performs its tasks.

[0032] Some embodiments include, but are not limited to, emulators, simulators, virtual machines, and hybrid virtual machines that provide OS-level virtualization (e.g., (Container). Some embodiments create and host one or more robot sessions as windows that include the UI of an application controlled by a corresponding manned automated robot. In some embodiments, the robot session window may only include the applications(s) that the client session robot is interacting with. As used herein, "window" may be applied to the following without departing from the scope of the invention: a window representing a UI displayed within a main UI, a second screen of a second display of a computing system, a virtual desktop, a UI that draws all applications launched within that environment (referred to as "children") and an isolated environment (i.e., a window, referred to as "host") in which all applications run (within the context of the host session), etc.

[0033] Running multiple sessions allows multiple bots to operate within their own sessions, while the user and the user session bot interact with the first session (e.g., the parent session). Therefore, the user can interact with applications not currently being used by the user session and the client session bot (e.g., the user can use...). Meanwhile, user chatbots are transferring data from... (Move to a web browser), or users and user chatbots and / or client chatbots may be able to interact with the same application (if that application is capable of doing so) (e.g., a user chatbot or client chatbot interacts with one instance in a web browser, while the user interacts with another instance).

[0034] In some embodiments, both the user and the bots (multiple robots) interact with the same application installation and file system. In some embodiments, changes made within the application via the bots (multiple robots) and the user are treated as if made by a single user, rather than affecting separate versions of the application and file system by the user and the bots. In other words, the application in such embodiments is local to the user. And so on. Furthermore, the local file system can be utilized without additional configuration. This differs from... Containers, for example, require additional configuration steps to provide applications running in the container with access to the host operating system's file system.

[0035] In some embodiments, any desired number of sessions for any number of bots can be created and used without departing from the scope of the invention. For example, a user and user session bot or another application or process may operate in a first session, a first client session bot may operate in a second session, a second client session bot may operate in a third session, and so on. In some embodiments, multiple bots may operate in a single session, potentially interacting with one or more common applications in turn.

[0036] The function for creating a session can be accessed via... Terminal Services sub-sessions are used to achieve this. For example, they can create a session back on the user's own machine without requiring the user to log out. The newly created session can be displayed as a sub-window and can contain and launch applications that already exist in the user's session. In other words, the separation between the user and the bot occurs at the UI layer. For example, if a file is deleted, this happens across all sessions running on the computing system.

[0037] In some embodiments, a hybrid virtual machine may be used, for example Hybrid virtual machines allow a remote UI to be used to run applications and bots within a container. Such embodiments isolate the bots contained within and grant permissions on a per-permission basis. Bots must "punch" anything outside the container. Thus, the container is essentially its own sandbox with configurable permissions on how it "escapes" (e.g., opens ports, installs folders, etc.). In some embodiments, a full virtual machine can be used.

[0038] Some implementations can be used for robotic process automation (RPA). Figure 1 This is an architectural diagram illustrating an RPA system 100 according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that allows developers to design and implement workflows. The designer 110 provides solutions for application integration and for automating third-party applications, managing information technology (IT) tasks, and business IT processes. The designer 110 facilitates the development of automation projects, which are graphical representations of business processes. In short, the designer 110 facilitates the development and deployment of workflows and robots.

[0039] Automation projects automate rule-based processes by giving developers control over the execution order and the relationships between a set of custom steps (defined as "activities" in this document) developed within the workflow. A business example of an embodiment of Designer 110 is UiPath Studio. TM Each activity may include actions such as clicking a button, reading a file, writing to a login panel, and so on. In some embodiments, workflows may be nested or embedded.

[0040] Some types of workflows may include, but are not limited to, sequences, flowcharts, flow charts (FSMs), and / or global exception handlers. Sequences are particularly well-suited for linear processes, enabling flow from one activity to another without making the workflow chaotic. Flowcharts are particularly well-suited for more complex business logic, enabling the integration of decisions and the connection of activities in more diverse ways through multi-branch logical operators. FSMs are particularly well-suited for large workflows. FSMs can use a limited number of states in their execution, triggered by conditions (i.e., transitions) or activities. Global exception handlers are particularly well-suited for determining workflow behavior when encountering runtime errors and for use in debugging processes.

[0041] Once the workflow is developed in designer 110, the execution of the business process is meticulously orchestrated by orchestrator 120, which orchestrates one or more robots 130 to execute the workflow developed in designer 110. A business example of an embodiment of orchestrator 120 is the UiPath Orchestrator. TMCommand Device 120 facilitates the management of resource creation, monitoring, and deployment within the environment. Command Device 120 can serve as one of the integration points or aggregation points for third-party solutions and applications.

[0042] Commander 120 manages a group of robots 130, connecting and executing them from a central point. The types of manageable robots 130 include, but are not limited to, manned robots 132, unattended robots 134, development robots (similar to unattended robots 134, but used for development and testing purposes), and non-production robots (similar to manned robots 132, but used for development and testing purposes). Manned robots 132 can be triggered by user events or scheduled to occur automatically and operate in parallel with humans on the same computing system. Manned robots 132 can be used by commander 120 to centralize process deployment and logging media. Manned robots 132 can assist human users in completing various tasks and can be triggered by user events. In some embodiments, processes cannot be started from commander 120 on this type of robot and / or they cannot run under screen lock. In some embodiments, manned robots 132 can be started simply from the robot tray or from a command prompt. In some embodiments, manned robots 132 should operate under human supervision.

[0043] Unattended robot 134 operates unattended in a virtual environment or on a physical machine and can automate many processes. Unattended robot 134 can be responsible for remote execution, monitoring, scheduling, and providing support for work queues. In some embodiments, debugging for all robot types can be run from designer 110. Both manned and unattended robots can automate various systems and applications, including but not limited to mainframes, web applications, VMs, and enterprise applications (e.g., by...). Those produced by [unspecified entity] and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computing applications, etc.).

[0044] Commander 120 may have various capabilities, including but not limited to provisioning, deployment, version management, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining a connection between robot 130 and commander 120 (e.g., a web application). Deployment may include ensuring the correct delivery of packaged versions to assigned robots 130 for execution. In some embodiments, version management may include managing unique instances of processes or configurations. Configuration may include maintaining and delivering robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs to a database (e.g., an SQL database) and / or another storage mechanism (e.g., a database). It provides the ability to store and quickly query large datasets. Commander 120 can provide interconnectivity by acting as a centralized communication point for third-party solutions and / or applications.

[0045] Robot 130 is an execution agent that runs the workflow established in designer 110. One business example of some embodiments of (multiple) robots 130 is the UiPath robot. TM In some embodiments, Robot 130 has Microsoft installed by default. Services managed by the Service Control Manager (SCM). Therefore, such a robot 130 can open interactive windows under the local system account. Session and has The right to services.

[0046] In some embodiments, the robot 130 may be installed in a user-mode. For such a robot 130, this means that it has the same rights as the user to whom the given robot 130 has been installed. This feature can also be used for high-density (HD) robots, which ensure full utilization of each machine to its maximum extent possible. In some embodiments, any type of robot 130 may be configured in an HD environment.

[0047] In some embodiments, robot 130 is divided into several components, each dedicated to a specific automation task. In some embodiments, robot components include, but are not limited to, SCM-managed robot services, user-mode robot services, actuators, agents, and command lines. The SCM-managed robot services manage and monitor... The session acts as an intermediary between the controller 120 and the execution host (i.e., the computing system on which the robot 130 is executed). These services utilize and manage the trusted certificate for the robot 130. The console application is launched by the SCM under the local system.

[0048] In some embodiments, user-mode robot services manage and monitor The session acts as a proxy between the command center 120 and the execution host. The user-mode robot service utilizes and manages the trusted certificate for robot 130. If the SCM-managed robot service is not installed, The application can be launched automatically.

[0049] The executor can A given job runs within a session (i.e., they can execute workflows). The executor is aware of the dots per inch (DPI) setting for each monitor. The agent can display available jobs in the system tray window. Presentation Architecture (WPF). A proxy can be a client of a service. A proxy can request to start or stop a job and change settings. The command line is a client of the service. The command line is a console application that can request to start jobs and wait for their output.

[0050] The breakdown of the components of robot 130, as explained above, helps developers, support users, and the computing system operate more easily, identifying and tracking what each component is performing. Specific behaviors can be configured for each component in this way, such as establishing different firewall rules for executors and services. In some embodiments, the executor may always know the DPI setting of each monitor. Therefore, workflows can be executed at any DPI, regardless of the configuration of the computing system on which they are created. In some embodiments, projects from designer 110 may also be independent of browser zoom levels. For applications that do not know the DPI or are intentionally marked as not knowing the DPI, DPI can be disabled in some embodiments.

[0051] Figure 2 This is an architectural diagram illustrating a deployed RPA system 200 according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be... Figure 1 The RPA system 100 may be a part of it. It should be noted that the client side, server side, or both may include any desired number of computing systems without departing from the scope of the invention. On the client side, the robot application 210 includes an actuator 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not be running on the computing system 210. The actuator 212 is running a process. Several business processes may run simultaneously, such as... Figure 2 As shown. In this embodiment, agent 214 (e.g., The service is a single contact point for all executors 212. In this embodiment, all messages are logged to the controller 230, which further processes them via the database server 240, the indexer server 250, or both. (See above reference...) Figure 1As discussed, actuator 212 can be a robot component.

[0052] In some embodiments, a robot represents an association between a machine name and a username. A robot can manage multiple actuators simultaneously. In computing systems that support multiple interactive sessions running concurrently (e.g., On a server (2012), multiple robots can run simultaneously, each in its own... A unique username is used in the session. This is referred to as the aforementioned HD bot.

[0053] Agent 214 is also responsible for sending the robot's status (e.g., periodically sending "heartbeat" messages indicating that the robot is still operating) and downloading the required version of the package to be executed. In some embodiments, communication between agent 214 and controller 230 is always initiated by agent 214. In the case of notification, agent 214 may open a WebSocket channel that controller 230 will later use to send commands to the robot (e.g., start, stop, etc.).

[0054] On the server side, the system includes: a presentation layer (web application 232, Open Data Protocol (OData) Representation State Transition (REST) ​​Application Programming Interface (API) endpoint 234, and notification and monitoring 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240 and indexer server 250). The controller 230 includes the web application 232, the OData REST API endpoint 234, the notification and monitoring 236, and the API implementation / business logic 238. In some embodiments, most actions performed by the user (e.g., via browser 220) in the interface of the controller 230 are executed by calling various APIs. Such actions may include, but are not limited to: starting a job on a robot, adding / removing data from a queue, scheduling jobs to run unattended, etc., without departing from the scope of the invention. The web application 232 is the visual layer of the server platform. In this embodiment, the web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the invention. In this embodiment, the user interacts with a webpage from web application 232 via browser 220 to perform various actions to control the controller 230. For example, the user can create robot groups, assign packages to robots, analyze logs for each robot and / or each process, start and stop robots, and so on.

[0055] In addition to web application 232, the controller 230 also includes a service layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the invention. The REST API is consumed by web application 232 and agent 214. In this embodiment, agent 214 is a supervisor of one or more bots on a client computer.

[0056] In this embodiment, the REST API covers configuration, logging, monitoring, and queuing functions. In some embodiments, the configuration endpoint can be used to define and configure application users, permissions, bots, assets, deployments, and environments. For example, the logging REST endpoint can be used to log various information, such as errors, explicit messages sent by bots, and other environment-specific information. The deployment REST endpoint can be used by bots to query the packaged version that should be executed if a start job command is used in the controller 230. The queuing REST endpoint can be responsible for queue and queue item management, such as adding data to the queue, obtaining transactions from the queue, setting the status of transactions, and so on.

[0057] The monitoring REST endpoint can monitor web application 232 and agent 214. The notification and monitoring API 236 can be a REST endpoint used to register agent 214, deliver configuration settings to agent 214, and send / receive notifications from the server and agent 214. In some embodiments, the notification and monitoring API 236 can also use WebSocket communication.

[0058] In this embodiment, the persistence layer includes a pair of servers: a database server 240 (e.g., an SQL server) and an indexer server 250. In this embodiment, the database server 240 stores configurations for robots, robot groups, associated processes, users, roles, schedules, etc. In some embodiments, this information is managed via a web application 232. The database server 240 can manage queues and queue items. In some embodiments, the database server 240 can store messages logged by the robot logs (in addition to or instead of the indexer server 250).

[0059] Indexer server 250, which is optional in some embodiments, stores and indexes information recorded by the robot logs. In some embodiments, indexer server 250 can be disabled via configuration settings. In some embodiments, indexer server 250 uses... It is an open-source full-text search engine project. Messages logged by the robot (e.g., activities using log messages or write lines) can be sent to the indexer server 250 via (multiple) logging REST endpoints, where they are indexed for future use.

[0060] Figure 3 This is an architectural diagram illustrating the relationship 300 between a designer 310, activities 320, 330, and a driver 340 according to an embodiment of the present invention. As described above, the developer uses the designer 310 to develop workflows executed by a robot. The workflow may include user-defined activities 320 and UI automation activities 330. Some embodiments are capable of identifying non-textual visual components in an image, referred to herein as computer vision (CV). Some CV activities involving such components may include, but are not limited to, clicking, typing, obtaining text, hovering, element presence, refreshing range, highlighting, etc. Clicking in some embodiments uses, for example, CV, optical character recognition (OCR), fuzzy text matching, and multi-anchor to identify an element and click it. Typing can use the above-described methods to identify elements and type within them. Obtaining text can use OCR to identify the location of specific text and scan it. Hovering can identify an element and hover over it. Element presence can use the techniques described above to check if an element is present on the screen. In some embodiments, there may be hundreds or even thousands of activities that can be implemented in the designer 310. However, any number and / or type of activities may be available without departing from the scope of the invention.

[0061] UI automation activities 330 are a subset of specific, lower-level activities (e.g., computer vision activities) written in lower-level code and facilitating interaction with the application through the UI layer. In some embodiments, UI automation activities 300 may simulate user input, such as through window messages. UI automation activities 330 facilitate these interactions via drivers 340 that allow the robot to interact with the desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc.

[0062] Driver 340 can interact with the OS at a low level, searching for hooks, monitoring keys, etc. These facilitate communication with... Integration of technologies such as [unspecified technologies]. For example, the "click" activity performs the same function via driver 340 in these different applications.

[0063] Figure 4 This is an architectural diagram illustrating an RPA system 400 according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be or include Figure 1 and / or Figure 2 The RPA system 100 and / or 200. The RPA system 400 includes multiple client computing systems 410 that operate the robot. The computing systems 410 are capable of communicating with the command computing system 420 via a web application running on them. The command computing system 420 is then capable of communicating with a data server 430 and an optional indexer server 400.

[0064] about Figure 1 and Figure 3 It should be noted that although web applications are used in these embodiments, any suitable client and / or server software may be used without departing from the scope of the invention. For example, the controller may run a server-side application that communicates with a non-web-based client software application on a client computing system.

[0065] Figure 5 This is an architectural diagram illustrating a computing system 500 configured to support inter-session automation for RPA robots according to an embodiment of the present invention. In some embodiments, the computing system 500 may be one or more of the computing systems depicted and / or described herein. The computing system 500 includes a bus 505 or other communication mechanism for communicating information, and multiple processors 510 coupled to the bus 505 for processing information. The multiple processors 510 may be any type of general-purpose or special-purpose processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The multiple processors 510 may also have multiple processing cores, and at least some of the cores may be configured to perform specific functions. Multiple parallel processing may be used in some embodiments. In some embodiments, at least one of the multiple processors 510 may be a neuromorphic circuit including processing elements that simulate biological neurons. In some embodiments, the neuromorphic circuit may not require typical components of a von Neumann computing architecture.

[0066] The computing system 500 also includes a memory 515 for storing information and instructions to be executed by the processor(s) 510. The memory 515 may comprise any combination of: random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage devices such as disks or optical discs, or any other type of non-transitory computer-readable medium or any combination thereof. The non-transitory computer-readable medium may be any available medium accessible by the processor(s) 510, and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both.

[0067] Additionally, the computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communication network via wireless and / or wired communication. In some embodiments, the communication device 520 may be configured to use Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiplexing (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), CDMA2000, Wideband CDMA (W-CDMA), High-Speed ​​Downlink Packet Access (HSDPA), and High-Speed ​​Uplink Packet Access (HSDPA). This invention may include, without departing from the scope of the invention, one or more antennas, which may be single, arrayed, phased, switched, beamforming, beam-manipulated, combinations thereof, or any other currently existing or future-implemented communication standards and / or protocols. In some embodiments, the communication device 520 may include one or more antennas, which may be single, arrayed, phased, switched, beamforming, beam-manipulated, combinations thereof, or any other antenna configuration without departing from the scope of the invention.

[0068] Multiple processors 510 are also coupled via bus 505 to a display 525 for displaying information to a user, such as a plasma display, liquid crystal display (LCD), light-emitting diode (LED) display, field emission display (FED), organic light-emitting diode (OLED) display, flexible OLED display, flexible substrate display, projection display, 4K display, high-definition display, retina display, etc. The display 525 may be configured as a touch (haptic) display, an in-plane switching (IPS) display, or any other suitable display. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, etc., using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse recognition, suppressed total internal reflection, etc. Any suitable display device and haptic I / O may be used without departing from the scope of the invention.

[0069] Keyboard 530 and cursor control devices 535 (such as a computer mouse, touchpad, etc.) are also coupled to bus 505 to enable a user to interface with computing system 500. However, in some embodiments, a physical keyboard and mouse may not be present, and the user may interact with the device solely through display 525 and / or touchpad (not shown). Any type and combination of input devices can be considered as a design choice. In some embodiments, no physical input devices and / or display are present. For example, a user may interact remotely with computing system 500 via another computing system with which they communicate, or computing system 500 may operate autonomously.

[0070] Memory 515 stores software modules that provide functionality when executed by processor(s) 510. These modules include an operating system 540 for computing system 500. The modules also include an inter-session automation module 545 configured to perform all or part of the processes described herein, or derivatives thereof. Computing system 500 may include one or more additional functionality modules 550 containing additional features.

[0071] Those skilled in the art will recognize that "system" can be embodied as a server, embedded computing system, personal computer, console, personal digital assistant (PDA), cellular phone, tablet computing device, quantum computing system, or any other suitable computing device, or combination of devices, without departing from the scope of the invention. Presenting the foregoing functionality as being performed by a "system" is not intended to limit the scope of the invention in any way, but rather to provide one example of many embodiments of the invention. In fact, the methods, systems, and apparatuses disclosed herein can be implemented in both local and distributed forms consistent with computing technologies including cloud computing systems.

[0072] It should be noted that some of the system features described in this specification have been presented as modules to more specifically emphasize their implementation independence. For example, modules may be implemented as hardware circuits containing custom VLSI circuits or gate arrays, off-the-shelf semiconductors (such as logic chips, transistors, or other discrete components). Modules may also be implemented in programmable hardware devices such as field-programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, and so on.

[0073] The module may also be implemented, at least in part, as software for execution by various types of processors. The identified unit of executable code may, for example, comprise one or more physical or logical blocks of computer instructions that may be organized, for example, as objects, procedures, or functions. However, the executable elements of the identified module do not need to be physically located together but may comprise different instructions stored in different locations that, when logically linked together, include the module and achieve the stated purpose of the module. Additionally, the module may be stored on a computer-readable medium, such as a hard disk drive, flash device, RAM, magnetic tape, and / or any other such non-transient computer-readable medium for storing data without departing from the scope of the invention.

[0074] In practice, executable code modules can be a single instruction or many instructions, and can even be distributed across several different code segments, different programs, and across several memory devices. Similarly, operational data can be identified and illustrated within the modules herein and can be represented in any suitable form and organized within any suitable type of data structure. Operational data can be collected as a single dataset or can be distributed across different locations, including different storage devices, and can exist at least in part simply as electronic signals on a system or network.

[0075] As described above, in supervised automation, robots and users work in parallel on the same computing system. Since robots in RPA often interact with computing systems in similar ways (e.g., generating mouse clicks and key events, simulating these events via APIs (e.g., using window messages), some embodiments create one or more robot sessions to host and run supervised automation within them. Unlike existing RPA systems, users benefit from the ability to interact with their computing system while (multiple) robots are running supervised automation in (multiple) robot sessions. In some embodiments, users can still monitor what the robot is doing and interact with it through (multiple) hosted automation windows used for (multiple) robot sessions. This effectively evolves RPA robots from simply performing tasks faster and more reliably than human users into true digital assistants capable of performing work in parallel with users, providing even greater productivity gains. In some embodiments, the user and (multiple) robot sessions may be running on a remote machine controlled by the user's computing system.

[0076] In some embodiments, the RPA robot may be running on the user's computing system and via a remote runtime (e.g., via UiPath remote runtime). TM Drives remote computing systems. UiPath remote runtime TM It facilitates remote applications (such as Citrix Virtual Applications and Desktops) TM) or desktop and dedicated Extensions (e.g., for use) of Extend or for Remote desktop Components for communication between extensions. UiPath remote runtime. TM Collect information about the targeted UI elements of the remote application and send that information to the corresponding extension so that the selector is generated directly in the UI Explorer. TM In some embodiments, the robot may use Microsoft's AppV technology, which "virtually delivers" client applications to the machine.

[0077] As previously stated, in some embodiments, both the user and the robot are interacting with the same application instance and file system. Figure 6 The illustration shows some applications and file systems 660 of a user computing system 600 accessed by a user in user session 610 and by an RPA robot in robot session 620 according to an embodiment of the present invention. For example, Figure 6 As can be seen, the web browser application 630, the spreadsheet application 640, and the email application 650 can be accessed by both the user session 610 and the manually managed automated robot session 620. The user session 610 and the manually managed automated robot session 620 can interact with the web browser application 630 and the email application 650 simultaneously.

[0078] However, when the robot is with When using the same file in a spreadsheet application (e.g., the user cannot interact with the application (e.g., the user may only be able to open a "read-only" view or the user may be completely prevented from opening the file). However, if automation is using the Office 365 API to interact with the user... Such restrictions do not apply to spreadsheets hosted on a web platform and accessed by users through a web interface. For example, Google... It is entirely delivered over the network and supports concurrent access. Users can receive messages indicating that the spreadsheet application 640 is locked, is being accessed by another "user" (i.e., a robot), etc. Both the user session 610 and the manually managed automated robot process 620 can also interact with the file system 660. Therefore, changes made in the application by the robot and the user through their respective sessions will be made as if they were made by a single user, rather than making separate versions of the application and file system apply to the user and the robot respectively.

[0079] Figure 7A-J is a screenshot illustrating an example of completing a form in a user session using an RPA bot running in a client session and a user session bot or another application or process running in a user session, according to an embodiment of the present invention. Figure 7A The image shows user session window 700, where the user can interact with applications in the UI, and no robot is currently running. The robot tray icon 710 is visible in the lower right corner of user session window 700.

[0080] exist Figure 7B In this process, the user launches a web browser and accesses the invoice creation page 720. Figure 7C In this scenario, the user pulls up the robot tray 712 (e.g., by clicking the robot tray icon 710) and selects the user session robot option 714 to execute on his or her computing system within a user session. After selecting the user session robot option 714, the user session robot window 730 appears. See also Figure 7D In this embodiment, the user session bot only displays status messages. However, the user session bot can have any desired workflow functionality without departing from the scope of the invention. In some embodiments, the user session bot automatically launches the client session bot, or an option is selected to run the user session bot to automatically launch two bots in their respective sessions.

[0081] In some embodiments, no user chatbot window is displayed, and the user chatbot may run in the background. In some embodiments, another application or process performs the functionality of the user chatbot. This functionality may include, but is not limited to, communicating with one or more client chatbots, retrieving data provided by the client chatbot(s), interacting with an application running in the user session to input retrieved information, etc. In some embodiments, instead of using a user chatbot, another application, or another process, the input is performed by the application itself, which is supposed to input the information retrieved by the client chatbot. In some embodiments, this may be done without simulating mouse clicks, key presses, etc.

[0082] exist Figure 7E In this scenario, the user pulls up the robot tray 712 and selects the client session robot option 716 to execute on his or her computing system in a client session. After selecting the client session robot option 716, the client session window 740 for the robot session appears as a child window on the screen, as shown below. Figure 7FAs shown. However, in some embodiments, the client session window 740 is not displayed, and the client session bot runs within the client session but is not visible to the user. In this embodiment, the client session bot window 742 for the client session bot is automatically launched within the client session window 740 and includes a button 744 to retrieve form data for the webpage 720 in the user session.

[0083] In some embodiments, the robot session window is not displayed, and the client session robot can be automatically started, operated, and closed without being visible to the user. In some embodiments, the client session robot can close its session after completing its workflow. In some embodiments, instead of starting from the robot tray, the client session can be initiated, and the client session robot can be started and operated without using the robot tray 712 (e.g., due to the user clicking a button in the application of the main session).

[0084] Go to Figure 7G After the user clicks button 744, the client-side chatbot begins retrieving form data for webpage 720. The data retrieved by the client-side chatbot may be provided to the user chatbot via IPC, stored in a spreadsheet, an object-oriented database, a plain text file, etc., or otherwise conveyed to or made available to the user chatbot. After the client-side chatbot completes its workflow, a message is displayed in the client-side chatbot window 742. See also... Figure 7H The user chatbot then begins populating the fields of webpage 720 with the form data retrieved by the client chatbot. The text entered by the user chatbot is visible to the user as long as webpage 720 is not covered by another window or minimized. While the user chatbot completes the form, the user can interact with other applications and remain productive.

[0085] Go to Figure 7I As can be seen from the background, the user chatbot has completed the form fields on webpage 720. The user can then close the client session window 740 and the user chatbot window 730. Either the user chatbot or the client chatbot can automatically close the client session, or the client session window 740 can remain open. The user can then submit the completed form. See also... Figure 7J .

[0086] Figure 8This is a flowchart illustrating a process 800 for inter-session automation according to an embodiment of the present invention. The process begins at 810 by launching a user session window. For example, this could be a main window associated with an operating system running on the user's computing system. Then, at 820, a user session process (e.g., an RPA bot, application, another process, etc.) is initiated within the user session. At 830, a client session is launched, and at 840, a client session bot is initiated within the client session. In some embodiments, for example, the client session may be launched in response to the client session bot being initiated or otherwise launched. The client session bot then executes its workflow at 850. In some embodiments, data retrieved and / or actions taken by the client session bot may be provided to or made available to the user session process. The user session process then accesses the data generated by the client session bot at 860 and, at 870, interacts with the appropriate application(s) based on that data (e.g., a webpage, spreadsheet, ERP application, sales application, etc.). In some embodiments, the application(s) interacting with may be the user session process(s) itself. In some embodiments, the client session may be automatically terminated at 880 after the client session bot has completed its execution.

[0087] In some embodiments, client sessions can be created via the operating system's sub-session API. (Provided by the operating system) Terminating a service sub-session or another sub-session may be used in some embodiments to create a second session without departing from the scope of the invention. Robotic tray applications (e.g., A robot agent desktop application, or another application configured to launch (multiple) robots, can then use the creation process in the operating system with appropriate parameters to launch the robot process in that sub-session. The robot tray application or other suitable application can then communicate with the client robot process using an appropriate protocol (e.g., a protocol established on a named pipeline).

[0088] In some embodiments, communication between two sessions with the client robot can be accomplished using IPC protocols. These protocols facilitate communication via networks, pipelines, Component Object Models (COM), Remote Procedure Calls (RPC), sockets, etc. Where supported, appropriate session creation mechanisms and IPC protocols may also be used on other operating systems. When a user clicks a button on the robot tray or causes another application to initiate such a function (e.g., by clicking a button), the robot tray application or other suitable application can use IPC protocols to send the command to the client session robot process. The client session robot can also send status notifications back to the robot tray application or other suitable process via IPC protocols (e.g., indicating that the robot is starting, running, paused, etc.).

[0089] Figure 9 This is a flowchart illustrating an inter-session automation process 900 according to an embodiment of the present invention, in which one robot runs in a user session and another robot runs in a client session. The process begins at 910 with the execution of the user session robot workflow. When the user session robot reaches an activity in its workflow that requires the client robot to perform at 920, at 930, the user session robot instructs the client session robot to execute its workflow and waits for it to complete.

[0090] At 940, the client-side chatbot completes its workflow and notifies the user-side chatbot. At 950, the user-side chatbot then uses the result of the client-side chatbot's execution to complete the remainder of its workflow. In some embodiments, the user-side chatbot and the client-side chatbot jointly complete a single logical workflow.

[0091] In some embodiments, multiple client-side chatbots may be used, and at least some of them may be used in different client sessions. This allows user chatbots to pass portions of workflow activities to client chatbots for completion. Figure 10 The illustration depicts scenario 1000 according to an embodiment of the present invention, wherein a portion of the workflow is completed by a user session robot, while other portions of the workflow are completed by client session robots. User session 1010 includes user session robot 1012 and application 1014. First client session 1020 includes client session robot 1022, second client session 1030 includes two client session robots 1032 and 1034, and third client session 1040 includes client session robot 1042. For example, user session robot 1012 may communicate with and control the operation of client session robots 1022, 1032, 1034, and 1042 via IPC. It should be noted that in some embodiments, the user session robot continues to execute its workflow while the client session robots(s) complete their respective workflows.

[0092] Figure 11 This is a flowchart 1100 illustrating the execution of a multi-robot assembly workflow between a user session robot U1 and a pair of client session robots C1 and C2 according to an embodiment of the present invention. C1 and C2 may be in the same client session or different client sessions. U1 begins executing its workflow and arrives at an activity that requires C1 to complete its workflow. U1 causes C1 to execute its workflow, and the calling activity waits. During this time, in some embodiments, U1 may perform other tasks.

[0093] After C1 completes its workflow, U1 continues execution until it reaches an activity that requires C2 to complete its workflow. U1 then causes C2 to execute its workflow, and the calling activity waits. In some embodiments, U1 may again perform other tasks while waiting for C2.

[0094] After C2 completes its workflow, U1 continues execution until it reaches an activity that again requests C1 to complete its workflow. This can be the same workflow activity as the previously executed activity, or it can be a different workflow or set of activities. U1 causes C1 to execute its workflow, and the calling activity waits. In some embodiments, U1 may again perform other tasks while waiting for C1. After C1 completes its workflow, U1 continues execution until U1's workflow is complete.

[0095] According to an embodiment of the present invention, Figure 8 , 9 The process steps performed in section 11 can be executed by a computer program, which is coded for execution by (multiple) processors. Figure 8 , 9 Instructions for at least a portion of the processes(s) described in 11. A computer program may be embodied on a non-transient computer-readable medium. The computer-readable medium may be, but is not limited to, hard disk drives, flash devices, RAM, magnetic tape, and / or any other such medium or combination of media for storing data. A computer program may include instructions for controlling (e.g.,) processor(s) of a computing system. Figure 5 The computing system 500 (multiple) processors 510) implement Figure 8 , 9 All or part of the coded instructions in the process steps described in Section 11 may also be stored on a computer-readable medium.

[0096] Computer programs can be implemented in hardware, software, or a hybrid manner. A computer program can consist of modules that communicate operationally with each other and are designed to transmit information or instructions for display. A computer program can be configured to operate on a general-purpose computer, an ASIC, or any other suitable device.

[0097] It will be readily understood that the components of the various embodiments of the invention as generally described and illustrated in the accompanying drawings can be arranged and designed in a wide variety of different configurations. Therefore, the detailed description of the embodiments of the invention as illustrated in the drawings is not intended to limit the scope of the claimed invention but merely to represent selected embodiments of the invention.

[0098] The features, structures, or characteristics of the invention described throughout this specification may be combined in one or more embodiments in any suitable manner. For example, references to "certain embodiments," "some embodiments," or similar language throughout this specification mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Therefore, the appearance of the phrases "in some embodiments," "in some embodiments," "in other embodiments," or similar language throughout the specification does not necessarily refer to all of the same set of embodiments, and the described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner.

[0099] It should be noted that references to features, advantages, or similar language throughout this specification do not imply that all features and advantages achievable using the invention should be present in any single embodiment of the invention. Rather, language relating to features and advantages is to be understood as meaning that a particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Therefore, discussions of features and advantages and similar language throughout this specification may, but do not necessarily, refer to the same embodiments.

[0100] Furthermore, the above-described features, advantages, and characteristics of the present invention can be combined in one or more embodiments in any suitable manner. Those skilled in the art will recognize that the invention can be practiced without one or more of the specified features or advantages in a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that are not present in all embodiments of the invention.

[0101] It will be readily understood by those skilled in the art that the invention as described above can be practiced using steps in a different order and / or using hardware elements in a different configuration than those disclosed. Therefore, although the invention has been described based on preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative structures will be readily apparent while remaining within the spirit and scope of the invention. For the purpose of defining the scope of the invention, reference should therefore be made to the appended claims.

Claims

1. A computer-implemented method, comprising: Initiate the user session procedure within the user session; Start the first client session robot process automation RPA robot in the first client session; Start the second client session RPA robot in the first client session or the second client session; The workflow of the first client session RPA robot is executed by the first client session RPA robot in the first client session; The workflow of the second client session RPA robot is executed by the second client session RPA robot in the first client session or the second client session; The first client session RPA robot enables the execution result of the workflow of the first client session RPA robot to be used in the user session process; The second client session RPA robot enables the execution results of the workflow of the second client session RPA robot to be used in the user session process; as well as The result of the execution of the workflow of the first client session RPA robot by the user session process interacts with the application in the user session.

2. The computer-implemented method according to claim 1, wherein... The interaction with the application during the user session includes entering information into a form, and The result of the execution of the RPA workflow includes data used by the user session process to input the information into the form.

3. The computer-implemented method of claim 1, wherein the user session process includes an RPA robot.

4. The computer-implemented method of claim 1, wherein the first client session RPA robot is visible in a window associated with the first client session.

5. The computer-implemented method of claim 1, wherein the first client session RPA robot is invisible.

6. The computer-implemented method of claim 1, wherein the user session process waits for the first client session RPA robot to complete the execution of the workflow of the first client session RPA robot.

7. The computer-implemented method of claim 1, wherein the user session process performs other tasks while waiting for the first client session RPA robot to complete the execution of the workflow of the first client session RPA robot.

8. The computer-implemented method of claim 1, wherein the first client session and the first client session RPA robot are initiated by the user session procedure.

9. The computer-implemented method according to claim 1, further comprising: After the first client session RPA robot completes the execution of the workflow of the first client session RPA robot, the first client session RPA robot and the first client session are automatically closed.

10. The computer-implemented method of claim 1, wherein the user session process includes an RPA robot, and the user session RPA robot and the first client session RPA robot jointly perform the execution of a larger workflow including the workflow of the first client session RPA robot.

11. A computer-implemented method, comprising: The workflow of the client-side session RPA robot is executed by the user-side session robot process automation RPA robot via inter-process communication (IPC) commands to the client-side session RPA robot. The workflow of the client-side RPA robot is executed by the client-side RPA robot in the client-side session; The client-side RPA robot enables the execution results of the workflow of the client-side RPA robot to be used by the user-side RPA robot; The user session RPA robot receives the result of the execution of the workflow from the client session RPA robot; as well as The results of the execution of the workflow by the user session RPA robot using the client session RPA robot are interacted with the application on the computing system.

12. The computer-implemented method according to claim 11, wherein... The interaction with the application on the computing system includes entering information into a form, and The result of the execution of the RPA workflow includes data used by the user session RPA robot to input the information into the form.

13. The computer-implemented method of claim 11, wherein the user session RPA robot waits for the client session RPA robot to complete the execution of the workflow of the client session RPA robot.

14. The computer-implemented method of claim 11, wherein the user session RPA robot performs other tasks while waiting for the client session RPA robot to complete the execution of the workflow of the client session RPA robot.

15. The computer-implemented method of claim 11, wherein the user session RPA robot and the client session RPA robot jointly perform the execution of a larger workflow including the workflow of the client session RPA robot.

16. A computer-implemented method, comprising: The workflow of the client-side session RPA robot is executed by the user-side session robot process automation RPA robot via inter-process communication (IPC) commands to the client-side session RPA robot. The workflow of the client-side RPA robot is executed by the client-side RPA robot in the client-side session; The client-side RPA robot enables the execution results of the workflow of the client-side RPA robot to be used by the user-side RPA robot; The user session RPA robot receives the result of the execution of the workflow from the client session RPA robot; as well as The results of the execution of the workflow by the user session RPA robot using the client session RPA robot are interacted with the application on the computing system, wherein... The user session RPA robot and the client session RPA robot work together to execute a larger workflow that includes the client session RPA robot.

17. The computer-implemented method according to claim 16, wherein... The interaction with the application during the user session includes entering information into a form, and The result of the execution of the RPA workflow includes data used by the user session RPA robot to input the information into the form.

Images