Robot Browser Embed

Abstract

One or more robots can be called from one or more applications for Robotic Process Automation (RPA). The process of calling a robot from an application may include launching an application from a computing system to call a robot link embedded within the application. The process may also include initiating a port discovery process from the application to identify a port, port details, and a token. The process may also include: generating a randomized code by the application, and calling a consent application that requests approval from a user of the computing system to call the robot from the application. The process may also include registering the randomized code with a local listener module and passing user information and a token to a global listener module. The process may also include: receiving a token and a port identifier from the global listener module, thereby allowing the application to authenticate itself to the robot and communicate with the robot, thereby completing the robot call processing.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. non-provisional patent application No. 16 / 706,581, filed on December 6, 2019, which claims priority under 35 U.S.C. §119 to Indian Patent Application No. 201911041286, filed on October 11, 2019. The entire contents of the technical solutions of these prior-filed applications are incorporated herein by reference. Technical Field

[0003] The present invention relates generally to Robotic Process Automation (RPA), and more particularly to invoking robotic processes for RPA from one or more applications. Background Art

[0004] Robots running on a local computer are typically invoked from the Robot tray ("tray") or the Commander Application Programming Interface (API). In either case, users can invoke a robot through the Robot tray or the Commander API. However, when a user is working on one or more different applications, the user cannot physically invoke the robot(s).

[0005] Therefore, it may be beneficial to invoke one or more robots without reaching the robot tray or commander API. Summary of the Invention

[0006] Certain embodiments of the present invention may provide solutions to problems and needs in the art that have not yet been fully identified, understood, or addressed by RPA technology. For example, some embodiments of the present invention involve invoking one or more robotic processes from one or more applications.

[0007] In one embodiment, a computer-implemented method includes launching an application from a computing system to invoke a robot link embedded within the application. The method also includes initiating a port discovery process from the application to identify a port, port details, and a token. The method also includes generating a randomized code by the application and invoking a consent application requesting approval from a user of the computing system to invoke the robot from the application; and registering the randomized code with a local listener module and passing the user information and the token to a global listener module. The method also includes receiving the token and the port identification from the global listener module, allowing the application to authenticate itself with the robot and communicate with the robot, thereby facilitating the robot interaction process.

[0008] In another embodiment, a computer program is embodied on a non-transitory computer-readable medium. The computer program is configured to cause at least one processor to: launch an application from the computing system to invoke a robot link embedded within the application. The computer program is further configured to cause the at least one processor to initiate a port discovery process from the application to identify a port, port details, and a token. The computer program is further configured to cause the at least one processor to generate a randomized code from the application and to invoke a consent application that requests approval from a user of the computing system to invoke the robot from the application. Additionally, the computer program is further configured to cause the at least one processor to register the randomized code with a local listener module and pass user information and a token to a global listener module; and to receive the token and port identification from the global listener module, allowing the application to authenticate itself with the robot and communicate with the robot, thereby facilitating a robot interaction process.

[0009] In yet another embodiment, a system includes a memory storing computer program instructions; and at least one processor configured to execute the computer program instructions, the instructions configured to cause the at least one processor to: launch an application from the computing system to invoke a robot link embedded within the application. The instructions are further configured to cause the at least one processor to initiate a port discovery process from the application to identify a port, port details, and a token; generate a randomized code by the application and invoke a consent application requesting approval from a user of the computing system to invoke the robot from the application. The instructions are further configured to cause the at least one processor to register the randomized code with a local listener module and pass user information and the token to a global listener module; and receive the token and port identification from the global listener module, thereby allowing the application to authenticate itself with the robot and communicate with the robot, thereby facilitating the robot interaction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to readily understand the advantages of certain embodiments of the present invention, a more particular description of the invention, briefly described above, will be rendered by reference to specific embodiments that are illustrated in the accompanying drawings. While it should be understood that these drawings depict only typical embodiments of the invention and, therefore, should not be considered to be limiting of the scope of the invention, the invention will be described and illustrated with additional specificity and detail through the use of the accompanying drawings, in which:

[0011] Figure 1 is an architectural diagram illustrating a robotic process automation (RPA) system according to an embodiment of the present invention.

[0012] Figure 2 is an architectural diagram illustrating an RPA system deployed according to an embodiment of the present invention.

[0013] Figure 3 is an architectural diagram illustrating the relationship among designers, activities, and drivers according to an embodiment of the present invention.

[0014] Figure 4 FIG. 4 is an architectural diagram illustrating an RPA system according to an embodiment of the present invention.

[0015] Figure 5 is an architectural diagram illustrating a computing system configured to invoke a robot from an application according to an embodiment of the present invention.

[0016] Figure 6 is a block diagram illustrating a desktop configuration for communicating between one or more sessions and a robotic tray according to an embodiment of the present invention.

[0017] Figure 7 is a flow chart illustrating a port discovery process according to an embodiment of the present invention.

[0018] Figure 8 is a flowchart illustrating an authentication process according to an embodiment of the present invention.

[0019] Figure 9 is a flowchart illustrating a robot service interaction process according to an embodiment of the present invention.

[0020] Figure 10 is a flow chart illustrating a process for facilitating a robotic interaction process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] Some embodiments involve invoking one or more robots from one or more applications to perform Robotic Process Automation (RPA). An application can be defined as any application on a computer that can be embedded in a browser or run Hypertext Markup Language (HTML) and JavaScript (JS) code. In other words, an application is not limited to a "web-based application" and can be any application that can be used to invoke a robot. For illustrative purposes, the term "application" will be used. In some embodiments, an application that can be used to invoke a robot may be from an untrusted domain. In some embodiments, a robot link is embedded in one or more applications, allowing one or more robots to be invoked. In one embodiment, a user of a computing system launches an application to invoke the robot link embedded within the application. From the application, a port discovery process can be initiated to identify a port, port details, and a token. The application can then generate a randomized code and invoke a consent application, requesting the computing system user's approval, to invoke the robot from the application. The randomized code can be registered with a local listener module, and the user information and token can be forwarded to a global listener module. The token and port identification can be received from the global listener module, allowing the application to authenticate itself with the robot and communicate with the robot.

[0022] Figure 1 The following is an architectural diagram illustrating an RPA system 100 according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that allows developers to design and implement workflows. Designer 110 provides solutions for integrating applications and automating third-party applications, managing information technology (IT) tasks, and managing business IT processes. Designer 110 supports the development of automation projects, which are graphical representations of business processes. In short, designer 110 supports the development and deployment of workflows and bots.

[0023] Automation projects enable the automation of rule-based processes by providing developers with control over the execution order and relationships between a set of custom steps (defined herein as "activities") developed in a workflow. One commercial example of an embodiment of the designer 110 is UiPath Studio TM Each activity may include an action, such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0024] Some types of workflows may include, but are not limited to, sequences, flow charts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suitable for linear processes, allowing for the flow from one activity to another without cluttering the workflow. Flow charts may be particularly suitable for more complex business logic, enabling the integration of decisions made in more diverse ways through multiple branching logic operators and the connection of activities. FSMs may be particularly suitable for larger workflows. FSMs may use a limited number of states in their execution, which are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suitable for determining workflow behavior and for debugging processes when encountering execution errors.

[0025] Once a workflow is developed in the designer 110, the execution of the business process is orchestrated by a conductor 120, which orchestrates one or more robots 130 that execute the workflow developed in the designer 110. One commercial example of an embodiment of the conductor 120 is the UiPath Orchestrator TM The director 120 supports the management of the creation, monitoring, and deployment of resources in the environment. The director 120 can serve as an integration point or one of the aggregation points with third-party solutions and applications.

[0026] The commander 120 can manage a fleet of robots 130, connecting and executing them from a centralized point. The types of robots 130 that can be managed include, but are not limited to, attended robots 132, unattended robots 134, development robots (similar to unattended robots 134, but used for development and testing purposes), and non-production robots (similar to attended robots 132, but used for development and testing purposes). Attended robots 132 are triggered by user events and operate on the same computing system as humans. Attended robots 132 can be used with the commander 120 to centralize process deployment and recording media. Attended robots 132 can assist human users with various tasks and can be triggered by user events. In some embodiments, processes cannot be started from the commander 120 on this type of robot and / or cannot be run from a locked screen. In some embodiments, attended robots 132 can only be started from the robot tray or from a command prompt. In some embodiments, attended robots 132 should only be run under human supervision.

[0027] Unsupervised robots 134 run in an unsupervised manner in a virtual environment and can automate many processes. Unsupervised robots 134 can be responsible for remote execution, monitoring, scheduling, and providing support for work queues. In some embodiments, debugging for all robot types can be run in the designer 110. Both supervised and unsupervised robots can automate various systems and applications, including but not limited to mainframes, web applications, VMs, enterprise applications (e.g., and computing system applications (such as desktop and notebook applications, mobile device applications, wearable computer applications, etc.).

[0028] The commander 120 may have various capabilities including, but not limited to, provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between robots 130 and the commander 120 (e.g., a web application). Development may include ensuring that package versions are correctly delivered to assigned robots 130 for execution. Configuration may include maintaining and delivering robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs into a database (e.g., a SQL database) and / or another storage mechanism (e.g., a SQL database). It provides the ability to store and quickly query large data sets.) Director 120 can provide interconnectivity by serving as a centralized point of communication for third-party solutions and / or applications.

[0029] The robot 130 is an execution agent that runs the workflow built into the designer 110. One commercial example of some embodiments of the robot(s) 130 is UiPath Robots TM In some embodiments, the robot 130 is installed by default with Microsoft As a result, such robots 130 can open interactive session, and has Permissions for services.

[0030] In some embodiments, a robot 130 can be installed in user mode. This means that for such a robot 130, it has the same permissions as the user who installed the given robot 130. This feature can also be used for high-density (HD) robots, which ensure that each machine is fully utilized to its maximum potential. In some embodiments, any type of robot 130 can be deployed in an HD environment.

[0031] In some embodiments, the robot 130 is divided into several components, each dedicated to a specific automation task. In some embodiments, the robot components include but are not limited to the robot service managed by the SCM, the user mode robot service, the executor, the agent, and the command line. The SCM starts a session and acts as a delegate between the director 120 and the execution host (i.e., the computing system on which the robot 130 is executed). These services are trusted by the robot 130 and manage the credentials of the robot 130. The console application is launched by the SCM under the local system.

[0032] In some embodiments, user-mode robot services manage and monitor The user mode robot service can be trusted by the robot 130 and manage the credentials of the robot 130. In the case that the SCM managed robot service is not installed, Applications can be launched automatically.

[0033] The actuator can be The executor can execute the workflow. The executor can know the dots per inch (DPI) setting of each monitor. The agent can be A Presentation Foundation (WPF) application that displays available jobs in a system tray window. An agent can be a client of the service. The agent can request to start or stop jobs and change settings. A command line is a client of the service. The command line is a console application that can request to start jobs and wait for their output.

[0034] Keeping the components of the robot 130 generally separate as explained above helps developers, support users, and computing systems more easily run, identify, and track what each component is executing. Special behaviors can be configured on a per-component basis in this way, such as setting different firewall rules for executors and services. In some embodiments, the executor can always be aware of the DPI setting of each monitor. As a result, workflows can be executed at any DPI regardless of the configuration of the computing system that created the workflow. In some embodiments, projects from the designer 110 can also be independent of the browser zoom level. In some embodiments, DPI can be disabled for applications that are DPI unaware or intentionally marked as unaware.

[0035] Figure 2 is an architectural diagram illustrating an RPA system 200 deployed according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be Figure 1 The RPA system 100 may be a part of the RPA system. It should be noted that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 210 includes an executor 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not be running on the computing system 210. The executor 212 is running the process. Figure 2 As shown in , several business items can be run simultaneously. In this embodiment, the agent 214 (e.g. The service) is the single point of contact for all executors 212. All messages in this embodiment are logged to the director 230, which further processes these messages via the database server 240, the indexer server 250, or both. Figure 1 As discussed, effector 212 may be a robotic component.

[0036] In some embodiments, the designer 216 is used to set inputs and outputs on the process. These inputs and outputs can be processed by the executor 212 via the agent 214. In addition, these inputs and outputs form a mechanism for communicating with the workflow during initialization and receiving results during or after execution completion.

[0037] In some embodiments, a robot represents an association between a machine name and a user name. A robot can manage multiple executors simultaneously. In a computing system that supports multiple interactive sessions running simultaneously (e.g., Server 2012), multiple robots can run simultaneously, each with a separate In conversation. This is referred to as the HD robot above.

[0038] The agent 214 is also responsible for communicating the robot's status (e.g., periodically sending "heartbeat" messages indicating that the robot is still functioning) and downloading the required versions of packages to be executed. In some embodiments, communication between the agent 214 and the commander 230 is always initiated by the agent 214. In a notification scenario, the agent 214 can open a WebSocket channel that is later used by the commander 230 to send commands to the robot (e.g., start, stop, etc.).

[0039] On the server side, the presentation layer (web application 232, Open Data Protocol (OData) Representational State Transfer (REST) ​​application programming interface (API) endpoint 234, and notification and monitoring 236), the service layer (API implementation / business logic 238), and the persistence layer (database server 240 and indexer server 250) are included. The commander 230 includes the web application 232, the OData REST API endpoint 234, notification and monitoring 236, and the API implementation / business logic 238. In some embodiments, most actions performed by the user in the commander 230 interface (e.g., via the browser 220) are performed by calling various APIs. Without departing from the scope of the present invention, such actions may include, but are not limited to, starting a job on a robot, adding / removing data from a queue, scheduling a job to run in an unsupervised manner, etc. The web application 232 is the visual layer of the server platform. In this embodiment, the web application 232 uses HTML and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the present invention. In this embodiment, a user interacts with a web page from web application 232 via browser 220 to perform various actions to control commander 230. For example, a user can create a group of robots, assign packages to robots, analyze logs for each robot and / or each process, start and stop robots, etc.

[0040] In addition to the web application 232, the director 230 also includes a service layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the present invention. The REST API is consumed by both the web application 232 and the agent 214. In this embodiment, the agent 214 is an overseer of one or more robots on a client computer.

[0041] In this embodiment, the REST API covers configuration, logging, monitoring, and queuing functionality. In some embodiments, the configuration endpoint can be used to define and configure application users, permissions, robots, assets, releases, and environments. The logging REST endpoint can be used to log various information, such as (for example) errors, explicit messages sent by the robot, and other environment-specific information. If the start job command is used in the commander 230, the deployment REST endpoint can be used by the robot to query the package version that should be executed. The queueing REST endpoint can be responsible for queue and queue item management, such as adding data to the queue, getting transactions from the queue, setting the status of transactions, etc.

[0042] The monitoring REST endpoint monitors the web application 232 and the agent 214. The notification and monitoring API 236 may be a REST endpoint for registering the agent 214, delivering configuration settings to the agent 214, and for sending / receiving notifications from the server and agent 214. In some embodiments, the notification and monitoring API 236 may also use WebSocket communication.

[0043] In this embodiment, the persistence layer includes a pair of servers—a database server 240 (e.g., a SQL server) and an indexer server 250. In this embodiment, the database server 240 stores configurations for robots, robot groups, associated processes, users, roles, schedules, and the like. In some embodiments, this information is managed by the web application 232. The database server 240 can manage queues and queue items. In some embodiments, the database server 240 can store messages logged by the robots (in addition to or in place of the indexer server 250).

[0044] In some embodiments, the optional indexer server 250 stores information logged by the robot and indexes the information. In some embodiments, the indexer server 250 can be disabled through a configuration setting. In some embodiments, the indexer server 250 uses It is an open source full-text search engine. Messages logged by the robot (e.g., using activities such as logging messages or writing lines) can be sent to the indexer server 250 via the logging REST endpoint(s), where they are indexed for future use.

[0045] Figure 33 is an architectural diagram illustrating the relationship 300 between a designer 310, activities 320, 330, and a driver 340 according to an embodiment of the present invention. As described above, developers use the designer 310 to develop workflows executed by a robot. The workflow may include user-defined activities 320 and UI automation activities 330. Some computer vision (CV) activities may include, but are not limited to, click, type, get text, hover, element presence, refresh range, highlight, and the like. In some embodiments, click uses, for example, CV, optical character recognition (OCR), fuzzy text matching, and multi-anchors to identify an element and then click on it. Type may use these to identify an element and the type within that element. Get text may identify the location of specific text and scan that location using OCR. Hover may identify an element and hover over it. Element presence may use the techniques described above to check whether an element is on the screen. In some embodiments, there may be hundreds or even thousands of activities that can be implemented in the designer 310. However, any number and / or type of activities may be available without departing from the scope of the present invention.

[0046] UI automation activities 330 are a subset of special lower-level activities (e.g., CV activities) written in lower-level code and facilitate interaction with the screen. UI automation activities 330 support these interactions via drivers 340 that allow the robot to interact with the desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc.

[0047] Driver 340 can interact with the OS at a low level to find hooks, monitor keys, etc. Drivers can help For example, the “click” activity plays the same role in these different applications via driver 340.

[0048] Figure 4 4 is an architectural diagram illustrating an RPA system 400 according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be or include Figure 1 and / or Figure 2 The RPA system 100 and / or 200 of the present invention is shown. The RPA system 400 includes multiple client computing systems 410 running robots. The computing systems 410 can communicate with a commander computing system 420 via web applications running on them. The commander computing system 420 can in turn communicate with a database server 430 and an optional indexer server 440.

[0049] Relative to Figure 2 and Figure 3It should be noted that although web applications are used in these embodiments, any suitable client / server software may be used without departing from the scope of the present invention. For example, the director may run a server-side application that communicates with a non-web-based client software application on a client computing system.

[0050] Figure 5 is an architectural diagram illustrating a computing system 500 configured to invoke a robot from one or more applications according to an embodiment of the present invention. In some embodiments, computing system 500 may be one or more of the computing systems depicted and / or described herein. Computing system 500 includes a bus 505 or other communication mechanism for transmitting information, and processor(s) 510 coupled to bus 505 for processing information. Processor(s) 510 may be any type of general-purpose or special-purpose processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. Processor(s) 510 may also have multiple processing cores, and at least some of the cores may be configured to perform specific functions. In some embodiments, multi-processing may be used. In certain embodiments, at least one of processor(s) 510 may be a neuromorphic circuit including processing elements that mimic biological neurons. In some embodiments, neuromorphic circuits may not require the typical components of the von Neumann computing architecture.

[0051] The computing system 500 also includes a memory 515 for storing information and instructions to be executed by the processor(s) 510. The memory 515 can be composed of any combination of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage (such as a magnetic or optical disk), or any other type of non-transitory computer-readable medium, or any combination thereof. Non-transitory computer-readable media can be any available media that can be accessed by the processor(s) 510 and can include volatile media, non-volatile media, or both. The media can also be removable, non-removable, or both.

[0052] Additionally, the computing system 500 includes a communication device 520 (such as a transceiver) to provide access to a communication network via wireless and / or wired connections. In some embodiments, the communication device 520 can be configured to use frequency division multiple access (FDMA), single carrier FDMA (SC-FDMA), time division multiple access (TDMA), code division multiple access (CDMA), orthogonal frequency division multiple access (OFDM), orthogonal frequency division multiple access (OFDMA), global system for mobile (GSM) communications, general packet radio service (GPRS), universal mobile telecommunications system (UMTS), cdma2000, wideband CDMA (W-CDMA), high speed downlink packet access (HSDC), etc., without departing from the scope of the present invention. In some embodiments, the communication device 520 may include one or more antennas that are singular, array, phased, switched, beamformed, beamsteered, combinations thereof, and / or any other antenna configurations without departing from the scope of the present invention.

[0053] The processor(s) 510 are further coupled via bus 505 to a display 525, such as a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, Display 525 may be configured as a touch (tactile) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, or the like using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse identification, frustrated total internal reflection, or the like. Any suitable display device and tactile I / O may be used without departing from the scope of the present invention.

[0054] A keyboard 530 and a cursor control device 535 (such as a computer mouse, touchpad, etc.) are further coupled to bus 505 to enable a user to interface with computing system 500. However, in some embodiments, a physical keyboard and mouse may not be present, and the user may interact with the device solely through display 525 and / or touchpad (not shown). Any type and combination of input devices may be used as a matter of design choice. In some embodiments, no physical input device and / or display is present. For example, a user may interact with computing system 500 remotely via another computing system in communication therewith, or computing system 500 may operate autonomously.

[0055] The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. The modules include an operating system 540 for the computing system 500. The modules also include a robot invocation module 545 configured to perform all or part of the processes described herein or their derivatives. The computing system 500 may include one or more additional function modules 550 that include additional functionality.

[0056] It will be appreciated by those skilled in the art that a "system" may be implemented as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device or combination of devices without departing from the scope of the present invention. Presenting the functions described above as being performed by a "system" is not intended to limit the scope of the present invention in any way, but is intended to provide an example of many embodiments of the present invention. Indeed, the methods, systems, and apparatus disclosed herein may be implemented in both localized and distributed forms consistent with computing technologies including cloud computing systems.

[0057] It should be noted that some of the system features described in this specification have been presented as modules in order to more specifically emphasize their implementation independence. For example, a module can be implemented as a hardware circuit comprising a custom very large scale integrated (VLSI) circuit or gate array, or an off-the-shelf semiconductor (such as a logic chip, transistor, or other discrete component). A module can also be implemented in a programmable hardware device such as a field programmable gate array, programmable array logic, a programmable logic device, a graphics processing unit, or the like.

[0058] Modules may also be implemented at least in part in software for execution by various types of processors. For example, an identified unit of executable code may include one or more physical or logical blocks of computer instructions, which may, for example, be organized into objects, procedures, or functions. Nevertheless, the executable file for an identified module need not be physically located in one place, but may include disparate instructions stored in different locations which, when logically combined together, comprise the module and achieve the intended purpose of the module. Additionally, a module may be stored on a computer-readable medium, which may be, for example, a hard drive, a flash memory device, RAM, a magnetic tape, and / or any other such non-transient computer-readable medium for storing data, without departing from the scope of the present invention.

[0059] In practice, a module of executable code can be a single instruction or many instructions, and can even be distributed over several different code segments, between different programs, and across multiple memory devices. Similarly, operational data can be identified and illustrated herein as within a module, and this operational data can be implemented in any suitable form and organized within a data structure of any suitable type. Operational data can be collected as a single data set, or can be distributed over different locations included on different storage devices, and can exist at least in part only as an electronic signal on a system or network.

[0060] Figure 6 6 is a block diagram illustrating a desktop configuration 600 of a computing system for communicating between one or more user sessions 602 and a robotic tray according to an embodiment of the present invention. Although a single user session 602 will be referenced in this example, multiple sessions may be executing on the desktop configuration 600.

[0061] In some embodiments, user session 602 includes an application running on a web browser 604. The application has a robot link 606 embedded (e.g., via a JavaScript (JS) standard development kit (SDK)). Robot link 606 provides the user with the ability to invoke a robot from within the application without having to access the robot tray. In some embodiments, robot link 606 is a JS-based SDK and is responsible for simplifying port discovery, authentication, interaction, and communication flow between the application and a protocol handler 608, a Hypertext Transfer Protocol (HTTP) listener 610, and an HTTP port discovery service 612. In embodiments where the application decides not to use robot link 606, the application can interact directly with protocol handler 608, HTTP listener 610, and HTTP port discovery service 612.

[0062] To invoke a bot, a user launches an application, accesses the bot through the application, and causes the bot link 606 embedded in the application to invoke the bot. A protocol handler 608, also known as a custom protocol handler application or consent application, allows the user to authenticate the user session 602. In some embodiments, the computing system 600 allows for the registration of custom protocol handlers (also known as custom URI schemes) on the machine, which can launch a specific application when called from anywhere on the machine. For example, http: / / launches a browser or mailto: / / launches an email client. In some embodiments, the protocol handler 608 can register a custom protocol handler upon installation, and the protocol used to invoke it.

[0063] The HTTP port discovery service 612, also known as the global listener module, performs port discovery in "Session 0" to provide dynamic allocation of ports. It should be understood that these embodiments are not limited to the HTTP protocol and that other communication channels or protocols may be used. In some embodiments, the HTTP port discovery service 612 operates on a well-known port on the computer system 600. This well-known port is known to the robot link 606 and is used for retrieving authentication tokens and port discovery for the robot HTTP listener 610. It should be understood that these embodiments are not limited to HTTP listeners and that other protocols besides HTTP may be used. The HTTP port discovery service 612 can be set to start automatically when the computing system 600 starts or when a user logs into the computing system 600. In this way, the HTTP port discovery service 612 is always available.

[0064] Additionally, in some embodiments, the protocol handler 608 may also start the HTTP port discovery service 612 if it is not already running. In those embodiments, the protocol handler 608 may have elevated permissions (or administrator privileges) to start HTTP port discovery. Since multiple user sessions are running on the computing system 600, the HTTP port discovery service 612 may be started in the machine session (or session 0 in Windows) to ensure that the robot HTTP listeners 610, protocol handlers 608, and robot links 606 from all user sessions can access the HTTP port discovery service 612.

[0065] In some embodiments, HTTP port discovery service 612 can run directly in user session 602. In those embodiments, HTTP port discovery service 612 and robot HTTP listener 610 are combined into a single service. Furthermore, in these embodiments, HTTP port discovery service 612 may not be needed because robot link 606 knows the well-known port of HTTP listener 610.

[0066] Returning to the user session 602, the robot HTTP listener (or local listener module) 610 facilitates communication between applications via the robot link 606 and the robot tray. The robot HTTP listener 610 is responsible for the web browser 604 interacting with the browser service 614 to run processes and send / receive data from the executor 212 of the robot 210. The robot HTTP listener 610 provides an HTTP API used by the robot link 606. Since the computing system has multiple user sessions running on it, and to provide security between different users on the same computing system 600, the robot HTTP listener 610 runs in the user session 602 and communicates with the robot service 614 on behalf of the specific user. When the user session 602 is started or started by the protocol handler 608, the robot HTTP listener 610 can begin communicating with the robot service 614 automatically. Since a single HTTP port on a computer can only be used by A single robot HTTP listener 610 is used on the server, so the robot HTTP listener 610 service starts on a random port each time to avoid conflicts when multiple user sessions are active on the same machine. At each start, the robot HTTP listener 610 registers the selected port and user session in which it is running with the HTTP port discovery service 612. This allows the HTTP port discovery service 612 to provide the port number to the robot link 606 for communication after the port discovery process.

[0067] It should be appreciated that both the HTTP port discovery service 612 and the robot HTTP listener 610 can be configured to listen on localhost or 127.0.0.1 to prevent other external machines from communicating with these services. These guarantees are provided by the operating system.

[0068] Robot service is Figure 2 The agent 214 in is responsible for managing the executor 212 as described above.

[0069] Some embodiments may include a port discovery process, an authentication process, and a robot service interaction. It should be understood that in some embodiments, both the port discovery and authentication processes may be combined into a single flow, but this need not be the case. In other embodiments, both processes may be performed separately.

[0070] It should be further appreciated that communications between the HTTP listener 610, protocol handler 608, HTTP port discovery service 612, and robot service 614 utilize authentication applications for application security and / or authentication services for service channels provided by the operating system (e.g., NTLM authentication).

[0071] Port discovery process

[0072] Figure 7 is a flow chart illustrating a port discovery process 700 according to an embodiment of the present invention. Since the robot HTTP listener 610 is configured to start on any unused random port, the robot link 606 initiates the port discovery process 700 to discover the port of the robot HTTP listener 610 before the robot link 606 can communicate with the port.

[0073] In one embodiment, when an application uses an API from robot link 606 and the port is unknown to robot link 606, process 700 begins by initiating a port discovery process at 702. At 704, robot link 606 calls protocol handler 608 using a custom protocol handler while passing token T1. This ensures that protocol handler 608 is called by the current user and not any other user session on the machine.

[0074] At 706, the protocol handler 608 passes the token T1 to the HTTP port discovery service 612. The protocol handler 608 also passes information about the user session under which it is running (e.g., session 0). The protocol handler 608 can also activate the robot HTTP listener service 610 and the HTTP port discovery service 612 if they are not already running.

[0075] At 708 , the robot link 606 also communicates with the HTTP port discovery service 612 , providing the token T1 , and requesting that the HTTP port discovery service 612 provide the robot HTTP listener 610 as the port to which it should communicate.

[0076] At 710, the HTTP port discovery service 612 looks at the information received from the protocol handler 608, which identifies which user the token T1 belongs to, and then uses the registered port list to identify the port of the robot HTTP listener 610 for that particular user. At 712, the HTTP port discovery service 612 replies to the robot link 606 with the port of the robot HTTP listener 610 with which it should communicate, completing the port discovery process.

[0077] Certification

[0078] Figure 8is a flow chart illustrating an authentication process 800 according to an embodiment of the present invention. When the robot HTTP listener 610 is running on an HTTP port on the computing system 600, any software running on the computing system 600 can access the HTTP port, even on behalf of another user. This can lead to a security risk, as another user or software running on the computing system 600 can call the robot process on the user's behalf. To mitigate this risk, each call to the robot HTTP listener 610 is authenticated and authorized before the call is accepted. This forces the robot link 606 to retrieve an authentication token for the robot HTTP listener 610 before the robot link 606 can communicate with it.

[0079] In some embodiments, process 800 begins at 802 when an application uses any API from robot link 606 and the robot link 606 does not know the port, initiating a port discovery process. At 804, after the port discovery process is complete, robot link 606 issues a call to robot HTTP listener 610. If robot HTTP listener 610 does not receive any token or a valid token, robot HTTP listener 610 rejects the call from robot link 606.

[0080] When the call is rejected, the robot link 606 needs to authenticate and authorize using a valid token before calling the robot HTTP listener 610 a second time. Therefore, at 806, the robot link 606 uses the custom protocol handler to call the protocol handler 608 while passing the token T1. This ensures that the protocol handler 608 is called for the current user and not for another user in another session on the same computing system 600. In some optional embodiments, at 808, the robot link 606 also displays a prompt on the application to notify the user to approve the request to use the token T1.

[0081] At 810, once invoked, the protocol handler 608 displays a user interface (UI) pop-up window for the user to approve the request from the robot link 606. In some embodiments, a token T1 may also be displayed to help the user authenticate the request, i.e., verify that the request is from a specific robot link 606 or application and not from another application or user on the computing system 600.

[0082] At 812, once the user approves the request, the protocol handler 608 passes the token T1 to the HTTP port discovery service 612, along with information about which user session it is running under. If the protocol handlers 608 are not running at this time, they can also start the robot HTTP listener 610 and the HTTP port discovery service 612.

[0083] At 814, robot link 606 also communicates with HTTP port discovery service 612, providing it with token T1 and requesting authentication token T2 for communication with robot HTTP listener 610. At 816, HTTP port discovery service 612 examines the information received from protocol handler 608 to identify the user to whom token T1 belongs. HTTP port discovery service 612 also uses the list of registered services to identify the robot HTTP listener 610 for that particular user. At 818, HTTP port discovery service 612 generates a new token T2 and passes it to robot HTTP listener 610 as a valid token for authentication. Token T2 may include additional information, such as the domain it is registered for, its creation time, and its expiration time, to aid in the authentication process. At 820, HTTP port discovery service 612 responds to robot link 606 with the authentication token T2 for the robot HTTP listener 610 it is requesting. At 822, the robot link 606 may now communicate with the robot HTTP listener 610 using the authentication token T2, thereby completing the authentication process.

[0084] In some embodiments, the authentication token T2 may also be generated by the protocol handler 608 and may communicate with the HTTP port discovery service 612 and the robot HTTP listener 610 instead of the HTTP port discovery service 612 generating the authentication token T2.

[0085] Robot service interaction

[0086] Some embodiments provide the Robot HTTP Listener 610 with various functions provided by the Robot Service 614. For example, the Robot HTTP Listener 610 can query the status of the Executor 212 or Robot 210, connect and / or disconnect with the Commander 230, list the processes available on the Robot 210, start, stop, pause, terminate, etc., processes on the Robot 210, query the status of processes running or executing on the Robot 210, and / or send and / or receive data when a process is started, executed, or completed. Specifically, some embodiments can enable an application to call an RPA to retrieve and / or send data from a local machine.

[0087] Figure 9 606 is a flowchart illustrating a bot service interaction process 900 according to an embodiment of the present invention. In some embodiments, an application interacts with a bot service 614 using a bot link 606. In these embodiments, the application can call any supported API / function / operation provided by the bot link 606.

[0088] In some embodiments, process 900 begins at 902, where robot link 606 ensures port discovery and authentication when an application calls any supported API, function, or operation. At 904, robot link 606 calls a specific API via HTTP on robot HTTP listener 610. Robot HTTP listener 610 communicates with robot service 614 at 906 to perform the operation requested by robot link 606. At 908, robot service 614 communicates with executor 212, robot 210, director 230, or any other application as needed to complete the requested operation. At 910, robot service 614 returns the results of the requested operation to robot HTTP listener 610, and at 912, robot HTTP listener 610 returns the results to robot link 606. At 914, robot link 606 returns the data to the application.

[0089] Figure 10 1000 is a flow chart illustrating a process 1000 for facilitating a robot interaction process according to an embodiment of the present invention. In this embodiment, process 1000 begins at 1002 by launching an application from a computing system to invoke a robot link embedded in the application, and at 1004, initiates a port discovery process from the application to identify a port, port details, and a token. At 1006, process 1000 continues with the application generating a randomized code and invoking a consent application requesting approval from a user of the computing system to invoke the robot from the application. At 1008, process 1000 also performs registration of the randomized code with a local listener module and passes user information and token to a global listener module. At 1010, process 1000 performs receipt of the token and port identification from the global listener module, thereby allowing the application to authenticate itself with the robot and communicate with the robot, thereby facilitating the robot interaction process.

[0090] It should be understood that an application with embedded robot links can be built in a local application development platform and can connect to both supervised and unsupervised robots. In some embodiments, the application can be used as an ".exe" file that runs on the desktop of a computing system, rather than as part of a web browser. In some other embodiments, the application can be docked on the screen so that the application can communicate with other applications simultaneously.

[0091] According to an embodiment of the present invention, Figure 7-10 The process steps performed in the embodiment of the present invention may be performed by a computer program encoded with instructions for causing a processor(s) to perform Figure 7-10The computer program may be implemented on a non-transitory computer readable medium. The computer readable medium may be, but is not limited to, a hard drive, a flash memory device, a RAM, a magnetic tape, and / or any other such medium or combination of media for storing data. The computer program may include a program for controlling the processor(s) of a computing system (e.g. Figure 5 (a plurality of) processors 510 of the computing system 500) to implement Figure 7-10 The coded instructions for all or part of the process steps described in the present invention may also be stored on a computer-readable medium.

[0092] A computer program may be implemented in hardware, software, or a hybrid implementation. A computer program may be composed of modules that are operable to communicate with each other and are designed to pass information or instructions for display. A computer program may be configured to operate on a general-purpose computer, an ASIC, or any other suitable device.

[0093] It will be readily understood that, as generally described and illustrated in the figures herein, the components of the various embodiments of the present invention may be arranged and designed in a variety of different configurations. Therefore, the detailed description of the embodiments of the present invention, as represented in the accompanying drawings, is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention.

[0094] The features, structures, or characteristics of the present invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, throughout this specification, references to "certain embodiments," "some embodiments," or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, throughout this specification, the appearance of the phrases "in certain embodiments," "in some embodiments," "in other embodiments," or similar language does not necessarily all refer to the same set of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0095] It should be noted that throughout this specification, references to features, advantages, or similar language do not imply that all features and advantages that can be achieved with the present invention should be or are present in any single embodiment of the present invention. Rather, language referring to features and advantages is understood to mean that a specific feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Therefore, throughout this specification, discussions of features and advantages, and similar language, may, but do not necessarily, refer to the same embodiment.

[0096] Furthermore, the described features, structures, and characteristics of the present invention may be combined in any suitable manner in one or more embodiments. Those skilled in the relevant art will recognize that the present invention may be practiced without one or more of the specific features or advantages of a particular embodiment. In other cases, additional features and advantages that may not be present in all embodiments of the present invention may be identified in certain embodiments.

[0097] It will be readily understood by those skilled in the art that the invention as described above may be practiced utilizing steps in a different order and / or utilizing hardware elements in configurations different from those disclosed. Thus, while the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative configurations will be apparent while remaining within the spirit and scope of the invention. Therefore, reference should be made to the appended claims for determining the limits and boundaries of the invention.

Claims

1. A computer-implemented method comprising: launching an application from a computing system to invoke a robot link embedded within the application; initiating a port discovery process from the application to identify a port, port details, and a first token; generating, by the application, a randomized code, and invoking a consent application that requests approval from a user of the computing system to invoke the robot from the application; registering the randomized code with a local listener module, and transmitting user information and the first token to a global listener module; as well as A second token and port identification are received from the global listener module, thereby allowing the application to authenticate itself with the robot and communicate with the robot, thereby facilitating a robot interaction process.

2. The computer-implemented method of claim 1 , wherein the application used to invoke the robot link is from an untrusted domain.

3. The computer-implemented method of claim 1 , wherein said facilitation of said robotic interaction process comprises: A specific application programming interface (API) on the robot listener module is called by the robot link through the communication medium.

4. The computer-implemented method of claim 3, wherein the robot listener module is configured to query the status of an executor or the robot, connect to, disconnect from, or both a commander, list available processes available on the robot, start, stop, pause, and / or terminate a process on the robot, query the status of a process running or being executed on the robot, and / or send and / or receive data while starting a process, while the process is already executing, or when the process completes execution.

5. The computer-implemented method of claim 3 , wherein said facilitation of said robotic interaction process further comprises: The robot listener module communicates with the robot service to perform the operation requested by the robot link.

6. The computer-implemented method of claim 5, wherein said facilitation of said robotic interaction process further comprises: The robotic service communicates with the actuators, robots and / or commanders to complete the requested operation.

7. The computer-implemented method of claim 6, wherein said facilitation of said robotic interaction process further comprises: A result of the requested operation is returned from the robot service to the application through the robot listener module and the robot link.

8. A non-transitory computer-readable medium storing a computer program, the program being configured to cause at least one processor to: launching an application from a computing system to invoke a robot link embedded within the application; initiating a port discovery process from the application to identify a port, port details, and a first token; generating, by the application, a randomized code, and invoking a consent application that requests approval from a user of the computing system to invoke the robot from the application; registering the randomized code with a local listener module, and transmitting user information and the first token to a global listener module; as well as A second token and port identification are received from the global listener module, thereby allowing the application to authenticate itself with the robot and communicate with the robot, thereby facilitating a robot interaction process.

9. The non-transitory computer-readable medium of claim 8, wherein the application used to invoke the robot link is from an untrusted domain.

10. The non-transitory computer-readable medium of claim 8, wherein the computer program is further configured to cause at least one processor to: A specific application programming interface (API) on the robot listener module is called by the robot link through the communication medium.

11. The non-transitory computer-readable medium of claim 10, wherein the robot listener module is configured to query the status of an executor or the robot, connect to, disconnect from, or both a commander, list available processes available on the robot, start, stop, pause, and / or terminate a process on the robot, query the status of a process running or being executed on the robot, and / or send and / or receive data while starting a process, while the process is already executing, or when the process completes execution.

12. The non-transitory computer-readable medium of claim 11 , wherein the computer program is further configured to cause at least one processor to: The robot listener module communicates with the robot service to perform the operation requested by the robot link.

13. The non-transitory computer-readable medium of claim 12, wherein the computer program is further configured to cause at least one processor to: The robotic service communicates with the actuators, robots and / or commanders to complete the requested operation.

14. The non-transitory computer-readable medium of claim 13, wherein the computer program is further configured to cause at least one processor to: The result of the requested operation is returned from the robot service to the application through the robot listener module and the robot link.

15. A computing system comprising: a memory storing machine-readable computer program instructions; as well as at least one processor configured to execute the computer program instructions, the instructions configured to cause the at least one processor to: launching an application from the computing system to invoke a robot link embedded within the application; initiating a port discovery process from the application to identify a port, port details, and a first token; generating, by the application, a randomized code, and invoking a consent application that requests approval from a user of the computing system to invoke the robot from the application; registering the randomized code with a local listener module, and transmitting user information and the first token to a global listener module; as well as A second token and port identification are received from the global listener module, thereby allowing the application to authenticate itself with the robot and communicate with the robot, thereby facilitating a robot interaction process.

16. The computing system of claim 15, wherein the application used to invoke the robot link is from an untrusted domain.

17. The computing system of claim 15, wherein the instructions are further configured to cause the at least one processor to: A specific application programming interface (API) on the robot listener module is called by the robot link through the communication medium.

18. The computing system of claim 17, wherein the robot listener module is configured to query the status of an executor or the robot, connect to, disconnect from, or both a commander, list available processes available on the robot, start, stop, pause, and / or terminate a process on the robot, query the status of a process running or being executed on the robot, and / or send and / or receive data while starting a process, while the process is already executing, or when the process completes execution.

19. The computing system of claim 18, wherein the instructions are further configured to cause the at least one processor to: The robot listener module communicates with the robot service to perform the operation requested by the robot link.

20. The computing system of claim 19, wherein the instructions are further configured to cause the at least one processor to: The robotic service communicates with the actuators, robots and / or commanders to complete the requested operation.

Images