Image Pattern Matching for Robotic Process Automation

Abstract

Disclosed herein is a computing system. The computing system includes a memory and a processor. The memory stores processor-executable instructions for a workflow recommendation assistant engine. The processor is coupled to the memory. The processor executes the workflow recommendation assistant engine to cause the computing device to analyze images of a user interface corresponding to a user activity, perform pattern matching of the images with respect to existing automation, and provide a prompt indicating that an existing automation matches the user activity.

Description

[Background technology]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Application No. 17 / 370,145, filed July 8, 2021, the contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of robotic process automation (RPA), and more particularly to image pattern matching for RPA.

[0003] Traditional operating systems and software systems may include script automation of user activities. Script automation is generally the process of leveraging existing scripts to achieve automation of user activities in a managed framework (i.e., without the need to develop and maintain custom scripts). However, script automation is limited to one-to-one, sequential mimicry of user actions and lacks a way to provide an intelligent automation experience of user activities. Summary of the Invention

[0004] According to one or more embodiments, a computing system is provided. The computing system includes a memory and at least one processor. The memory stores processor-executable instructions for a workflow recommendation assistant engine. The at least one processor is coupled to the memory. The at least one processor is configured to execute the workflow recommendation assistant engine to cause the computing device to analyze images of a user interface corresponding to a user activity, perform pattern matching of the one or more images with at least one existing automation, and provide a prompt indicating that the at least one existing automation matches the user activity.

[0005] According to one or more embodiments, the above-described computing system embodiments may be implemented as methods, apparatus, devices, and / or computer program products.

[0006] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, in which like reference numerals indicate like elements and in which: [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an environment illustrating the development, design, operation, and / or execution of robotic process automation (RPA) in accordance with one or more embodiments.

[0008] [Figure 2] FIG. 1 is an illustration of a computing system according to one or more embodiments.

[0009] [Figure 3] 1 is a block diagram illustrating a method performed by one or more processors in a local computing system according to one or more embodiments.

[0010] [Figure 4] 1 is an example of a user interface of a workflow recommendation assistant engine according to one or more embodiments.

[0011] [Figure 5] FIG. 2 is a block diagram illustrating a method performed by a workflow recommendation assistant engine according to one or more embodiments.

[0012] [Figure 6] 1 is a block diagram illustrating a method performed by a workflow recommendation assistant engine according to one or more embodiments.

[0013] Disclosed herein is a workflow recommendation assistant engine (hereinafter, "recommendation engine"). The recommendation engine includes software and / or hardware for image pattern matching for robotic process automation (RPA), etc. That is, the recommendation engine can be implemented in a computing system through a combination of hardware and software (e.g., processor-executable code that requires routing to hardware).

[0014] According to one or more embodiments, the recommendation engine may provide specific configurations and tooling for robotic process automation methods in terms of an intelligent automation experience. The intelligent automation experience includes using robotic intelligence (e.g., machine learning and artificial intelligence) to perform background monitoring / analysis of user activity for the purpose of identifying and suggesting RPAs for the user to use. The background / analysis monitoring may include capturing images of the user activity (e.g., images surrounding the user activity or images as screenshots), and the identification and suggestion of RPAs includes processing the captured images to detect and suggest RPA matches (e.g., pattern matching).

[0015] For example, configuring and tooling a robotic process automation method with a recommendation engine may include providing a lightweight service (e.g., low system resource usage, such as a small memory footprint and low CPU usage) that runs in the background of the software system (e.g., residing outside of the workflow) that takes a hash of what a user is doing (e.g., capturing an image), sequences the hash, and runs the hash against a repository of existing automations (e.g., in sequence and / or out of sequence) to determine if there is a match. If the configuring and tooling robotic process automation method determines there is a match, the recommendation engine may suggest to the user that there is an existing automation (e.g., RPA, etc.) that does what the user is doing (e.g., mimics the user activity) along with providing the user with an opportunity to view, select, and / or start existing automations.

[0016] Thus, the configuration of the recommendation engine and tooling robotic process automation method includes multi-stage data manipulation of captured images, interface components, data, and configurations associated with the software system. One or more technical effects, advantages, and benefits of the recommendation engine include providing a user with an intelligent automation experience of user activities that is not available through traditional script automation.

[0017] 1 illustrates an environment 100 for developing, designing, operating, and / or executing robotic process automation (RPA) systems in accordance with one or more embodiments. Environment 100 may be a design and deployment computing platform that includes one or more components. Note that while a single block is shown for each of one or more components of environment 100, the single block is representative of that component or components.

[0018] As shown, the environment 100 may include a designer 110 that further includes a workflow recommendation assistant engine (“recommendation engine”) 111. The recommendation engine 111 may include a studio component or module 120 that uses one or more driver components 128 and / or internal engines 129 to generate one or more robots (e.g., robot 122) that perform one or more activities 124 and / or provide one or more user interface (UI) automations 126. The robot 122 may also interact with other driver components 131 and modules 132 within the designer 110. Additionally, the environment 100 may include a conductor 140 and an executor 170 that may download / obtain / send the robot 122 so that the robot 122 may be provided to the environment 100 as an unattended robot 174 and an attended robot 178 of the executor 170.

[0019] Designer 110, conductor 140, and executor 170 represent computing devices, apparatuses, and / or systems, including hardware, software, or a combination thereof. Designer 110, conductor 140, and executor 170, and any software thereon (e.g., recommendation engine 111), are configured to interact with a user or operator to receive input and provide output. In one example, executor 170 can be or implement a mobile computing and / or mobile device environment.

[0020] According to one or more embodiments, the designer 110 may be referred to as a studio platform, a development platform, and / or an online platform. The designer 110 may include one or more engines (e.g., a recommendation engine 111), a development environment (e.g., a studio 120), etc., and may be configured to generate code, instructions, commands, etc. for robots (e.g., one or more robots 122, attended robots 174, and attended robots 178) to perform or automate one or more workflows (e.g., to provide one or more activities 124 and / or one or more UI automations 126).

[0021] Recommendation engine 111 is generally software and / or hardware that implements robotic process automation. In this regard, recommendation engine 111 is detailed as a dotted box to illustrate the scalability and portability of recommendation engine 111 within environment 100. For example, recommendation engine 111 may be implemented as separate instances throughout environment 100, as illustrated by recommendation engines 181 and 191.

[0022] Additionally, according to one or more embodiments, the recommendation engine 111 may be part of a framework / mechanism (e.g., environment 100) that automatically implements code within the studio 120 to design the robot 122. The recommendation engine 111 may also enable a developer and / or orchestrator (e.g., a user or operator) to code, program, and / or design the robot 122 (e.g., via the studio 120). In embodiments, the studio 120 may provide application integration with third-party applications, automation of management information technology (IT) tasks, and / or business IT processes. In general, the recommendation engine 111 may include robotic intelligence that looks at a set of known captured images (e.g., small photographs) against a currently captured image, aligns the sequence, determines if there is a match, and presents the match in favor.

[0023] The robot 122 (as well as the unattended robot 174 and the attended robot 178) may be an application, applet, script, etc., and may execute and / or automate one or more workflows (e.g., represented in part by one or more activities 124 and / or providing one or more UI automations 126). Workflows may include, but are not limited to, task sequences, flowcharts, finite state machines (FSMs), global exception handlers, transparent UIs, etc. Task sequences may be linear processes for handling linear tasks across one or more applications or windows. Flowcharts may be configured to handle complex business logic, enabling decision integration and activity connection in more diverse ways through multiple branching logic operators. FSMs may be configured for large workflows. FSMs may use a finite number of states during their execution that can be triggered by conditions, transitions, activities, etc. Global exception handlers may be configured to determine the behavior of a workflow when an execution error occurs, such as for process debugging. Transparent UIs may be software operations relative to the underlying operating system (OS) or hardware. Non-limiting examples of actions that may be accomplished by one or more workflows may be one or more of performing a login, filling out a form, information technology (IT) administration, etc. To execute a workflow for UI automation, the robot 122 may need to uniquely identify a particular screen element, such as a button, checkbox, text field, label, etc., regardless of application access or application development. Examples of application access may include local, virtual, remote, cloud, Citrix®, VMWare®, VNC®, Windows® Remote Desktop, Virtual Desktop Infrastructure (VDI), etc.Examples of application development can be win32, Java, Flash, Hypertext Markup Language (HTML), HTML5, Extensible Markup Language (XML), JavaScript, C#, C++, Silverlight, etc.

[0024] According to one or more embodiments, during development, robot 122 may be generated in / by designer 110 (e.g., studio 120). Furthermore, during deployment, robot 122 may be, for example, managed, controlled, or configured in / by conductor 140 (e.g., in recommendation engine 181). Also, during deployment, robot 122 may be executed in executor 170. In one example, user input (e.g., selections on a display) may be submitted to designer 110 by a user or operator. From the selections that designer 110 may provide to robot 122, robot 122 may determine representative data for region(s) of the visual display (e.g., selected by the user or operator). As part of RPA, multi-dimensional shapes (e.g., squares, rectangles, circles, polygons, freeforms, etc.) can be utilized in the development and runtime of UI robots in conjunction with computer vision (CV) behaviors and / or machine learning (ML) models.

[0025] According to one or more embodiments, UI automation 126 may be performed by the robot 122 using an internal driver 128, and the recommendation engine 111 may perform activities using a module 130 (e.g., a CV activity module or engine). A driver component 132 may be utilized for UI automation by the robot 122 to obtain elements of the UI. The driver component 132 may include, but is not limited to, an OS driver, a browser driver, a virtual machine driver, an enterprise driver, etc. In certain configurations, the module 130 may be a driver used for UI automation. Furthermore, any of these actions by the robot 122 may be implemented on the client (e.g., as an unattended robot 174 and an attended robot 178 of the executor 170).

[0026] Conductor 140 may direct or command robots 122, 174, and 178 or executor 170 to execute or monitor workflows in a mainframe, web, virtual machine, remote machine, virtual desktop, enterprise platform, online platform, desktop app(s), browser, or any such client, application, or program. Conductor 140 may act as a central or semi-central point for directing or commanding one or more robots (e.g., robots 122, 174, and 178) to automate a computing platform (e.g., environment 100).

[0027] According to one or more embodiments, conductor 140 may be configured to provide provisioning, deployment, configuration, queuing, monitoring, logging, and / or interconnectivity. Provisioning may include creating and maintaining connections or communications between one or more robots, executors 170, and conductor 14. Deployment may include ensuring delivery of package versions (for execution) to assigned robots. Configuration may include maintaining and delivering robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot-specific data and maintaining user permissions. Logging may include storing and indexing logs in a database (e.g., an SQL database) and / or another storage mechanism (e.g., ElasticSearch®, which stores large data sets and provides the ability to quickly query them). Conductor 140 may provide interconnectivity by serving as a centralized point of communication for third-party solutions and / or applications.

[0028] According to one or more embodiments, one or more robots and executors 170 may be configured as unattended robots 174 and / or attended robots 178. In the case of unattended operation, the automation by the unattended robot 174 may be performed without third-party input or control. In the case of attended operation, the automation by the attended robot 178 may be performed by receiving input, commands, instructions, guidance, etc. from a third-party component. The unattended robot 174 and / or attended robot 178 may operate or execute in a mobile computing environment or mobile device environment.

[0029] According to one or more embodiments, one or more robots and executors 170 may be configured as execution agents that execute workflows built in designer 110. A commercial example of a robot(s) for UI or software automation is UiPath Robots™. In some embodiments, one or more robots and executors 170 may install the Microsoft Windows Service Control Manager (SCM) management service by default. As a result, such robots may open interactive Windows sessions under the local system account and have Windows service rights.

[0030] According to one or more embodiments, one or more robots and executors 170 may be installed in user mode. For example, one or more robots may have the same rights as the user to which the given robot is installed. This feature may also be enabled in high-density (HD) robots, allowing each machine to be fully utilized for maximum performance, such as in HD environments.

[0031] According to one or more embodiments, one or more robots and executors 170 may be split, distributed, etc. into several components, each dedicated to a specific automation task or activity. Robot components may include an SCM management robot service, a user-mode robot service, an executor, an agent, a command line, etc. The SCM management robot service may manage or monitor Windows sessions and act as a proxy between the conductor 140 and the execution host (i.e., the computing system on which one or more robots execute). These services may trust and manage credentials for one or more robots and executors 170. The user-mode robot service may manage and monitor Windows sessions and act as a proxy between the conductor 140 and the execution host. The user-mode robot service may be trusted and may manage credentials for the robots. If the SCM management robot service is not installed, a Windows application may be launched automatically. In one example, executors 170 may run a given job under a Windows session (e.g., they may execute the workflows described herein) and may be aware of dots per inch (DPI) settings per monitor. An agent may be a Windows Presentation Foundation (WPF) application that displays available jobs in a system tray window. An agent may be a client of a service. An agent may ask to start or stop a job or change settings. A command line may be a client of a service. A command line is a console application that can request the start of a job and wait for its output.

[0032] According to one or more embodiments, the separation of one or more robot and / or executor 170 components helps developers, support users, and computing systems more easily execute, identify, and track execution by each component. In this manner, special behaviors can be configured for each component, such as setting different firewall rules for executor 170 and services. In some embodiments, executor 170 may be aware of per-monitor DPI settings. As a result, workflows may run at any DPI regardless of the configuration of the computing system on which the workflow was created. Additionally, projects from designer 110 may be made independent of browser zoom levels. For applications that are not DPI-aware or are intentionally marked as not-aware, DPI may be disabled in some embodiments.

[0033] 2, a computing system 200 according to one or more embodiments is illustrated. Computing system 200 may represent any computing device, apparatus, and / or computing environment, including hardware, software, or a combination thereof (e.g., hardware supporting recommendation engines 111, 181, and 191 of FIG. 1). Furthermore, embodiments of the disclosed computing system 200 may include apparatuses, systems, methods, and / or computer program products at any possible technical level of integration.

[0034] Computing system 200 includes a device 205 (e.g., designer 110, conductor 140, and executor 170 of FIG. 1 ) having one or more central processing units (CPU(s)), collectively or generally referred to as a processor 210. Processor 210, also referred to as a processing circuit, is coupled to a system memory 220 and various other components via a system bus 215. Computing system 200 and / or device 205 may be adapted or configured to perform as an online platform, a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing device, a cloud computing device, a mobile device, a smartphone, a fixed mobile device, a smart display, a wearable computer, etc.

[0035] Processor 210 may be any type of general-purpose or special-purpose processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), a controller, a multi-core processing unit, a three-dimensional processor, a quantum computing device, or any combination thereof. Processor 210 may have multiple processing cores, at least some of which may be configured to perform specific functions. Multi-parallel processing may also be configured. Furthermore, at least processor 210 may be a neuromorphic circuit that includes processing elements that mimic biological neurons.

[0036] The bus 215 (or other communication mechanism) is configured to communicate information or data to various other components, such as the processor 210 , the system memory 220 , and the adapters 225 , 226 and 227 .

[0037] System memory 220 is an example of a (non-transitory) computer-readable storage medium in which repository 229 may be located / managed and / or software 230 may be stored. Repository 229 may be a database (e.g., an SQL database) and / or another storage mechanism. According to one or more embodiments, repository 229 may include automation and other data (e.g., images, segments, hashes, videos, frames, source data, robot videos, source code, etc.) as described herein for access by software 230. For example, during a training phase of software 230, a matching model may be built and stored in repository 229. In this regard, repository 229 stores frames of video including target elements / actions and the space surrounding those elements as training data for the matching model.

[0038] The software 230 may be stored as software components, modules, engines, instructions, etc., for execution by the processor 210 to operate the device 205, such as those described herein with reference to Figures 3-5. The system memory 220 may include any combination of read-only memory (ROM), random-access memory (RAM), internal or external flash memory, embedded static RAM (SRAM), solid-state memory, cache, static storage such as a magnetic or optical disk, or any other type of volatile or nonvolatile memory. The non-transitory readable storage medium may be any medium accessible by the processor 210 and may include volatile media, non-volatile media, etc. For example, the ROM may include a basic input / output system (BIOS) coupled to the system bus 215 that controls certain basic functions of the device 205, and the RAM is readable / writable memory coupled to the system bus 215 for use by the processor 210. The non-transitory computer-readable storage medium may include any medium, removable, non-removable, etc.

[0039] 1, software 230 in FIG. 2 may represent recommendation engine 111 and components therein such that memory 220 and processor 210 can logically design / configure / provide robot 122, which further performs one or more activities 124 and / or provides one or more user interface (UI) automations 126 using one or more driver components 128 and / or internal engines 129. Software 230 in FIG. 2 may also represent an operating system for device 205 for computing system 200.

[0040] According to one or more embodiments, software 230 may be configured in hardware, software, or a hybrid implementation. Software 230 may be configured with modules that operatively communicate with each other and pass information or instructions. According to one or more embodiments, software 230 may provide one or more user interfaces on behalf of an operating system or other applications and / or directly as needed to provide an intelligent automation experience. User interfaces include, but are not limited to, internet browsers, graphic user interfaces, windowing interfaces, and / or other visual interfaces for applications, operating systems, file folders, etc. Accordingly, user activity may include any interaction or manipulation of a user interface provided by software 230.

[0041] Software 230 may further include custom modules for performing application-specific processing or derivatives thereof, such that computing system 200 can include additional functionality. For example, according to one or more embodiments, software 230 may be configured to store information, instructions, commands, or data that are executed or processed by processor 210 to enable operations 231, 232, 233, and 234. For example, recommendation engine 111 of FIG. 1 , implemented as software 230 of FIG. 2 , may provide computing system 200 with a particular configuration and tooling for robotic process automation methods, such that, along with any other operations / actions (234) described herein, computing system 200 may monitor (231) user activity on the computing device, capture one or more images of a user interface, analyze (232) the one or more images, perform pattern matching with at least one existing automation, and provide (233) a prompt indicating that at least one existing automation matches the user activity. According to one or more embodiments, software 230 may be configured to pre-process the captured image, such as by executing a perceptual hashing algorithm to generate various forms of multimedia snippets or fingerprints (also referred to as pHashings or pHashes).

[0042] Other behaviors / actions may include, but are not limited to, utilizing RPA agent logs, adding source code metadata to fame when creating robot videos, not recording the cursor when recording videos, excluding the cursor from videos based on known system cursors, performing keyframe caching, pre-building functionality for automated processing, highlighting elements with shapes (e.g., rectangles) as the robot is running, and querying the shape of elements as the robot is running.

[0043] For example, according to one or more embodiments, the recommendation engine 111 of FIG. 1 implemented as software 230 of FIG. 2 may perform offloading operations. The offloading operations include identifying automations and triggers (e.g., by hash matching) rather than intentionally coding for specific applications. In this manner, the intentional coding is offloaded to the recommendation engine 111 of FIG. 1. Additionally, the recommendation engine 111 of FIG. 1 implemented as software 230 of FIG. 2 provides the user with the ability / option to execute matching automations.

[0044] Additionally, modules of software 230 may be implemented as hardware circuits including custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips and transistors, or other discrete components, programmable hardware devices (e.g., field programmable gate arrays, programmable array logic, programmable logic devices), graphics processing units, etc. Modules of software 203 may be implemented at least in part in software for execution by various types of processors. According to one or more embodiments, an identified unit of executable code may comprise one or more physical or logical blocks of computer instructions, which may be organized as, for example, an object, a procedure, a routine, a subroutine, or a function. Executables of identified modules may be co-located or stored in different locations such that when logically combined they constitute a module. A module of executable code may be a single instruction, one or more data structures, one or more datasets, multiple instructions, etc., and may be distributed across multiple different code segments, different programs, multiple memory devices, etc. The operational or functional data may be identified and illustrated within modules of the software 230 and may be embodied and organized in any suitable form within any suitable type of data structure.

[0045] With respect to adapters 225, 226, and 227 of FIG. 2, device 205 may include, among other things, an input / output (I / O) adapter 225, a device adapter 226, and a communications adapter 227. In one or more embodiments, I / O adapter 225 may support a variety of wireless technologies, including Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiplexing (OFDM), Global System for Mobile (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), cdma2000, Wideband CDMA (W-CDMA), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), High-Speed ​​Packet Access (HSPA), Long Term Evolution (LTE), LTE Advanced (LTE-A), and the like. It may be configured as a small computer system interface (SCSI) in the context of any of the following: 802.11x, Wi-Fi, Zigbee, Ultra-Wideband (UWB), 802.16x, 802.15, Home Node-B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near-Field Communications (NFC), 5th Generation (5G), New Radio (NR), or any other wireless or wired device / transceiver for communication.The device adapters 226 interconnect input / output devices such as a display 241 , a keyboard 242 , and control devices 243 (eg, cameras, speakers, etc.) to the system bus 215 .

[0046] Communications adapter 226 interconnects system bus 215 with network 250, which may be an external network, allowing device 205 to communicate data with other such devices (e.g., local computing device 255 and even remote computing system 256 via network 260). In one embodiment, adapters 225, 226, and 227 may be connected to one or more I / O buses that are connected to system bus 215 through intermediate bus bridges. I / O buses suitable for connecting peripherals such as hard disk controllers, network adapters, and graphics adapters typically include a common protocol such as Peripheral Component Interconnect (PCI).

[0047] Display 241 is configured to provide one or more UIs or graphical UIs (GUIs) that can be captured and analyzed by software 230 as a user interacts with device 205. Examples of display 241 may include, but are not limited to, plasma, liquid crystal display (LCD), light emitting diode (LED), field emission display (FED), organic light emitting diode (OLED) display, flexible OLED display, flexible substrate display, projection display, 4K display, high definition (HD) display, Retina (copyright) display, in-plane switching (IPS) display, etc. For input / output (I / O), display 241 may be configured as a touch using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse recognition, frustrated total internal reflection, etc., three-dimensional (3D) touch, multi-input touch, or multi-touch display, as will be appreciated by those skilled in the art.

[0048] A keyboard 242 and control device 243, such as a computer mouse, touchpad, touchscreen, keypad, etc., may further be coupled to system bus 215 for providing input to device 205. Additionally, one or more inputs may be provided remotely to computing system 200 via another computing system in communication therewith (e.g., local computing device 255 and / or remote computing system 256), or device 205 may operate autonomously.

[0049] According to one or more embodiments, the functionality of device 205 relating to software 230 may also be embodied on local computing device 255 and / or remote computing system 256, as represented by separate instances of recommendation engine 290. It should be noted that one or more images (e.g., screenshots) may be stored in a common repository (e.g., repository 229) located on device 205, local computing device 255, and / or remote computing system 256 and may be downloaded (on demand) to and / or from device 205, local computing device 255, and / or remote computing system 256, respectively.

[0050] 3, a block diagram illustrates a process flow 300 executed by one or more processors in a computing system, according to one or more embodiments. More particularly, process flow 300 is an exemplary implementation of a recommendation engine (e.g., recommendation engine 111 of FIG. 1 and / or software / recommendation engines 231, 281, and 291 of FIG. 2) described herein to provide a particular configuration and tooling robotic process automation method for an intelligent automation experience.

[0051] Process flow 300 begins at block 310, where a recommendation engine monitors user activity. Monitoring user activity may include recording video, capturing one or more frames of video, capturing images, capturing screenshots, capturing image segments, etc. The recommendation engine may provide a lightweight service (e.g., small memory footprint and low system resource usage, such as low CPU usage) that runs in the background of a computing system, taking a hash of what the user is doing (e.g., capturing images), among other things. According to one or more embodiments, the recommendation engine may provide a real-time scenario application or a delayed scenario application. A real-time scenario application may use recording software, dedicated minimal recording, or other software to capture screenshots and native data that is then currently processed. A delayed scenario application may use any recording software described herein to generate screenshots and native data that is stored in a database (e.g., repository 229) for later review. According to one or more embodiments, the recording software may be implemented by or as part of a robot. For example, a recommendation engine may involve providing a lightweight service (e.g., small memory footprint and low system resource usage, such as low CPU usage) that runs in the background of a software system (e.g., exists outside of the workflow) that takes a hash of what the user is doing (e.g., capturing an image).

[0052] For example, an attended RPA captures images (e.g., of the surroundings of user activity or as screenshots) as a user works or operates the computing system (e.g., as they interact with the UI or GUI provided by the display 241). The recommendation engine may preprocess the images, and any captured images may be preprocessed in real time. Preprocessing may include, but is not limited to, image segmentation, image or segment hashing, mode inversion, edge detection, step detection, change detection, feature detection, feature extraction, image cropping, and other enhancements or mechanisms for image processing to improve matching.

[0053] According to one or more embodiments of preprocessing, images and / or screenshots may be segmented into smaller pieces before hashes are calculated. Each segment is then hashed using a pHash. Note that all captured images may be hashed as a single bitmap. For example, each pHash may contain 8 bytes, with at least five hashes representing a segment, captured image, or screenshot. In some cases, one pHash may represent a target of user activity, while other segments may represent geometric configurations (e.g., left, bottom, top, etc.). Furthermore, hashing the image may include one-way encryption to protect the confidentiality of user activity while maintaining its integrity. Note that the size of each hash is relatively small (e.g., 8 bytes) compared to the size of the captured image, making it distributable.

[0054] According to one or more embodiments of the preprocessing, the mode flipping may include at least changing the user interface between a light mode and a dark mode, which are user interface color and / or brightness schemes that may be more adaptive based on the time of day (e.g., day vs. night). According to one or more embodiments of the preprocessing, cropping the image may include adjusting and / or resizing the screenshot to effectively capture the area surrounding the user activity.

[0055] At block 330, the recommendation engine analyzes the captured image or a segment thereof. Analyzing the captured image or segment may include pattern matching (e.g., performing pattern matching of one or more images and / or sequences with respect to at least one existing automation). According to one or more embodiments, pattern matching includes cross-checking one or more characteristics of the captured image or segment for some pattern, such as using a matching model described herein. The characteristics may be the type of action, the number of actions, the proximity of actions, etc., along with interactions with a particular screen or user interface element (e.g., a button, checkbox, text field, label, etc.). For example, each hash of the sequence may be analyzed to extract characteristics and compared with other characteristics of other stored hashes (e.g., representing existing automations), and the generated hash may be run in real time, in sequence and / or out of sequence, against the stored hashes to determine whether there is a match. Note that the recommendation engine's RPA (e.g., software 230 on device 205) may provide the sequence of generated hashes, while another instance of the recommendation engine (e.g., on local computing device 255 and / or remote computing system 256) may provide the stored hashes.

[0056] The characteristics may be determined by / within a window. More specifically, the window may be a buffer that holds a fixed number of captured images or segments. The window may then be used to optimize some or various characteristics based on this fixed number. In one example, the window may be a sliding window. The size of the sliding window may be dynamically increased or decreased by the recommendation engine, for example, as the fixed number of captured images or segments increases or decreases. Dynamically changing the size of the sliding window may be an automatic reconfiguration to optimize the performance and matching of the recommendation engine.

[0057] According to one or more embodiments, when analyzing a captured image or a segment thereof, the recommendation engine determines the size of a sliding window and fills the sliding window in a first-in, first-out manner with a sequence of generated hashes representing the captured image or a segment thereof. The sequence of generated hashes within the sliding window is then compared to a sequence of stored hashes (e.g., existing automation stored in a repository, such as those located in repository 229, local computing device 255, and / or remote computing system 256 of FIG. 2). More specifically, characteristics of the sequence of generated hashes within the sliding window are compared to characteristics of the sequence of stored hashes. The recommendation engine may use a threshold to determine how loosely or perfectly the characteristics match between the generated hashes and the stored hashes. In this way, if the number of characteristics of the sequence of hashes generated within the sliding window that match the characteristics of the sequence of stored hashes is equal to or greater than the threshold, then the recommendation engine determines that there is a match.

[0058] In block 350, the recommendation engine takes action if there is a match, such as presenting the match to the user.

[0059] For example, the recommendation engine's RPA may preset a prompt in the UI or GUI provided by the display 241 indicating that an existing automation matches the current set of user actions (i.e., characteristics determined from the captured image in the window). The RPA prompt may further provide the user with options for the existing automation to take over or execute, and / or a request to set a trigger. The prompt may also present or include a mechanism to visually indicate what matched (e.g., so that the user can see how the match was created by their specific actions). A trigger may be a user-defined condition that starts an automation. One or more triggers may be used to start any particular automation, such that a particular automation may be started in multiple ways. According to one or more embodiments, the recommendation engine may perform intentional actions in relation to the triggers. If the repository contains 10 automations that the recommendation engine can automatically trigger, the recommendation engine may create and execute a new automation package with 10 triggers that, when fired, invokes the 10 automations, or it may run the 10 automations in the background, with each automation waiting for its internal trigger to invoke the automation.

[0060] Turning to FIG. 4, a user interface 400 according to one or more embodiments is shown. User interface 400 is an example of a prompt for block 350 in FIG. 3. User interface 400 is titled "Workflow Recommendation Assistant" and includes a table 410 showing matching automations. Table 410 includes columns such as workflow name, match score, details, relevance (e.g., yes / no vote for applicability), and execution. In this example, only automations with matching codes are shown (e.g., "Workflow 8"). Because the names of one or more automated processes are not necessarily descriptive (e.g., "Workflow 8"), the recommendation engine may add a description (e.g., purpose) to verify that a user action matched the relevant automation according to one or more embodiments. With respect to the match score, the recommendation engine may determine a scale for how close a manual process is to an existing automation. For example, if a threshold is set to 99%, only existing automations that received a score equal to or greater than this threshold may be presented. The table shows a score of 100%. The threshold value may be set to any value between 0 and 100 based on the dynamic configuration of the recommendation engine described herein. Additionally, the recommendation engine may include one or more inferences regarding one or more threshold values, such that a cascade of existing automations may be presented by the user interface 400. For example, the recommendation engine may only enter existing automations that are 95% accurate or greater into the table 410, while making any existing automations that are at least 80% accurate discoverable.

[0061] One or more items in table 410 may be selectable, as indicated by underlined text. For example, selecting workflow 8 may drive a subsequent prompt to view its source code. Additionally, selecting yes or no may provide direct user feedback to the recommendation engine as to whether workflow 8 is a match, and selecting run may run workflow 8. Upon selecting details, subsequent prompts, shown as user interfaces 430a and 430b, are displayed to show the user how the automatic matching was performed. In this regard, the subsequent prompts toggle between one or more diagrams (e.g., user interface 430a at time 0 and user interface 430b at time 0+1), illustrating with arrows how the performed steps match with steps in workflow 8.

[0062] At block 370, the recommendation engine stores the generated sequence of hashes, along with the stored sequence of hashes and any triggers, to augment the repository for further matching. This storage further contributes to building and / or strengthening a matching model for machine learning. In turn, at block 390, the recommendation engine trains on the matching model to better identify cases where a match can be approximated. In this manner, the recommendation engine may feed back information from the training to dynamically adjust sliding windows, select features, determine the range of captured images, and the like. According to one or more embodiments, training the recommendation engine may include capturing frame images, segments, and / or hashes while attempting a match (e.g., with respect to comparison of geometric configurations, etc.). In this regard, the recommendation engine monitors the user as they perform one or more steps of the process and uses the captured frame images, segments, and / or hashes to build a repository. For example, the recommendation engine may save and use frames of a video that include a target element / action and the space surrounding that element as training data.

[0063] Thus, one or more technical effects, advantages, and benefits of the method 300 of FIG. 3 as the recommendation engine 111 of FIG. 1 implemented as the software 230 of FIG. 2 include identifying whether the process being developed has already been developed (e.g., or a portion thereof), analyzing the complex process through complex reasoning (e.g., reducing the probability that the observed activity is just automated), considering whether to provide an indication (e.g., screen matching) based on a confidence level (e.g., a score for screen process matching), and suggesting at least pre-built components even if the entire automated workflow does not match.

[0064] 5-6, block diagrams illustrating methods 500 and 600 performed by the environment 100 of FIG. 1 and / or the computing system 200 of FIG. 2, according to one or more embodiments. Note that the method 600 of FIG. 6 further describes the method 500 of FIG. 5.

[0065] Method 500 begins at block 510, where software 230 retrieves one or more automated processes (e.g., automated workflows) from a repository. In this regard, source data 512, robot video 514, and source code 516 from each of the one or more automated processes are used as inputs for comparison (e.g., at block 530). Source data 512 can be any source data including screenshots of elements (e.g., characteristics) of interest in the automated workflow. Source data 512 can be created in a mobile device environment. Robot video 514 includes captured images of an executing robot (RPA) in the automated workflow. Source code 516 includes any underlying automated workflow and / or robot.

[0066] In block 520, the software 230 captures the manual process. In this regard, user activity 523 and screen video 524 from the manual process are used as inputs for comparison (e.g., in block 530). User activity 523 can be any source data, including screenshots of characteristics (e.g., user actions) or elements that are the subject of the manual process. User activity 523 can be identified pixel-based and / or with optical character recognition (OCR) operations. Screen video 524 includes captured images of the manual process.

[0067] In block 530, software 230 executes recognition logic to perform a comparison between one or more automated processes of block 510 and the manual process of block 520. In this regard, the recognition logic of software 230 matches user actions with existing automation. Note that the recognition logic can be extended to include other forms of process documentation, such as detecting standard operating procedures and / or highly frequent action sequences for which no documentation or automation exists to prompt the user.

[0068] Turning to FIG. 6 , method 600 describes an example operation of the recognition logic of block 530 of FIG. 5 , according to one or more embodiments. Note that for ease of explanation, similar elements and items from FIG. 5 are reused in FIG. 6 . In this regard, elements such as source data 512, robot video 514, and source code 516 (shown in dashed lines) from an automated process, as well as user activity 523 and screen video 524 (shown in dashed lines) from a manual process, are reused as inputs to method 600 that occur further within recognition logic 530. Furthermore, note that block 601 is a core function to recognition logic 530, while block 602 is optional. In this regard, as each image in the window is viewed in turn in block 610, a target element may be matched in block 620 according to inputs 512, 514, 523, and 524.

[0069] With respect to optional block 602, the recommendation engine may further obtain activity definitions from source code in block 640, match activities in block 650, and match sequences in block 660 according to inputs 516, 523, and 524.

[0070] At decision block 670, the recommendation engine determines whether the matching criteria are met. If the matching criteria are not met, the method 600 returns (as indicated by the no arrow) to block 610. If the matching criteria are met, the method 600 proceeds (as indicated by the yes arrow) to block 680. At block 680, the recommendation engine returns the matching automation processes via prompts as described herein. According to one or more embodiments, the recommendation engine may also be used when a user (e.g., an RPA developer) is creating a new automation. In this regard, the recommendation engine may check what activities the user is performing and present the user with information that corresponding automations already exist.

[0071] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatuses, systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the flowcharts and block diagrams of the figures. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or actions or executes a combination of special-purpose hardware and computer instructions.

[0072] Although features and elements are described above in certain combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with the other features and elements. For example, in any of the methods and processes described herein, the described steps may be performed out of order in any order, and substeps not explicitly described or labeled may be performed. When used in reference to "A or B," it may include A, B, or A and B, and may be expanded as with longer lists. When using the notation X / Y, it may include X or Y. Alternatively, when using the notation X / Y, it may include X and Y. The X / Y notation may be expanded as with longer lists having the same described logic. Also, "coupled" or "operably coupled" may mean that objects are connected, but there may be zero or more intermediate objects between the connected objects. Also, any combination of the disclosed features / elements may be used in one or more embodiments.

[0073] Furthermore, the methods and processes described herein may be implemented in a computer program, software, and / or firmware (e.g., a computer program product) embodied in a computer-readable medium for execution by a computer or processor. That is, a computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a controller, processor, etc. to perform aspects of the present invention.

[0074] A computer-readable storage medium may be a tangible device capable of retaining and storing computer-readable program instructions. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not intended to be a transitory signal itself, such as, for example, radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire. Examples of computer-readable storage media include, but are not limited to, registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, optical media such as compact disks (CDs) and digital versatile disks (DVDs), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), and memory sticks.

[0075] The computer-readable program instructions described herein may be communicated and / or downloaded to a respective controller, processor, etc. from an apparatus, device, computer, or external storage via a connection, such as, for example, network communication. The computer-readable program instructions for carrying out operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute the computer-readable program instructions, utilizing state information from the computer-readable program instructions to personalize the electronic circuit to perform aspects of the present invention.

[0076] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0077] The description of various embodiments herein has been presented for purposes of explanation, but is not intended to be exhaustive or to limit the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications or technical improvements over commercially available technologies, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computing system a memory configured to store processor-executable instructions for a workflow recommendation assistant engine; at least one processor coupled to the memory; Including, The at least one processor executes the workflow recommendation assistant engine, capturing one or more images of a user interface corresponding to user activity; analyzing the one or more images and segments of the one or more images for action characteristics and for a plurality of robotic process automation applications by performing pattern matching, the action characteristics being user actions determined from the one or more images and segments captured within a sliding window, the pattern matching comprising: extracting the action characteristics and comparing the action characteristics with target characteristics of each of the plurality of robotic process automation systems, the action characteristics being compared in-sequence and out-of-sequence; determining a matched set of robotic process automations from the plurality of robotic process automations based on the pattern matching comparison; providing a prompt presenting the matched set of robotic process automation tasks; a computing system configured to cause the computing system to execute the

2. The at least one processor executes the workflow recommendation assistant engine, pre-processing the one or more images prior to performing the pattern matching to generate one or more corresponding hashes; The computing system of claim 1 , configured to cause the computing system to execute:

3. The computing system of claim 2 , wherein the pre-processing of the one or more images is performed with respect to mode inversion of the user interface.

4. The computing system of claim 1 , wherein the capturing of the one or more images includes capturing an area surrounding the user activity.

5. The computing system of claim 1 , wherein the sliding window is utilized to maintain a fixed number of the one or more images.

6. 10. The computing system of claim 1, wherein the workflow recommendation assistant engine determines a size of the sliding window and fills the sliding window with a sequence of generated hashes representing the one or more images in a first-in-first-out fashion.

7. 2. The computing system of claim 1, wherein the pattern matching comprises comparing a generated sequence of hashes representing the one or more images to stored sequences of hashes representing the plurality of robotic process automation images to determine the match set.

8. The computing system of claim 1 , wherein the prompt includes an option to run or set a trigger for each of the robotic process automation tasks in the match set.

9. The computing system of claim 1 , wherein the prompt presents a table of the match sets.

10. 1. A method of a workflow recommendation assistant engine stored as processor-executable instructions in a memory coupled to at least one processor, the method comprising: capturing one or more images of a user interface corresponding to user activity; analyzing the one or more images and segments of the one or more images for action characteristics and for a plurality of robotic process automation applications by performing pattern matching, the action characteristics being user actions determined from the one or more images and segments captured within a sliding window, the pattern matching comprising: extracting the action characteristics; comparing the action characteristics to target characteristics of each of the plurality of robotic process automation systems, the action characteristics being compared in-sequence and out-of-sequence; determining a matched set of robotic process automations from the plurality of robotic process automations based on the pattern matching comparison; providing a prompt that presents the matched set of robotic process automation.

11. The at least one processor executes the workflow recommendation assistant engine, The method of claim 10 , configured to cause a computing system to pre-process the one or more images prior to performing the pattern matching to generate one or more corresponding hashes.

12. The method of claim 11 , wherein the capturing of the one or more images includes capturing an area surrounding the user activity.

13. The method of claim 10 , wherein the sliding window is utilized to maintain a fixed number of the one or more images.

14. 11. The method of claim 10, wherein the workflow recommendation assistant engine determines a size of the sliding window and fills the sliding window with a sequence of generated hashes representing the one or more images in a first-in-first-out fashion.

15. 11. The method of claim 10, wherein the pattern matching comprises comparing a generated sequence of hashes representing the one or more images to stored sequences of hashes representing the plurality of robotic process automation devices to determine the match set.

16. The method of claim 10 , wherein the prompt includes an option to run or set a trigger for each of the robotic process automations in the matched set.

17. 10. The computing system of claim 1, wherein a score is generated and associated with each of the robotic process automations in the match set, each score indicating how well a manual process represented in the one or more images of the user activity matches a particular robotic process automation in the match set.

18. 10. The computing system of claim 1, wherein the workflow recommendation assistant engine takes a hash of the user activity, captures the one or more images of the user interface, sequences the hash, and runs the hash against a repository of existing robotic automation hashes in and out of the sequence to determine if there are any matching sets.

19. 10. The computing system of claim 1, wherein the action characteristics include utilizing robotic process automation agent logs, adding source code metadata to fame when creating robotic process automation videos, not recording the cursor when recording videos, excluding the cursor from the video based on a known cursor, performing keyframe caching, pre-building functionality for automated processing, highlighting elements with shapes when robotic process automation is running, and querying the shape of elements when robotic process automation is running.

20. 1. A computing system comprising: a memory configured to store processor-executable instructions for a workflow recommendation assistant engine; at least one processor coupled to the memory; Including, The at least one processor executes the workflow recommendation assistant engine, pre-processing one or more images of a user interface captured during a user activity to generate one or more corresponding hashes including action characteristics, the action characteristics including type of action, number of actions, proximity of actions, interactions with the user interface, and interactions with elements of the user interface; Maintaining a fixed number of the one or more corresponding hashes based on a sliding window, wherein the workflow recommendation assistant engine determines the size of the sliding window and fills the sliding window with a sequence of the one or more corresponding hashes in a first-in, first-out manner; pattern matching the one or more corresponding hashes within the sliding window by comparing the action characteristics in-sequence and out-of-sequence with target characteristics of each of a plurality of robotic process automations to determine a matching set of robots from the plurality of robotic process automations; dynamically changing the size of the sliding window by automatic reconfiguration to optimize performance of the pattern matching; presenting the matched set of robots in the user interface via prompts; a computing system configured to cause the computing system to execute the

Images