System and computer-implemented method for analyzing a test automation workflow for robotic process automation (RPA)

Abstract

Disclosed herein are systems and computer-implemented methods for analyzing test automation workflows associated with robotic process automation (RPA) applications. The computer-implemented method includes receiving a test automation workflow associated with the RPA application and analyzing the test automation workflow based on a set of predefined test automation rules via an artificial intelligence (AI) model associated with a workflow analyzer module. The computer-implemented method further includes determining one or more metrics associated with the analyzed test automation workflow and generating, via the AI ​​model, remediation activity data based on the determined one or more metrics.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Non - Provisional Application No. 17 / 082,561, filed Oct. 28, 2020, is a partial continuation thereof, and claims priority to Indian Patent Application No. 201911053188, filed Dec. 20, 2019, and claims priority to U.S. Non - Provisional Patent Application No. 16 / 931,917, filed Jul. 17, 2020, entitled "SYSTEM AND COMPUTER - IMPLEMENTED METHOD FOR ANALYZING A ROBOTIC PROCESS AUTOMATION (RPA) WORKFLOW", the entire contents of which are incorporated herein by reference.

[0002] The present invention generally relates to robotic process automation (RPA), and more specifically, to the analysis of test automation workflows for RPA.

Background Art

[0003] Generally, RPA can automate simple and repetitive tasks of manual input by a user into a computing system. Currently, manual user input is increasingly being performed by software robots using RPA tools. Since RPA is the performance of relatively simple and repeatable tasks that exist in large quantities within an enterprise, it promotes the spread of software automation. Currently, RPA tools are being utilized that assist software developers in designing, performing, deploying, and testing simple and repetitive tasks of an enterprise. For example, these tasks are designed using design tools and deployed using deployment tools. There are several designer tools (such as software tools) for designing the workflows of simple and repetitive tasks of RPA applications. Additionally, there are several software tools for testing such RPA applications using test automation.

[0004] However, these software tools lack the analysis of workflows for identifying and removing potential defects in test automation. For example, developers develop test automation with software tools. The developed test automation is transferred to the test team for identification. The test team will later be pointed out for defects and sent back. Therefore, manual testing of test automation is required, which is time-consuming and costly in terms of procedures. Furthermore, it is more difficult to debug defects in the test automation workflow in real time and avoid defects during execution.

[0005] Therefore, there is a need for software tools that can enable developers to design the workflow of test automation for RPA applications and debug defects in the test automation workflow at the design stage.

Summary of the Invention

[0006] Certain embodiments of the present invention provide solutions to problems and needs in the art that have not yet been fully identified, evaluated, or solved by current RPA technologies. For example, some embodiments of the present invention relate to the analysis of the workflow of test automation for RPA applications for identifying and removing potential defects or errors.

[0007] In an embodiment, a system and a computer-implemented method for analyzing a test automation workflow related to a robotic process automation (RPA) application are disclosed. The computer-implemented method includes receiving a test automation workflow related to the RPA application. The computer-implemented method includes analyzing the test automation workflow based on a set of predefined test automation rules via an AI model related to a workflow analyzer module. The computer-implemented method also includes determining one or more metrics related to the analyzed test automation workflow. The computer-implemented method further includes generating corrective activity data based on the determined one or more metrics via the AI model.

[0008] In another embodiment, a system for analyzing a test automation workflow related to a robotic process automation (RPA) application is disclosed. The system includes at least one processor and a memory storing instructions. The instructions are configured to cause the at least one processor to receive a test automation workflow related to the RPA application and analyze the test automation workflow based on a set of predefined test automation rules via an AI model related to a workflow analyzer module. The computer program instructions are further configured to cause the at least one processor to determine one or more metrics related to the analyzed test automation workflow and generate corrective activity data based on the determined one or more metrics via the AI model.

[0009] In yet another embodiment, the computer program is stored on a non-transitory computer-readable medium. The program is configured to receive, by at least one or more processors, a test automation workflow related to an RPA application and analyze the test automation workflow based on a set of predefined test automation rules via an AI model related to a workflow analyzer module. The program is further configured to determine, by one or more processors, one or more measurement criteria related to the analyzed test automation and generate, via the AI model, modified activity data based on the determined one or more measurement criteria.

Brief Description of the Drawings

[0010] For the advantages of specific embodiments of the present invention to be readily understood, a more specific description of the present invention briefly described above is depicted with reference to the specific embodiments illustrated in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the present invention and are not considered to limit its scope, but the present invention will be described and explained in further particularity and detail by using the following accompanying drawings.

[0011]

Figure 1

[0012]

Figure 2

[0013]

Figure 3

[0014]

Figure 4

[0015]

Figure 5

[0016]

Figure 6

[0017]

Figure 7

[0018]

Figure 8

[0019]

Figure 9

[0020]

Figure 10

[0021]

Figure 11

DETAILED DESCRIPTION OF THE INVENTION

[0022] (Detailed Description of the Embodiment) Some embodiments relate to a system (hereinafter also referred to as a "computing system") configured to analyze a test automation workflow related to an RPA application (also referred to as a "test automation workflow") to identify and remove potential defects in the RPA test automation workflow. In some embodiments, the computing system receives a test automation workflow from a design module and analyzes the received workflow to identify and remove defects. For example, the computing system uses an artificial intelligence (AI) model to analyze the workflow based on a set of predefined test automation rules. The AI model is pre-trained with standard test automation workflows, all possible errors within the workflow, and standard robotic enterprise framework documents. In some exemplary embodiments, a standard RPA workflow or any RPA workflow is converted into test cases or imported as test cases from a test automation project to train the AI model. One or more metrics are determined to generate corrective activity data from the analyzed test automation workflow.

[0023] In some embodiments, the AI model generates corrective activity data based on one or more determined measurement criteria. The corrective activity data is used to perform corrective activities for the analyzed test automation workflow. The corrective activity data includes a proposed message (e.g., a claim) or details instructing a user (e.g., a developer or tester) on how to perform a corrective activity for the analyzed workflow. The modified test automation file is configured to have improved execution time and storage requirements compared to the received workflow of the test automation. Further, the improvement in execution time and storage requirements reduces the computational overhead on the computing system. In this way, the test automation workflow is analyzed to debug defects before deployment using the computing systems and computer-implemented methods disclosed herein.

[0024] FIG. 1 is an architectural diagram showing an RPA system 100 according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that enables a developer or user to design and implement a workflow. The designer 110 provides solutions for application integration and automates third-party applications, administrative information technology (IT) tasks, and business IT processes. The designer 110 facilitates the development of an automation project that is a graphical representation of a business process. Briefly, the designer 110 facilitates the development and deployment of workflows and robots.

[0025] Automation projects enable the automation of rule-based processes by giving developers control over the execution order and relationships between a custom set of steps developed in a workflow defined herein as "activities". A commercial example of an embodiment of Designer 110 is UiPath Studio (trademark). Each activity includes actions such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows are nested or embedded.

[0026] Some types of workflows include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers, etc. Sequences are particularly suitable for linear processes that enable the flow from one activity to another without cluttering the test automation workflow. Flowcharts are particularly suitable for more complex business logic and enable the integration of decision-making and the connection of activities in more diverse ways through multiple branching logic operators. FSMs are particularly suitable for large-scale workflows. FSMs use a finite number of states triggered by conditions (i.e., transitions) or activities during their execution. Global exception handlers are particularly suitable for determining the behavior of the workflow when encountering execution errors or for debugging the process.

[0027] When a workflow is developed within Designer 110, the execution of the business process is coordinated by Conductor 120, which coordinates one or more robots 130 that execute the workflows developed within Designer 110. A commercial example of an embodiment of Conductor 120 is UiPath Orchestrator (trademark). Conductor 120 facilitates the management of the generation, monitoring, and deployment of resources in the environment. Conductor 120 operates as an integration point with third-party solutions and applications.

[0028] The conductor 120 manages all the robots 130 and connects and executes the robots 130 from a central point. The types of robots 130 to be managed include, but are not limited to, attended robots 132, unattended robots 134, development robots (similar to unattended robots 134 but used for development and testing purposes), and non-production robots (similar to attended robots 132 but used for development and testing purposes). The attended robots 132 are triggered by user events and operate alongside humans on the same computing system. The attended robots 132 are used together with the conductor 120 for centralized process deployment and logging media. The attended robots 132 assist human users in achieving various tasks and are triggered by user events. In some embodiments, the process cannot be started from the conductor 120 on this type of robot and / or they cannot be executed under a locked screen. In certain embodiments, the attended robots 132 are started from a robot tray or from a command prompt. The attended robots 132 operate under human supervision in some embodiments.

[0029] The unattended robot 134 operates without human attendance in a virtual environment and automates many processes. The unattended robot 134 is responsible for providing support for remote execution, monitoring, scheduling, and work queues. Debugging for all robot types is performed in some embodiments by the designer 110. Both the attended robot 132 and the unattended robot 134 automate various systems and applications including, but not limited to, mainframes, web applications, virtual machines (VMs), enterprise applications (e.g., those generated by SAP®, SalesForce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0030] The conductor 120 has various capabilities including, but not limited to, provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning includes creating and maintaining a connection between the robot 130 and the conductor 120 (e.g., a web application). Deployment includes ensuring the correct delivery of the package version to the robot 130 assigned for execution. Configuration includes maintaining and delivering the robot environment and process configurations. Queuing includes providing management of queues and queue items. Monitoring includes tracking specific data of the robot and maintaining user permissions. Logging includes saving and indexing logs to a database (e.g., an SQL database) and / or another storage mechanism (e.g., ElasticSearch® which provides the ability to store large datasets and execute queries quickly). The conductor 120 provides interconnectivity by operating as a central point of communication for third - party solutions and / or applications.

[0031] Robot 130 includes an execution agent that executes the workflow constructed by Designer 110. One commercial example of some embodiments of Robot(s) 130 is UiPath Robots (trademark). In some embodiments, by default, Robot 130 installs the Microsoft Windows (registered trademark) Service Control Manager (SCM) management service. As a result, Robot 130 opens an interactive Windows (registered trademark) session under the local system account and has the rights of a Windows (registered trademark) service.

[0032] In some embodiments, Robot 130 is installed in user mode. For such a Robot 130, it means having the same rights as the user on whose machine the given Robot 130 is installed. This feature is also available for high-density (HD) robots that ensure maximum utilization of each machine. In some embodiments, any type of Robot 130 is configured in an HD environment.

[0033] Robot 130 in some embodiments is divided into multiple components, each specialized for a specific automation task. Robot components in some embodiments include, but are not limited to, the SCM management robot service, the user mode robot service, the executor, the agent, and the command line. The SCM management robot service manages and monitors the Windows (registered trademark) session and operates as a proxy between the Conductor 120 and the execution host (i.e., the computing system on which Robot 130 is executed). These services are entrusted with managing the credentials of Robot 130. The console application is launched by the SCM under the local system.

[0034] In some embodiments, the user-mode robot service manages and monitors the Windows® session and operates as a proxy between the conductor 120 and the execution host. The user-mode robot service is entrusted with managing the qualification information of the robot 130. If the SCM management robot service is not installed, the Windows® application is automatically launched.

[0035] The executor performs (i.e., workflows) jobs given under the Windows® session. The executor is aware of the dots per inch (DPI) settings per monitor. The agent includes a Windows® Presentation Foundation (WPF) application that displays jobs available in the system tray window. The agent includes a client of the service. The agent sends requests to start or stop jobs and change settings. The command line is a client of the service. The command line is a console application that requests the start of a job and waits for its output.

[0036] As described above, the fact that the components of the robot 130 are split helps developers, support users, and computing systems to more easily execute, identify, and track what each component is doing. In this way, special behaviors are configured for each component, such as setting different firewall rules for the executor and the service. The executor always recognizes the DPI settings per monitor in some embodiments. As a result, the workflow is executed at any DPI, regardless of the configuration of the computing system on which the workflow was created. Also, in some embodiments, projects from the designer 110 are made independent of the browser zoom level. In the case of applications marked as not recognizing or intentionally not recognizing DPI, DPI is disabled in some embodiments.

[0037] Figure 2 is an architecture diagram showing the deployed RPA system 200 according to an embodiment of the present invention. In some embodiments, the RPA system 200 is or is part of the RPA system 100 of FIG. 1. It should be noted that either the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 210 includes an executor 212, an agent 214, and a designer 216 (e.g., designer 110). However, in some embodiments, the designer 216 is not running on the robot application 210. The executor 212 executes processes. As shown in FIG. 2, a plurality of business projects (i.e., executors 212) are executed simultaneously. The agent 214 (e.g., Windows® service) is, in this embodiment, a single connection point for all executors 212. All messages in this embodiment are recorded in the conductor 230, which further processes them via the database server 240, the indexer server 250, or both. As described above with respect to FIG. 1, the executor 212 is a robot component.

[0038] In some embodiments, the robot represents an association between a machine name and a user name. The robot manages multiple executors simultaneously. In a computing system (such as Windows® Server 2012) that supports multiple interactive sessions running simultaneously, multiple robots are running simultaneously, each running in a separate Windows® session using a unique user name. This is referred to as the above-mentioned HD robot.

[0039] Agent 214 is also responsible for sending the state of the robot (e.g., periodically sending a "heartbeat" message indicating that the robot is still functioning) and downloading the required version of the package to be executed. The communication between Agent 214 and Conducter 230 is, in some embodiments, always initiated by Agent 214. In the notification scenario, Agent 214 opens a WebSocket channel that is later used by Conducter 230 to send commands (e.g., start, stop, etc.) to the robot.

[0040] On the server side, there are a presentation layer (web application 232, Open Data Protocol (OData) Representational State Transfer (REST) Application Programming Interface (API) endpoint 234, notification and monitoring API 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240, indexer server 250). The conductor 230 includes the web application 232, the OData REST API endpoint 234, the notification and monitoring API 236, and the API implementation / business logic 238. In some embodiments, most actions that a user performs at the interface of the conductor 220 (e.g., via the browser 220) are executed by calling various APIs. Such operations include, but are not limited to, launching jobs on a robot, adding / removing data in a queue, scheduling jobs to be executed unattended, etc., without departing from the scope of the present invention. The web application 232 is the visual layer of the server platform. In this embodiment, the web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, script language, or any other format may be used without departing from the scope of the present invention. The user interacts with the web page from the web application 232 via the browser 220 in this embodiment to perform various operations for controlling the conductor 230. For example, the user can create a robot group, assign packages to robots, analyze logs for each robot and / or process, start and stop robots, etc.

[0041] In addition to the web application 232, the conductor 230 also includes a service layer that exposes an OData REST API endpoint 234. However, other endpoints are included without departing from the scope of the present invention. The REST API is consumed by both the web application 232 and the agent 214. The agent 214 is, in this embodiment, a supervisor for one or more robots on a client computer.

[0042] The REST API of this embodiment covers configuration, logging, monitoring, and queuing functions. The configuration endpoints are used, in some embodiments, to define and configure the users, permissions, robots, assets, releases, and environments of the application. The logging REST endpoints are used to log various information such as, for example, errors, explicit messages sent by robots, and other environment-specific information. The deployment REST endpoints are used by robots to query the version of the package to be executed when a job start command is used in the conductor 230. The queuing REST endpoints are responsible for the management of queues and queue items such as adding data to a queue, retrieving transactions from a queue, and setting the status of a transaction.

[0043] The monitoring of the REST endpoints monitors the web application 232 and the agent 214. The notification and monitoring API 236 is related to the REST endpoints used for the registration of the agent 214, the distribution of configuration settings to the agent 214, and the sending and receiving of notifications from the server and the agent 214. The notification and monitoring API 236 uses WebSocket communication in some embodiments.

[0044] In this embodiment, the persistent layer includes a pair of server-database servers 240 (e.g., SQL servers) and an indexer server 250. The database server 240 in this embodiment stores configurations such as robots, robot groups, related processes, users, roles, schedules, etc. In some embodiments, this information is managed via the web application 232. The database server 240 manages queues and queue items. In some embodiments, the database server 240 stores (in addition to or instead of the indexer server 250) messages recorded by the robots.

[0045] Optionally in some embodiments, the indexer server 250 stores information recorded by the robots and creates indexes. In certain embodiments, the indexer server 250 is deactivated via configuration settings. In some embodiments, the indexer server 250 uses ElasticSearch (registered trademark), an open-source project full-text search engine. Messages recorded by the robots (e.g., using activities such as log messages or line writes) are sent to the indexer server 250 via logging REST endpoint(s), where they are indexed for future use.

[0046] Figure 3 is an architecture diagram showing the relationship 300 between a designer 310, user-defined activities 320, user interface (UI) automation activities 330, and a driver 340 according to an embodiment of the present invention. As described above, a developer uses the designer 310 to develop a workflow to be performed by a robot. According to some embodiments, the designer 310 is a design module of an integrated development environment (IDE) that enables a user or developer to perform one or more functions related to the workflow. The functions include editing, coding, debugging, browsing, saving, modifying, etc. for the workflow. In some exemplary embodiments, the designer 310 facilitates analyzing the workflow. Further, in some embodiments, the designer 310 is configured to compare two or more workflows, such as in a multi-window user interface. The workflow includes user-defined activities 320 and UI automation activities 330. In some embodiments, non-text visual components in an image can be identified, which is referred to herein as computer vision (CV). Some CV activities related to such components include, but are not limited to, click, type, get text, hover, detect presence of an element, update scope, highlight, etc. In some embodiments, click identifies an element using, for example, CV, optical character recognition (OCR), fuzzy text matching, and multi-anchor and clicks it. Type identifies an element using the above and types within the element. Getting text identifies a location of specific text and scans it using OCR. Hover identifies an element and hovers over it. Detecting the presence or absence of an element uses the techniques described above to check whether the presence or absence of an element on the screen is detected. In some embodiments, there are hundreds or even thousands of activities implemented in the designer 310. However, any number and / or type of activities can be utilized without departing from the scope of the present invention.

[0047] The UI automation activity 330 is a subset of special low-level activities described in low-level code (e.g., CV activities) that facilitate interaction with the screen. In some embodiments, the UI automation activity 330 includes activities related to debugging or fixing defects in the workflow. The UI automation activity 330 facilitates these interactions via a driver 340 that enables the robot to interact with the desired software. For example, the driver 340 includes an operating system (OS) driver 342, a browser driver 344, a VM driver 346, an enterprise application driver 348, and the like.

[0048] The driver 340 interacts with the OS driver 342 at a low level, such as by looking for hooks or monitoring keys. They may facilitate integration with Chrome (registered trademark), IE (registered trademark), Citrix (registered trademark), SAP (registered trademark), etc. For example, a "click" activity plays the same role in these different applications via the driver 340. The driver 340 enables the execution of RPA applications in the RPA system.

[0049] Figure 4 is an architectural diagram showing an RPA system 400 according to an embodiment of the present invention. In some embodiments, the RPA system 400 includes the RPA systems 100 and / or 200 of FIGS. 1 and / or 2. The RPA system 400 includes a plurality of client computing systems 410 (e.g., executing robots). In some embodiments, the plurality of client computing systems 410 are configured to analyze a workflow. Further, the analyzed workflow is deployed to the plurality of client computing systems 410. The computing system 410 communicates with a conductor computing system 420 via a web application executed thereon. The conductor computing system 420, in turn, communicates with a database server 430 (e.g., database server 240) and any indexer server 440 (e.g., any indexer server 250).

[0050] Regarding FIGS. 1 and 3, although web applications are used in these embodiments, it should be noted that any suitable client / server software can be used without departing from the scope of the present invention. For example, the conductor executes a server-side application that communicates with a non-web-based client software application on the client computing system.

[0051] Figure 5 is an architectural diagram showing a computing system 500 configured to analyze a test automation workflow related to an RPA application, according to an embodiment of the present invention. In some embodiments, computing system 500 includes one or more computing systems depicted and / or described herein. Computing system 500 includes a bus 510 or other communication mechanism for communicating information, and one or more processors 520 coupled to bus 510 for processing information. The processor(s) 520 includes any type of general or special-purpose processor, including a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor(s) 520 also has multiple processing cores, and at least some of the cores are configured to perform specific functions. In some embodiments, parallel processing is used. In certain embodiments, at least one processor(s) 520 is a neuromorphic circuit that includes processing elements that mimic biological neurons. In some embodiments, the neuromorphic circuit does not require the typical components of a von Neumann computing architecture.

[0052] Computing system 500 further includes a memory 530 for storing information and instructions to be executed by processor(s) 520. Memory 530 is composed of a random access memory (RAM), a read-only memory (ROM), a flash memory, a cache, a static storage device such as a magnetic disk or optical disk, or other types of non-transitory computer-readable media, or any combination thereof. The non-transitory computer-readable media is any available media accessible by processor(s) 520 and includes volatile media, non-volatile media, or both. Also, the media includes removable, non-removable, or both.

[0053] Furthermore, computing system 500 includes a communication device 540, such as a transceiver, to provide access to a communication network via a wireless and / or wired connection. In some embodiments, the communication device 540 is configured to use frequency division multiple access (FDMA), single carrier FDMA (SC-FDMA), time division multiple access (TDMA), code division multiple access (CDMA), orthogonal frequency division multiplexing (OFDM), orthogonal frequency division multiple access (OFDMA), global system for mobile communications (GSM), general packet radio service (GPRS), universal mobile telecommunications system (UMTS), cdma2000, wideband CDMA (W-CDMA), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), high-speed packet access (HSPA), long term evolution (LTE), LTE advanced (LTE-A), 802.11x, Wi-Fi, Zigbee, ultra-wideband radio (UWB), 802.16x, 802.15, Home Node-B (HnB), Bluetooth, radio frequency identification (RFID), Infrared Data Association (IrDA), near-field communications (NFC), fifth generation (5G), New Radio (NR), any combination thereof, and / or any other currently existing or future-implemented communication standard and / or protocol without departing from the scope of the present invention.In some embodiments, the communication device 540 includes one or more antennas that, without departing from the scope of the present invention, are a single antenna, an array antenna, a phased antenna, a switched antenna, a beamforming antenna, a beam steering antenna, combinations thereof, and / or any other antenna configuration.

[0054] The processor(s) 520 is further coupled via the bus 510 to a display 550 such as a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, a Retina (registered trademark) display, an IPS (In-Plane Switching) display, or any other suitable display for presenting information to the user. The display 550 is configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, etc. using a resistive method, a capacitive method, a surface acoustic wave (SAW) capacitive method, an infrared method, an optical imaging method, a dispersion signal method, an acoustic pulse recognition method, a frustrated total internal reflection method, etc. Any suitable display device and haptic I / O may be used without departing from the scope of the present invention.

[0055] Keyboard 560 and cursor control device 570, such as a computer mouse, touchpad, etc., are further coupled to bus 510 to enable a user to interface with the computing system. However, in certain embodiments, there is no physical keyboard and mouse, and the user interacts with the device only via display 550 and / or a touchpad (not shown). The type and combination of any input device are used as a matter of design choice. In certain embodiments, there is no physical input device and / or display. For example, the user interacts remotely with computing system 500 via another computing system that is communicating with computing system 500, and computing system 500 operates autonomously.

[0056] Memory 530 stores software modules that provide functionality when executed by processor(s) 520. The modules include operating system 532 for computing system 500. The modules further include workflow analyzer module 534 configured to execute all or part of the processes described herein or derivatives thereof. Computing system 500 also includes one or more additional functional modules 536 that include additional functionality. In some embodiments, workflow analyzer module 534 is configured to analyze test automation created for a software application. Workflow analyzer module 534 is also configured to verify the configuration of all activities and check for missing data, inaccurate data, and / or the like.

[0057] One skilled in the art would understand that the "system" can be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device, or a combination of devices, without departing from the scope of the present invention. Presenting the functions described above as being performed by a "system" is not intended to limit the scope of the present invention in any way, but rather to provide an example of many embodiments of the present invention. In fact, the methods, systems, and devices disclosed herein are implemented in a localized form and a distributed form that is consistent with computing technologies including cloud computing systems.

[0058] It should be noted that some of the system features described herein are presented as modules in order to emphasize implementation independence more. For example, a module can be implemented as a hardware circuit including custom very large scale integration (VLSI) circuits or gate arrays, logic chips, transistors, or other off-the-shelf semiconductors such as individual components. Also, a module can be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, and graphics processing units.

[0059] The module is also at least partially implemented in software for being performed by various types of processors. For example, a specified unit of executable code includes one or more physical or logical blocks of computer instructions that are organized as, for example, objects, procedures, or functions. Nevertheless, a specified module that is executable need not be physically located together and may include modules when logically combined and include separate instructions stored in different locations for achieving the purpose stated for the module. Further, the module is stored in a computer-readable medium such as, for example, a hard disk drive, a flash device, RAM, a tape, and / or any other non-transitory computer-readable medium used for storing data without departing from the scope of the present invention.

[0060] In fact, a module of executable code may be a single instruction, or a number of instructions, and even further, may be distributed among multiple different code segments, between different programs, and among multiple memory devices. Similarly, the operating data is specified within the module, shown herein, and embodied and organized in any suitable form within any suitable type of data structure. The operating data is collected as a single data set or is distributed in different locations across different storage devices and exists at least partially simply as electronic signals on a system or network.

[0061] FIG. 6 is an architectural diagram showing a workflow analyzer module 600 according to an embodiment of the present invention. In some embodiments, the workflow analyzer module 600 is similar to or the same as the workflow analyzer module 534 illustrated in FIG. 5. Also, in some embodiments, the workflow analyzer module 600 is embodied within the designer 110. The workflow analyzer module 600 includes a data collection sub-module 610, an analyzer sub-module 620, and a measurement criterion determination module 630, which are executed by a processor(s) 520 to perform their specific functions to analyze the workflow of test automation related to the RPA application.

[0062] The data collection sub-module 610 receives the workflow of test automation as a data file from the designer 110. The data file includes, but is not limited to, a solution design document (SDD), a process design instruction (PDI), an object design instruction (ODI), or business process (BP) code. For example, a user or developer uses the designer 110 to develop a data file of the workflow of test automation. In some embodiments, the data collection sub-module 610 receives the workflow of test automation as a combination of a set of test cases. A test case defines a single test that is performed to achieve specific software test objectives, such as exercising a specific program path or verifying compliance with specific requirements, and includes specifications of input, execution conditions, test procedures, and expected results.

[0063] In certain embodiments, the data collection submodule 610 provides an enable option to the user. For example, when the user enables the enable option, the data collection submodule 610 obtains one or more test activities (i.e., sequences) of a test automation workflow of an RPA application (such as live data from the user). Further, the test automation workflow or sequence of test automation workflows obtained by the data collection submodule 610 is also used by the analyzer submodule 620.

[0064] In some embodiments, the analyzer submodule 620 includes a training submodule 622, an artificial intelligence (AI) model (hereinafter referred to as "AI model") 624, and a test automation rule submodule 626. The analyzer submodule 620 analyzes the test automation workflow to output the analyzed workflow of the test automation. In some embodiments, the analyzer submodule 620 uses the AI model 624 to analyze the received workflow of the test automation.

[0065] In some embodiments, the AI model 624 corresponds to a pre-trained AI model that analyzes the received workflow of test automation. In some embodiments, the AI model 624 is pre-trained based on training data. In some exemplary embodiments, the training data is stored in the training sub-module 622. The training data includes at least one of a standard test automation workflow, an error in the test automation workflow, and a standard framework document. The training data also includes sequences within the test automation workflow and all possible defects related to the test automation workflow (including solutions for addressing the defects). In some exemplary embodiments, the training data is based on previous functional tests of web and mobile applications, visual tests of user interfaces, and the positions of UI elements and automated correction element selectors. In some embodiments, the defects include human errors such as incorrect data input for test automation or missing data input in test automation. In another example, the AI model 624 uses the knowledge of the training data to predict defects related to the test automation workflow and outputs an analyzed test automation workflow (also referred to as the "analyzed test automation workflow"). The analyzed test automation workflow includes the test automation workflow and the respective predicted defect information.

[0066] In one embodiment, the AI model 624 includes an ML model such as a recurrent neural network model (e.g., a long short-term memory (LSTM) model). Also, in certain embodiments, the ML model is self-trained. For example, the ML model is trained to learn one or more patterns in the test automation workflow. The one or more patterns correspond to tests that are repeated in previous workflows of test automation. The ML model provides the one or more patterns to the AI model 624 for analysis of the test automation workflow related to the RPA application. In some exemplary embodiments, the ML model is a pre-constructed ML model stored in the memory 530. In some alternative embodiments, the ML model is customized by the user or accessed from an open platform (e.g., an open source community), a third-party institution, etc. For example, when a defect occurs in the test automation workflow at runtime, the ML model learns the defect and learns how to address the defect.

[0067] In certain embodiments, the AI model 624 uses a test automation rule sub-module 626 to analyze the received workflow. In some embodiments, the test automation rule sub-module 626 includes a set of predefined test automation instructions for analyzing the workflow (referred to as the "set of predefined test automation rules"). These sets of predefined test automation rules include a predefined number of loops, a predefined number of conditions, a structured design, at least one verification point, one or more annotations, a global exception handler, and one or more conditions. The global exception handler is related to unexpected environmental problems. Each of the sets of predefined test automation rules is associated with each of the sets of test cases of the test automation workflow.

[0068] In some embodiments, a predefined number of loops and predefined number of conditions are utilized to verify that the test results of each of the set of test cases can be compared over time. Examples of loops include "while loop", "do while", "For each loop", etc. Further, examples of conditions include "if condition", "switch condition", etc. In some embodiments, the structural design includes a Behavior Driven Development (BDD) test design structure having "Given", "When and Then" sections. In some embodiments, each test case of the set of test cases includes at least one verification point.

[0069] In some embodiments, one or more conditions of a predefined set of test automation rules are related to checking the use of a similar sequence several times through a test case. Such sequences are extracted into another reusable workflow or library. For this purpose, the analyzer sub-module 620 utilizes this condition to extract one or more redundant sequences of one or more of the set of test cases into another reusable library for the analysis of the test automation workflow. For example, two sequences of test cases that are redundantly used to check the functionality of a module of a test automation workflow are extracted as a reusable library for the analysis of the test automation workflow. This reusability improves the maintainability of the test cases for analyzing the test automation workflow. Thus, the test automation workflow for an RPA application (such as a desktop application) is analyzed in an efficient and feasible manner.

[0070] It should be understood that the test automation rule sub-module 626 performs each rule of a set of predefined test automation rules on the test automation workflow and outputs the analyzed workflow of the test automation. The rules may further include one or more additional rules and one or more additional categories, without departing from the scope of the embodiments. The analyzed workflow includes the test automation workflow and a report including the validity of the rules. In some embodiments, the test automation rule sub-module 626 provides the user with selection options for selecting one or more rules from a set of predefined test automation rules. Further, the test automation rule sub-module 626 performs the one or more selected rules on the test automation workflow and outputs the analyzed workflow of the test automation.

[0071] In some embodiments, the analyzed workflow of the test automation is provided to the measurement criteria determination sub-module 630 of the workflow analyzer module 600. The measurement criteria determination sub-module 630 determines one or more measurement criteria related to the analyzed workflow of the test automation to generate the modification activity data. In some exemplary embodiments, the modification activity data is stored in a modification module (not shown in FIG. 6). The modification activity data is used to execute the modification activities of the analyzed test automation workflow.

[0072] According to some embodiments, the workflow analyzer module 600 further includes one or more additional modules, such as a modification module (not shown). The modification module performs one or more modification activities using one or more measurement criteria determined by the measurement criteria determination module 630. In some exemplary embodiments, the modification activities include providing the user with feedback regarding the better potential of the test automation workflow or activity, generating a report regarding one or more measurement criteria related to the test automation workflow, generating a warning message or an error message related to the test automation workflow at the issue time or compile time, or outputting the activity number and activity name corresponding to the error activity within the test automation workflow. Further, the feedback also enables the user to handle exceptions, validate data, and route an AI model (e.g., AI model 624) for retraining, which provides continuous improvement to the AI model 624.

[0073] In some embodiments, the modification module provides the user with feedback regarding the better potential of the test automation workflow. According to some exemplary embodiments, the feedback includes a proposed message for modifying the modified workflow of the test automation or the analyzed workflow of the test automation. The proposed message includes a claim or any other information for modifying the test automation workflow.

[0074] According to some embodiments, the feedback is provided by the AI model 624. For example, the AI model is trained using best practice documents and frameworks (e.g., Robotic Enterprise framework) to construct high-quality test automation workflows for RPA applications. In some embodiments, the correction module generates a report on the metrics related to the analyzed workflow of the test automation. In some embodiments, the report on the generated metrics is presented as a percentage. In certain embodiments, the correction module generates a warning message or an error message related to the analyzed workflow of the test automation. The warning message or error message includes a summary containing details or information related to the defects of the analyzed workflow of the test automation.

[0075] In some embodiments, the AI model 624 predicts one or more defects (e.g., incorrect input data) in the test automation workflow based on the determined one or more metrics. The one or more metrics include one or more of an extensibility value related to the analyzed workflow, a maintainability value related to the analyzed workflow, a readability value related to the analyzed workflow, a clarity value related to the analyzed workflow, an efficiency value related to the analyzed workflow, a cyclomatic complexity value related to the analyzed workflow, a reusability value related to the analyzed workflow, a reliability value related to the analyzed workflow, or an accuracy value related to the analyzed workflow. In some embodiments, the one or more metrics are presented in percentage form (via the display 550). Further, the AI model 624 corrects the test automation workflow to remove one or more defects.

[0076] In this way, when the workflow analyzer module 600 is executed by the processor(s) 520, it performs the above-described operations to debug the test automation workflow before the deployment of the analyzed workflow for the RPA application. In some embodiments, the deployment is executed after defect removal. The RPA application is deployed after defect removal. As a result, at the design stage, an accurate test automation workflow for the RPA application will be designed or developed. The accurate workflow of test automation includes instructions that are as few as possible for performing the user-defined process (i.e., a workflow with fewer storage requirements and less execution time). For example, the workflow analyzer module 600 identifies defects related to the workflow (including activities that fail a set of rule validations) and modifies the workflow to remove the defects in order to design an accurate workflow for test automation of the RPA application.

[0077] In some embodiments, the workflow analyzer module 600 removes defects by using interleaving techniques (e.g., interleaved code development). Further, the accurate workflow provides improved metrics (e.g., improved reliability values, reusability values, accuracy values, etc.) compared to the workflow with defects. In some further embodiments, the workflow analyzer 600 integrates with various CI / CD (continuous integration and continuous delivery) tools as well as other applications and services to provide timing analysis.

[0078] In summary, the workflow analyzer module 600 represents a tool for static code review of existing workflows. For example, the workflow analyzer module 600 uses predefined rules hardcoded for how a test automation workflow should look, enabling a user to define their rules based on the predefined rules. These predefined rules may include, for example, a company's policies. Further, the AI model 624 (or AI component) of the workflow analyzer module 600 is pre-trained, for example, for test automation workflows based on a large set of customer data.

[0079] In one example, during the design time of a workflow, the workflow analyzer module 600 analyzes the structure of the workflow and sends notifications regarding potential problems, warnings, and improvements. Based on the defined rules or policies, these notifications may be suggestions or may prevent the user from publishing the workflow if the workflow does not meet the defined rules or policies.

[0080] For example, refer to FIG. 10 which is a GUI 1000 showing a designer panel according to an embodiment of the present invention. In this embodiment, when the workflow analyzer module is triggered, the workflow analyzer module scans the entire workflow and analyzes whether the workflow is compliant. Based on the predefined rules or policies, the workflow analyzer module compares the workflow against the rules or policies and generates notifications (e.g., warnings) for non-compliant rules or policies.

[0081] It should be understood that predefined rules or policies are predefined by one or more users (e.g., test experts) based on case data of past experiences in test automation customer projects. In some embodiments, test cases include at least one assertion and a limited number of loops and conditions to ensure that test results can be reliably compared over time. Also, in some embodiments, test cases are atomic and executable without additional preconditions, are not replicated, and may rather include selectors extracted into an object browser.

[0082] It should be noted that customers may also provide customer-specific rules for the test automation workflow analyzer module. In some embodiments, customers define their own code rules to enforce their company-specific standards and guidelines. In this embodiment, each test case should have log entries for each step and version within GitHub (trademark).

[0083] It should be further noted that certain embodiments include pre-trained rules of potential AI for the test automation workflow analyzer. In such embodiments, the AI algorithm may include a pre-trained set of rules. These pre-trained sets of rules are collected from large datasets from existing customer projects. For example, FIG. 11 is a workflow diagram showing a system 1100 for continuously updating a database including a pre-trained set of rules according to an embodiment of the present invention.

[0084] As shown in FIG. 11, customer 1105 can execute multiple workflows on various computing systems 1110 1 ... 1110 N hosting UiPath Studio (trademark). In some embodiments, each computing system 1110 1 ... 1110 NIt may continuously send data to UiPath Cloud 115. UiPath Cloud 1115 may include a data collector 1120 and a machine learning (ML) algorithm 1125. The data collector 1120 may collect data received from each computing system 110 1 ~1110 N . This data includes information about its test automation workflow, including how the workflow was created, how frequently the workflow is executed, etc. For example, the ML algorithm 1125 may determine whether the saved test automation workflow is good and can be used to derive rules. In one example, the ML algorithm checks the following: whether the test cases were executed at the customer's execution frequency without any exceptions, how stable the test cases were over time, etc. Based on this analysis, the ML algorithm creates a set of predefined rules and can provide feedback of the set of predefined rules to the test automation workflow analyzer module, which is executed on the computing system 1110 1 …1110 N .

[0085] FIG. 7 is a block diagram showing a representation 700 depicting a set of predefined test automation rules according to an embodiment of the present invention. In an exemplary embodiment, the set of predefined test automation rules includes a structured design such as a test design structure 702, a predefined number of loops 704, a predefined number of conditions 706, annotations 708, one verification point 710, and a global exception handler 712 related to unexpected environmental problems.

[0086] FIG. 8 is a GUI showing a user interface 800 for the analysis of a test automation workflow 802 according to an embodiment of the present invention. The workflow 802 includes test cases such as test case 804 or combinations of one or more test cases. In some embodiments, the workflow 802 is provided as an input to the computing system 500 (i.e., a workflow from a user). The computing system 500 performs a workflow analyzer module 534 to analyze the workflow 700 using the AI model 624. In some exemplary embodiments, the AI model 624 is deployed using a drag-and-drop function (not shown in FIG. 8) in the UI 800.

[0087] FIG. 9 is a flowchart showing a method 900 for analyzing a test automation workflow related to an RPA application according to an embodiment of the present invention. In some embodiments, the method 900 starts at step 910.

[0088] At step 910, the method 900 includes receiving a test automation workflow of an RPA application. In some embodiments, the workflow for test automation is obtained as a workflow file. The workflow file includes, but is not limited to, a solution design document (SDD), a process design instruction (PDI), an object design instruction (ODI), or business process (BP) code. In some other embodiments, the workflow is obtained as one or more activities from a desktop recorder.

[0089] In step 920, method 900 includes analyzing a test automation workflow using an AI model (e.g., AI model 624) of workflow analyzer module 600 based on a set of predefined test automation rules (e.g., set 700 of test automation rules). In some exemplary embodiments, AI model 624 corresponds to a pre-trained AI model that includes training data. The training data includes at least one of a standard test automation workflow, errors in a test automation workflow, and standard framework documents. In some embodiments, the set of predefined test automation rules is performed based on the pre-trained AI model.

[0090] In step 930, method 900 includes determining one or more metrics related to the analyzed test automation workflow. For example, the one or more metrics include one or more of a scalability value related to the analyzed workflow, a maintainability value related to the analyzed workflow, a readability value related to the analyzed workflow, an efficiency value related to the analyzed workflow, a cyclomatic complexity value related to the analyzed workflow, or an accuracy value related to the analyzed workflow.

[0091] In step 940, method 900 includes generating corrective activity data based on the determined one or more metrics via the AI model. In some embodiments, the corrective activity data is used to perform corrective activities for the test automation workflow. The corrective activities include predicting one or more defects in the test automation workflow based on the determined one or more metrics via the AI model and modifying the workflow to remove the one or more defects via the AI model.

[0092] The process steps executed in FIG. 9 are executed by a computer program that encodes instructions to a processor (s) to execute at least a part of the process (es) described in FIG. 9 according to an embodiment of the present invention. The computer program is stored in a non - transitory computer - readable medium. The computer - readable medium is a hard - disk drive, a flash device, RAM, a tape, and / or any other such medium or combination of media used to store data, but is not limited thereto. The computer program includes encoded instructions for controlling a processor (s) of a computing system (e.g., the processor (s) 520 of the computing system 500 in FIG. 5) to implement all or part of the process steps described in FIG. 9, which is also stored in a computer - readable medium.

[0093] The computer program can be implemented in a hardware, software, or hybrid implementation. The computer program can be composed of modules that communicate operably with each other and is designed to send information or instructions to a display. The computer program can be configured to operate on a general - purpose computer, an ASIC, or any other suitable device.

[0094] It will be readily understood that the components of the various embodiments of the present invention can be arranged and designed in a variety of different configurations as generally described and illustrated herein. Accordingly, the detailed description of the embodiments of the present invention as represented in the accompanying figures is not intended to limit the scope of the present invention as claimed, but represents only selected embodiments of the present invention.

[0095] The features, structures, or characteristics of the invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, references throughout this specification to "certain embodiments", "some embodiments", or similar language mean that the particular features, structures, or characteristics described in connection with the embodiments are included in at least one embodiment of the invention. Thus, the appearances of "in certain embodiments", "in some embodiments", "in other embodiments", or similar language throughout this specification are not necessarily referring to the same group of all embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0096] It should be noted that references throughout this specification to features, advantages, or similar language do not mean that all of the features and advantages realized by the invention should be, or are, in any single embodiment of the invention. Rather, language referring to features and advantages is understood to mean that a particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Thus, discussions of features and advantages throughout this specification, as well as similar language, refer to the same or different embodiments, without necessarily being to the same embodiment.

[0097] Furthermore, the described features, advantages, and characteristics of the invention may be combined in any suitable manner in one or more embodiments. Those skilled in the relevant art will recognize that the invention may be practiced without one or more of the specific features or advantages of a particular one or more embodiments. In other instances, additional features and advantages may be recognized in particular embodiments that are not present in all embodiments of the invention.

[0098] Those having ordinary skill in the art will readily understand that the present invention as described above may be practiced using steps in a different order and / or using hardware elements of a different configuration than those disclosed. Accordingly, while the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain changes, modifications, and alternative configurations may become apparent while remaining within the spirit and scope of the invention. Therefore, reference should be made to the appended claims to determine the scope of the present invention.

Claims

1. A computer-implemented method for analyzing a test automation workflow related to a robotic process automation (RPA) application, the method comprising: Receiving the test automation workflow related to the RPA application; Analyzing the test automation workflow based on a set of predefined test automation rules via an artificial intelligence (AI) model related to a workflow analyzer module; Determining one or more metrics related to the analyzed workflow of the test automation; Generating correction activity data based on the determined one or more metrics via the AI model.

2. The method of claim 1, wherein the set of predefined test automation rules includes at least one of a predefined number of loops, a predefined number of conditions, a predefined structured design, or at least one verification point.

3. The method of claim 1, further comprising performing a correction activity for the analyzed workflow of the test automation based on the correction activity data.

4. Performing the correction activity further comprises: Predicting one or more defects in the test automation workflow based on the determined one or more metrics via the AI model; Modifying the test automation workflow via the AI model to remove the one or more defects.

5. The method of claim 1, wherein the test automation workflow includes a combination of a set of test cases.

6. The method of claim 5, further comprising extracting one or more redundant sequences of at least one test case of the set of test cases into another reusable library for analysis of the test automation workflow.

7. The method of claim 1, wherein the AI model corresponds to a pre-trained AI model including training data.

8. Analyzing the workflow of the test automation further includes performing the set of predefined test automation rules based on the pre-trained AI model, the method according to claim 7.

9. The pre-training data includes at least one of a standard test automation workflow, an error in the test automation workflow, and a standard framework document, the method according to claim 7.

10. A system for analyzing a workflow of test automation related to a robotic process automation (RPA) application, the system comprising: A memory configured to store instructions; At least one processor, and The instructions are configured to cause the at least one processor to perform the following, a system. Receiving the workflow of the test automation related to the RPA application; Analyzing the workflow of the test automation based on a set of predefined test automation rules via an artificial intelligence (AI) model related to a workflow analyzer module; Determining one or more metrics related to the analyzed workflow of the test automation, and Generating correction activity data based on the determined one or more metrics via the AI model.

11. The set of predefined test automation rules includes at least one of a predefined number of loops, a predefined number of conditions, a predefined structured design, or at least one verification point, the system according to claim 10.

12. The at least one processor is further Configured to execute instructions to perform correction activities for the analyzed workflow of the test automation based on the correction activity data, the system according to claim 10.

13. To execute the correction activity, the at least one processor is further Predicting one or more defects in the workflow of the test automation based on the determined one or more metrics via the AI model, The system according to claim 12, configured to perform the instruction to modify the workflow of the test automation via the AI model to remove the one or more defects.

14. The system according to claim 10, wherein the workflow of the test automation includes a combination of a set of test cases.

15. The system according to claim 14, wherein the at least one processor is further configured to perform the instruction to extract one or more redundant sequences of at least one test case of the set of test cases into another reusable library for analysis of the workflow of the test automation.

16. The AI model corresponds to a pre-trained AI model including training data, and the training data includes at least one of a standard test automation workflow, an error in the test automation workflow, and a standard framework document. The system according to claim 10.

17. The system according to claim 16, wherein the at least one processor is further configured to perform a set of predefined test automation rules based on the pre-trained AI model for analyzing the workflow of the test automation.

18. A computer program stored on a non-transitory computer-readable medium, the computer program comprising: at least one processor receiving a test automation workflow related to an RPA application; analyzing the test automation workflow based on a set of predefined test automation rules via an AI model related to a workflow analyzer module; determining one or more metrics related to the analyzed test automation; generating modification activity data based on the determined one or more metrics via the AI model.

Images