Graphic user interface heatmap of most used system transaction with test automation details
The decision engine addresses the challenge of analyzing complex SAP system usage data by generating a heatmap that simplifies data analysis and identifies test opportunities, enhancing test coverage and relevance in SAP systems.
Patent Information
- Application Number
- JP2024066637
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-04-25
- Filing Date
- 2024-04-17
- Publication Date
- 2024-11-07
AI Technical Summary
Businesses face challenges in understanding the usage of traditional software packages like SAP systems due to the vast number of transactions, with existing data collection programs generating unnormalized and cumbersome data that are difficult to analyze, requiring specialized knowledge and taking significant time and effort.
A method involving a decision engine for integrated data transformation and visualization, which normalizes and aggregates data using weight-wise transformations to generate a graphical user interface heatmap depicting the most used system transactions with test automation details, enabling easy analysis and identification of test opportunities.
Provides a user-friendly heatmap that reduces analysis time and guides users in understanding testing opportunities, ensuring that automated tests cover actually used transactions, thereby improving test coverage and relevance.
Smart Images

Figure 2024157530000006 
Figure 2024157530000007 
Figure 2024157530000008
Abstract
Description
[Technical field]
[0001] The present disclosure relates generally to automation, and more specifically, to a graphic user interface heat map of most used system transactions with test automation details. [Background technology]
[0002] Enterprises utilize traditional software packages (e.g., SAP systems) for their daily operations. The traditional software packages are comprehensive and a single software instance of the traditional software package can enable thousands of transactions (e.g., SAP transactions, SAP Fiori applications, SAP Webdynpro, SAP WinGUI, SAP WebGUI, etc.). Therefore, considering the huge volume of transactions, it is difficult for enterprises to understand the usage of the traditional software packages (i.e., compiling and processing usage data). Furthermore, when the traditional software package receives an update, the enterprises need to carry out testing for any changes in each single software instance for all test cases.
[0003] Conventional collection programs can be run on a minute-by-minute basis to accumulate usage data for conventional software packages. However, conventional collection programs generally do not extract, analyze, or even manage the accumulated usage data. At their best, conventional collection programs provide gigabytes of data that are not normalized and are tedious and time-consuming to analyze by only the most specialized conventional programs.
[0004] For example, ST03(N) is a SAP standard transaction used for system performance monitoring and system workload analysis. However, using ST03(N) for analysis can be difficult and complicated for several reasons. For example, to use ST03(N), end users need to have specialized technical knowledge and understanding of SAP system architecture, database concepts, and performance metrics. Without specialized technical knowledge and understanding, it is extremely difficult to interpret the information provided by ST03(N). Furthermore, the amount of data generated by ST03(N) can be huge, such as over 10GB of technical data. Analyzing such an amount of data is time consuming. And such a data volume is a cumbersome and confusing burden for end users. Furthermore, the data can be spread across different locations, which can be very difficult to navigate. Also, there is little technical documentation on how to use ST03(N), much less documentation on the many application servers, work processes, transactions, and user layers.
[0005] Therefore, a solution is needed for obtaining usage data and deriving analytics from the usage data. Summary of the Invention
[0006] According to one or more embodiments, a method for transforming and visualizing integrated data is provided. The method is performed by a decision engine implemented as a computer program in a computing environment. The method includes performing data integration to normalize the aggregated data to integrated data by subjecting the aggregated data to a weight-wise transformation to output the integrated data. The method includes performing data transformation and visualization by determining a final rating for the integrated data and generating a user interface on a display including a heat map depicting the integrated data according to the final rating.
[0007] So that the advantages of certain embodiments of the present specification may be readily understood, a more detailed description is provided by reference to certain embodiments that are illustrated in the accompanying drawings. It is understood that these drawings depict only typical embodiments and therefore should not be considered limiting of the scope thereof, but one or more embodiments of the present specification will be described and explained with additional specificity and detail using the accompanying drawings. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is an architectural diagram illustrating an automation system, according to one or more embodiments.
[0009] [Diagram 2] FIG. 1 is an architectural diagram illustrating an RPA system, according to one or more embodiments.
[0010] [Diagram 3] FIG. 1 is an architecture diagram illustrating a deployed RPA system, according to one or more embodiments.
[0011] [Figure 4] FIG. 2 is an architecture diagram illustrating the relationships between designers, activities, and drivers, according to one or more embodiments.
[0012] [Diagram 5] FIG. 1 is an architectural diagram illustrating a computing system in accordance with one or more embodiments.
[0013] [Figure 6] FIG. 1 illustrates an example of a neural network trained to recognize graphical elements in an image, according to one or more embodiments.
[0014] [Figure 7] FIG. 2 illustrates an example of a neuron, according to one or more embodiments.
[0015] [Figure 8] 1 is a flowchart illustrating a process for training an AI / ML model(s) in accordance with one or more embodiments.
[0016] [Figure 9] 1 is a flowchart illustrating a process according to one or more embodiments.
[0017] [Figure 10] FIG. 2 illustrates an interface according to one or more embodiments.
[0018] [Figure 11] FIG. 2 illustrates an interface according to one or more embodiments.
[0019] [Figure 12] FIG. 2 illustrates an interface according to one or more embodiments.
[0020] [Figure 13] FIG. 2 illustrates an interface according to one or more embodiments.
[0021] [Figure 14] FIG. 2 illustrates an interface according to one or more embodiments.
[0022] [Figure 15] FIG. 2 illustrates an interface according to one or more embodiments.
[0023] [Figure 16] FIG. 2 illustrates an interface according to one or more embodiments.
[0024] Unless otherwise indicated, like reference numerals denote corresponding features consistently throughout the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0025] SUMMARY OF THE DISCLOSURE Embodiments herein relate to automation. More particularly, embodiments herein relate to automatically generating a graphic user interface heat map of most used system transactions with test automation details. The graphic user interface heat map can be automatically generated by one or more computing systems performing a decision engine, which further performs robotic process automation (RPA). According to one or more embodiments, the decision engine generally performs operations or methods of obtaining usage data of a software system and deriving analytics from the usage data. The decision engine operations or methods include utilizing the derived analytics to generate a graphic user interface heat map as a synthesis, aggregation, and visualization of the software system usage. The decision engine can enable the graphic user interface heat map to depict the most used transactions in the software system with test automation details. For example, the decision engine can provide instant discovery of test automation opportunities for the software system via the graphic user interface heat map. According to one or more embodiments, the decision engine can also aggregate the usage data extracted by the collection program, calculate and / or convert the usage data into respective weights, and visualize the usage data as a graphic user interface heat map. Additionally, the decision engine can enable a graphic user interface heat map to depict an indication of test relevance and test prioritization.
[0026] FIG. 1 is an architecture diagram illustrating a hyperautomation system 100, according to one or more embodiments. As used herein, "hyperautomation" refers to an automation system that brings together process automation components, integration tools, and technologies that enhance the ability to automate work. For example, RPA may be used at the core of the hyperautomation system in some embodiments, and in certain embodiments, the automation capabilities may be extended with artificial intelligence and / or machine learning (AI / ML), process mining, analytics, and / or other advanced tools. As the hyperautomation system learns processes, trains AI / ML models, and uses analytics, for example, more and more knowledge work can be automated, and computing systems within an organization, for example, both those used by individuals and those that run autonomously, can all be engaged to become participants in the hyperautomation process. In some embodiments, the hyperautomation system allows users and organizations to discover, understand, and scale automation efficiently and effectively.
[0027] The hyperautomation system 100 includes user computing systems, such as a desktop computer 102, a tablet 104, and a smartphone 106. However, any desired computing system may be used, including, but not limited to, a smart watch, a laptop computer, a server, an Internet of Things (IoT) device, and the like, without departing from the scope of one or more embodiments herein. Also, while three user computing systems are shown in FIG. 1, any suitable number of computing systems may be used without departing from the scope of one or more embodiments herein. For example, in some embodiments, tens, hundreds, thousands, or millions of computing systems may be used. The user computing systems may be actively used by a user or may run automatically, without much or any user input.
[0028] Each computing system 102, 104, 106 has a respective automated process 110, 112, 114(s) running thereon. The automated process 102, 104, 106(s) may include, but are not limited to, an RPA robot, a part of an operating system, a downloadable application(s) for the respective computing system, any other suitable software and / or hardware, or any combination thereof, without departing from the scope of one or more embodiments herein. In some embodiments, one or more of the process 110, 112, 114(s) may be a listener. The listener may be an RPA robot, a part of an operating system, a downloadable application for the respective computing system, or any other software and / or hardware, without departing from the scope of one or more embodiments herein. Indeed, in some embodiments, the logic of the listener(s) is implemented partially or completely via physical hardware.
[0029] The listeners monitor and record data regarding user interactions with their respective computing systems and / or the operation of unattended computing systems, and transmit the data over a network (e.g., a local area network (LAN), a mobile communications network, a satellite communications network, the Internet, any combination thereof, etc.) to the core hyperautomation system 120. The data may include, but is not limited to, which button was clicked, where the mouse was moved, text entered into a field, one window was minimized and another opened, the application associated with the window, etc. In certain embodiments, the data from the listeners may be transmitted periodically as part of a heartbeat message. In some embodiments, the data may be transmitted to the core hyperautomation system 120 after a predetermined amount of data has been collected, after a predetermined period of time has passed, or both. One or more servers, such as server 130, receive and store the data from the listeners in a database, such as database 140.
[0030] An automated process can execute logic developed during design time in a workflow. In the case of RPA, a workflow may include a set of steps, defined herein as "activities," that are executed in sequence or some other logic flow. Each activity may include an action such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.
[0031] The RPA long-running workflow in some embodiments is a master project that supports service orchestration, human intervention, and long-running transactions in an unattended environment. U.S. Patent No. 10,860,905 is incorporated by reference for all it contains. Human intervention occurs when a particular process requires human input to handle exceptions, approvals, or validation before proceeding to the next step of the activity. In this case, process execution is suspended, freeing up the RPA robot until the human task is completed.
[0032] Long-running workflows may support workflow fragmentation through persistence activities, and may combine with invocation process and non-user interaction activities to orchestrate human tasks with RPA robot tasks. In some embodiments, multiple or multiple computing systems may participate in the execution of the logic of a long-running workflow. Long-running workflows may execute in sessions to facilitate rapid execution. In some embodiments, long-running workflows may execute application programming interface (API) calls and orchestrate background processes that may include activities that execute in the long-running workflow session. In some embodiments, these activities may be invoked by an invocation process activity. Processes with user interaction activities that execute in a user session may be invoked by starting a job from a conductor activity (conductors are described in more detail later in this specification). In some embodiments, users may interact through a task that requires a form to be completed in the conductor. An activity may be included that causes the RPA robot to wait for the form task to complete and then resume the long-running workflow.
[0033] One or more of the automated processes 110, 112, 114(s) are in communication with the core hyper-automation system 120. In some embodiments, the core hyper-automation system 120 may execute a conductor application on one or more servers, such as server 130. While one server 130 is shown for illustrative purposes, multiple or numerous servers, either in close proximity to one another or in a distributed architecture, may be employed without departing from the scope of one or more embodiments herein. For example, one or more servers may be provided for conductor functions, AI / ML model serving, certification, governance, and / or any other suitable functions without departing from the scope of one or more embodiments herein. In some embodiments, the core hyper-automation system 120 may incorporate or be part of a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, or the like. In certain embodiments, the core hyper-automation system 120 may host multiple software-based servers on one or more computing systems, such as server 130. In some embodiments, one or more servers of the core hyper-automation system 120, such as server 130, may be implemented via one or more virtual machines (VMs).
[0034] In some embodiments, one or more of the automated processes 110, 112, 114(s) may invoke one or more AI / ML models 132 deployed on or accessible by the core hyper-automation system 120. The AI / ML models 132 may be trained for any suitable purpose, as described in more detail herein, without departing from the scope of one or more embodiments herein. In some embodiments, two or more of the AI / ML models 132 may be chained (e.g., in series, in parallel, or a combination thereof) such that they collectively provide coordinated output(s). The AI / ML models 132 may perform or assist in computer vision (CV), optical character recognition (OCR), document processing and / or understanding, semantic learning and / or analysis, analytical prediction, process discovery, task mining, testing, automated RPA workflow generation, sequence extraction, clustering detection, speech-to-text translation, any combination thereof, and the like. However, any desired number and / or type(s) of AI / ML models may be used without departing from the scope of one or more embodiments herein. By using multiple AI / ML models, the system can develop a complete picture of what is going on, for example, on a given computing system. For example, one AI / ML model can perform OCR, another can detect buttons, another can compare sequences, etc. Patterns may be determined by the AI / ML models individually or collectively by multiple AI / ML models. In certain embodiments, one or more AI / ML models are deployed locally on at least one of the computing systems 102, 104, 106.
[0035] In some embodiments, multiple AI / ML models 132 can be used. Each AI / ML model 132 is an algorithm (or model) that runs on data, and the AI / ML model itself can be, for example, a deep learning neural network (DLNN) of trained artificial "neurons" that are trained on training data. In some embodiments, the AI / ML model 132 can have multiple layers that perform various functions, such as statistical modeling (e.g., Hidden Markov Models (HMMs)), and can utilize deep learning techniques (e.g., long short-term memory (LSTM) deep learning, encoding of prior hidden states, etc.) to perform desired functions.
[0036] In some embodiments, the hyper-automation system 100 can provide four main groups of functionality: (1) discovery, (2) build automation, (3) management, and (4) engagement. In some embodiments, the automation (e.g., executed on a user computing system, server, etc.) may be executed by a software robot, such as an RPA robot. For example, attended robots, unattended robots, and / or test robots can be used. Attended robots work with a user (e.g., via UiPath Assistant™) to assist the user with a task. Unattended robots work independently of the user and can run in the background without the user's knowledge. Test robots are unattended robots that run test cases against an application or RPA workflow. In some embodiments, test robots can run in parallel on multiple computing systems.
[0037] A discovery function can discover various opportunities for business process automation and provide automated recommendations. Such functions may be implemented by one or more servers, such as server 130. The discovery function may include providing an automation hub, process mining, task mining, and / or task capture, in some embodiments. An automation hub (e.g., UiPath Automation Hub™) can provide a mechanism to manage automation rollouts with visibility and control. Automation ideas can be crowdsourced from employees, for example, via submission forms. Calculations of feasibility and return on investment (ROI) for automating these ideas can be provided, documentation can be collected for future automations, and collaboration can be provided to move automations from discovery to building faster.
[0038] Process mining (e.g., via UiPath Automation Cloud™ and / or UiPath AI Center™) refers to the process of collecting and analyzing data from applications (e.g., enterprise resource planning (ERP) applications, customer relationship management (CRM) applications, email applications, call center applications, etc.) to indicate what end-to-end processes exist within an organization, how they can be effectively automated, and what the impact of automation would be. This data can be collected, for example, by listeners from user computing systems 102, 104, 106 and processed by a server, such as server 130. In some embodiments, one or more AI / ML models 132 can be used for this purpose. This information may be exported to an automation hub to speed implementation and avoid manual information transfer. The goal of process mining can be to increase business value by automating processes within an organization. Some examples of process mining goals include, but are not limited to, increasing profits, improving customer satisfaction, complying with regulations and / or contracts, increasing employee efficiency, etc.
[0039] Task mining (e.g., via UiPath Automation Cloud™ and / or UiPath AI Center™) identifies and aggregates workflows (e.g., employee workflows) and then applies AI to expose patterns and variations in routine tasks and score such tasks for ease of automation and potential savings (e.g., time and / or cost savings). One or more AI / ML models 132 may be employed to discover repeated task patterns in the data. Repetitive tasks that are ripe for automation can then be identified. This information, in some embodiments, may be initially provided by a listener and analyzed on a server of the core hyperautomation system 120, such as server 130. Findings from task mining (e.g., Extensible Application Markup Language (XAML) process data) can be exported to process documents or to designer applications, such as UiPath Studio™, to more quickly create and deploy automations.
[0040] Task mining in some embodiments may include taking screenshots involving user actions (e.g., mouse click locations, keyboard input, application windows and graphical elements with which the user was interacting, timestamps of the interactions, etc.), collecting statistical data (e.g., performance time, number of actions, text input, etc.), editing and annotating screenshots, specifying the types of actions to be recorded, etc.
[0041] Task capture (e.g., via UiPath Automation Cloud™ and / or UiPath AI Center™) automatically documents attended processes as users work on them or provides a framework for unattended processes. Such documentation may include the desired tasks to be automated in the form of process definition documents (PDDs), skeleton workflows, capturing actions for each part of the process, recording user actions, and automatically generating comprehensive workflow diagrams with details about each step, Microsoft Word™ documents, XAML files, etc. Configurable workflows can, in some embodiments, be exported directly to designer applications such as UiPath Studio™. Task capture can simplify the requirements gathering process for both subject matter experts describing the process and Center of Excellence (CoE) members providing production-grade automation.
[0042] Building automations can be accomplished through a designer application (e.g., UiPath Studio™, UiPath StudioX™, or UiPath Web™). For example, an RPA developer at an RPA development facility 150 can use an RPA designer application 154 at a computing system 152 to build and test automations for various applications and environments, such as web, mobile, SAP®, and virtualized desktop. API integration may be provided for various applications, technologies, and platforms. Predefined activities, drag-and-drop modeling, and workflow recorders may facilitate automation with minimal coding. Document understanding capabilities can be provided through drag-and-drop AI skills for data extraction and interpretation that invoke one or more AI / ML models 132. Such automation can handle virtually any document type and format, including tables, checkboxes, signatures, and handwriting. When data is validated or exceptions are handled, this information can be used to retrain the respective AI / ML models to improve their accuracy over time.
[0043] The integration services allow developers to seamlessly combine, for example, user interface (UI) automation and API automation. Automations that require APIs or span both API and non-API applications and systems may be built. A repository (e.g., UiPath Object Repository™) or marketplace (e.g., UiPath Marketplace™) for pre-built RPA and AI templates and solutions may be provided to enable developers to more quickly automate a wide variety of processes. Thus, when building automations, the hyper-automation system 100 may provide user interfaces, development environments, API integrations, pre-built and / or custom-built AI / ML models, development templates, integrated development environments (IDEs), and advanced AI capabilities. The hyper-automation system 100, in some embodiments, may enable development, deployment, management, configuration, monitoring, debugging, and maintenance of RPA robots, thereby providing automation of the hyper-automation system 100.
[0044] In some embodiments, components of the hyper-automation system 100, such as the designer application(s) and / or external rules engine, provide support for managing and enforcing governance policies to control various functions provided by the hyper-automation system 100. Governance is the ability for an organization to put policies in place to prevent users from developing automations (e.g., RPA robots) that can perform actions that could harm the organization, such as violating the EU General Data Protection Regulation (GDPR), the US Health Insurance Portability and Accountability Act (HIPAA), third-party application terms of use, etc. Because developers could otherwise create automations that violate privacy laws, terms of service, etc. while executing those automations, some embodiments implement access control and governance restrictions at the robot and / or robot design application level. This can provide an additional level of security and compliance to the automation process development pipeline in some embodiments by preventing developers from relying on unapproved software libraries that pose security risks or that may violate or operate in violation of policies, regulations, privacy laws, and / or privacy policies. See U.S. Non-provisional Patent Application No. 16 / 924,499, which is incorporated by reference for all it contains.
[0045] The management functions can provide management, deployment, and optimization of automation across an organization. The management functions can include orchestration, test management, AI capabilities, and / or insights, in some embodiments. The management functions of the hyper-automation system 100 can also serve as an integration point with third party solutions and applications for automation applications and / or RPA robots. The management functions of the hyper-automation system 100 can include, but are not limited to, facilitating provisioning, deployment, configuration, queuing, monitoring, logging, and interconnection of RPA robots, among others.
[0046] Conductor applications such as UiPath Orchestrator (which may be offered in some embodiments as part of UiPath Automation Cloud™ or on-premise, in a VM, in a private or public cloud, in a Linux™ VM, or as a cloud-native single container suite via UiPath Automation Suite™) provide orchestration capabilities for deploying, monitoring, optimizing, scaling, and securing RPA robot deployments. Test suites (e.g., UiPath Test Suite™) can provide test management to monitor the quality of deployed automation. Test suites can facilitate test planning and execution, requirements fulfillment, and defect traceability. Test suites may include comprehensive test reporting.
[0047] Analytics software (e.g., UiPath Insights™) can track, measure, and manage the performance of deployed automation. Analytics software can align automation operations with specific key performance indicators (KPIs) and strategic results of the organization. Analytics software can present results in the form of dashboards for better understanding by human users.
[0048] Data services (e.g., UiPath Data Service™) may bring data to a single, scalable, secure location, for example, stored in database 140, with a drag-and-drop storage interface. Some embodiments may provide low-code or no-code data modeling and storage to automation, while ensuring seamless access, enterprise-grade security, and scalability of data. AI capabilities may be provided by an AI center (e.g., UiPath AI Center™), which facilitates the incorporation of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options can make such capabilities accessible even to non-data scientists. Deployed automation (e.g., RPA robots) may invoke AI / ML models from the AI center, such as AI / ML models 132. The performance of the AI / ML models may be monitored, trained, and improved using human-verified data, such as provided by data review center 160. Human reviewers can provide labeled data to the core hyper-automation system 120 via review applications 152 on computing systems 154. For example, a human reviewer may verify that the predictions made by the AI / ML model 132 are accurate or may otherwise provide corrections. This dynamic input may then be saved as training data for retraining the AI / ML model 132 and stored in a database, such as database 140. The AI center can then schedule and complete a training job to train a new version of the AI / ML model using the training data. Both positive and negative examples may be stored and used to retrain the AI / ML model 132.
[0049] Engagement capabilities engage humans and automation as one team for seamless collaboration in a desired process. In some embodiments, low-code applications can be built (e.g., via UiPath Apps™) to connect browser tabs and legacy software, even when APIs are lacking. Applications can be quickly created using a web browser, for example, via a rich library of drag-and-drop controls. Applications can be connected to a single automation or multiple automations.
[0050] Action Center (e.g., UiPath Action Center™) provides an easy and efficient mechanism to hand off processes from automation to humans and vice versa. Humans may provide approval or escalation or make exceptions. Automation can then execute the automated functions of a given workflow.
[0051] A local assistant may be provided as a launch pad for users to launch automations (e.g., UiPath Assistant™). This functionality may be provided, for example, in a tray provided by the operating system and may enable users to interact with RPA robots and RPA robot-powered applications on the computing system. The interface may list automations approved for a given user and enable the user to execute them. These may include out-of-the-box automations from an automation marketplace, an internal automation store in an automation hub, etc. Once automations are executed, they may run as local instances in parallel with other processes on the computing system so that the user can use the computing system while the automation is performing its actions. In certain embodiments, the assistant is integrated with a task capture functionality to enable users to document soon-to-be-automated processes from the assistant's launch pad.
[0052] Chatbots (e.g., UiPath Chatbots™), social messaging applications, and / or voice commands can enable users to execute automations. This can simplify access to information, tools, and resources that users need to interact with customers or perform other activities. Conversations between people can be easily automated, just like any other process. Triggered RPA robots thus launched can potentially use plain language commands to perform operations such as checking order status, posting data in CRM, etc.
[0053] In some embodiments, end-to-end measurement and governance of automation programs at any scale can be provided by the hyper-automation system 100. As such, analytics (e.g., via UiPath Insights™) may be used to understand the performance of automations. Data modeling and analytics using any combination of available business metrics and operational insights can be used for various automation processes. Custom-designed and pre-built dashboards allow for visualizing data across desired metrics, discovering new analytical insights, tracking performance indicators, discovering ROI for automations, monitoring telemetry running on user computing systems, detecting errors and anomalies, and debugging automations. To manage automations across the entire automation lifecycle, an automation management console (e.g., UiPath Automation Ops™) may be provided. Organizations can govern how automations are built, what users can do with them, and which automations users can access.
[0054] The hyper-automation system 100, in some embodiments, provides an iterative platform: processes can be discovered, automations can be built, tested, and deployed, performance can be measured, automation usage can be easily provided to users, feedback can be obtained, AI / ML models can be trained and retrained, and the process can repeat itself, facilitating a more robust and effective set of automations.
[0055] FIG. 2 is an architecture diagram illustrating an RPA system 200, according to one or more embodiments. In some embodiments, the RPA system 200 is part of the hyper-automation system 100 of FIG. 1. The RPA system 200 includes a designer 210 that enables developers to design and implement workflows. The designer 210 can provide solutions for application integration, as well as automating third-party applications, management information technology (IT) tasks, and business IT processes. The designer 210 can facilitate the development of automation projects, which are graphical representations of business processes. Simply put, the designer 210 facilitates the development and deployment (represented by arrow 211) of workflows and robots. In some embodiments, the designer 210 can be an application running on a user's desktop, an application running remotely in a VM, a web application, or the like.
[0056] An automation project enables rule-based process automation by giving developers control over the order of execution and relationships between a custom set of steps developed in a workflow, defined herein as "activities" as described above. One commercial example of an embodiment of the designer 210 is UiPath Studio™. Each activity may include an action such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.
[0057] Some types of workflows may include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suitable for linear processes that allow flow from one activity to another without disrupting the workflow. Flowcharts may be particularly suitable for more complex business logic, allowing for the integration of decisions and the connection of activities in more diverse ways through multiple branching logical operators. FSMs may be particularly suitable for large workflows. FSMs may use a finite number of states in their execution that are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suitable for determining the behavior of a workflow when an execution error is encountered and for debugging the process.
[0058] Once a workflow is developed in Designer 210, the execution of the business process is orchestrated by Conductor 220, which coordinates one or more Robots 230 that execute the workflow developed in Designer 210. One commercial example of an embodiment of Conductor 220 is UiPath Orchestrator™. Conductor 220 facilitates the creation, monitoring, and management of deployment of resources in an environment. Conductor 220 can act as an integration point with third party solutions and applications. Accordingly, in some embodiments, Conductor 220 may be part of the core hyper-automation system 120 of FIG. 1.
[0059] The conductor 220 may manage all the robots 230, or may connect and execute the robots 230 from a centralized point (as represented by arrow 231). Types of robots 230 that may be managed include, but are not limited to, attended robots 232, unattended robots 234, development robots (similar to unattended robots 234, but used for development and testing purposes), and non-production robots (similar to attended robots 232, but used for development and testing purposes). Attended robots 232 are triggered by user events and operate alongside humans on the same computing system. Attended robots 232 can be used with the conductor 220 for a centralized process deployment and logging medium. Attended robots 232 may assist human users in accomplishing various tasks and may be triggered by user events. In some embodiments, processes cannot be initiated from the conductor 220 for this type of robot and / or cannot be run under a locked screen. In certain embodiments, the attended robot 232 can only be launched from the robot tray or from a command prompt. In some embodiments, the attended robot 232 should run under human supervision.
[0060] Unattended robots 234 can run unattended in a virtual environment and automate many processes. Unattended robots 234 can be responsible for providing remote execution, monitoring, scheduling, and work queue support. In some embodiments, debugging of all robot types can be performed in designer 210. Both attended and unattended robots can automate a variety of systems and applications (represented by dashed box 290), including, but not limited to, mainframes, web applications, VMs, enterprise applications (e.g., those manufactured by SAP®, SalesForce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).
[0061] The conductor 220 may have various functions (represented by arrows 232) including, but not limited to, provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between the robots 230 and the conductor 220 (e.g., web applications). Deployment may include ensuring correct delivery of package versions to the robots 230 assigned for execution. Configuration may include maintaining and delivering robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs in a database (e.g., a Structured Query Language (SQL) database or a NoSQL database) and / or another storage mechanism (e.g., ElasticSearch®, which provides the ability to store and quickly query large data sets). The conductor 220 can provide interconnectivity by acting as a centralized point of communication for third-party solutions and / or applications.
[0062] Robot 230 is an execution agent that implements the workflow built in designer 210. One commercial example of some embodiments of robot(s) 230 is UiPath Robots™. In some embodiments, robot 230 installs Microsoft Windows Service Control Manager (SCM) management services by default. As a result, such robot 230 can open an interactive Windows session under the local system account and have Windows service rights.
[0063] In some embodiments, a robot 230 can be installed in user mode. For such a robot 230, this means that a given robot 230 has the same rights as the user who installed it. This feature may also be available for high density (HD) robots to ensure maximum utilization of each machine. In some embodiments, any type of robot 230 can be configured in an HD environment.
[0064] The robot 230 in some embodiments is divided into multiple components, each dedicated to a particular automation task. The robot components in some embodiments include, but are not limited to, an SCM managed robot service, a user mode robot service, an executor, an agent, and a command line. The SCM managed robot service manages and monitors the Windows session and acts as a proxy between the conductor 220 and the execution host (i.e., the computing system on which the robot 230 executes). These services are trusted with and manage the credentials of the robot 230. The console application is launched by the SCM under local system.
[0065] The user-mode robot service in some embodiments manages and monitors the Windows session and acts as a proxy between the conductor 220 and the execution host. The user-mode robot service can be trusted with and manage the credentials of the robot 230. If the SCM management robot service is not installed, the Windows application can be launched automatically.
[0066] An Executor can run a given job under a Windows session (i.e., an Executor can execute a workflow). An Executor may be aware of per-monitor dots per inch (DPI) settings. An Agent may be a Windows Presentation Foundation (WPF) application that displays available jobs in a system tray window. An Agent may be a client of a Service. An Agent can request to start or stop jobs and change settings. A Command Line is a client of a Service. A Command Line is a console application that can request to start a job and wait for its output.
[0067] Splitting up the components of the robot 230 as described above helps developers, support users, and computing systems more easily execute, identify, and track what each component is accomplishing. In this way, special behaviors can be configured per component, such as setting different firewall rules for executors and services. In some embodiments, the executor can always be aware of the DPI setting per monitor. As a result, workflows can be executed at any DPI regardless of the configuration of the computing system on which the workflow was created. In some embodiments, projects from the designer 210 can be independent of the browser zoom level. In some embodiments, DPI can be disabled for applications that are not DPI aware or are intentionally marked as not aware.
[0068] The RPA system 200 in this embodiment is part of a hyper-automation system. A developer can use the designer 210 to build and test RPA robots that utilize AI / ML models deployed to the core hyper-automation system 240 (e.g., as part of its AI center). Such RPA robots can send inputs for execution of the AI / ML model(s) and receive outputs therefrom via the core hyper-automation system 240.
[0069] As mentioned above, one or more of the robots 230 may be listeners. These listeners may provide information to the core hyper-automation system 240 about what users are doing when they use the computing system. This information may then be used by the core hyper-automation system for process mining, task mining, task capture, and the like.
[0070] An assistant / chatbot 250 may be provided on a user computing system to allow the user to launch an RPA local robot. The assistant may be located, for example, in the system tray. The chatbot may have a user interface so that the user can see the text in the chatbot. Alternatively, the chatbot may not have a user interface and may run in the background, listening to what the user is saying using the computing system's microphone.
[0071] In some embodiments, data labeling may be performed by a user of the computing system on which the robot is executing, or on another computing system to which the robot provides information. For example, if the robot invokes an AI / ML model to perform CV on an image for a VM user, but the AI / ML model does not correctly identify a button on the screen, the user might draw a rectangle around the misidentified or unidentified component and provide the correct identification in text. This information may be provided to the core hyperautomation system 240 and then later used to train a new version of the AI / ML model.
[0072] Figure 3 is an architecture diagram illustrating a deployed RPA system 300, according to one or more embodiments. In some embodiments, the RPA system 300 may be part of the RPA system 200 of Figure 2 and / or the hyper-automation system 100 of Figure 1. The deployed RPA system 300 may be a cloud-based system, an on-premise system, a desktop-based system, etc. that provides an enterprise-level, user-level, or device-level automation solution for the automation of different computing processes.
[0073] It should be noted that the client side 301, the server side 302, or both, may include any desired number of computing systems without departing from the scope of one or more embodiments herein. On the client side 301, the robot application 310 includes an executor 312, an agent 314, and a designer 316. However, in some embodiments, the designer 316 may not be running on the same computing system as the executor 312 and the agent 314. The executor 312 is running a process. As shown in FIG. 3, several business projects may be running simultaneously. The agent 314 (e.g., a Windows service) is a single connection point for all the executors 312 in this embodiment. All messages in this embodiment are logged into the conductor 340, which further processes them via a database server 355, an AI / ML server 360, an indexer server 370, or any combination thereof. As described above with respect to FIG. 2, the executor 312 may be a robotic component.
[0074] In some embodiments, a robot represents an association between a machine name and a username. A robot can manage multiple executors simultaneously. In computing systems that support multiple interactive sessions running simultaneously (e.g., Windows Server 2012), multiple robots may run simultaneously, each running in a separate Windows session using a unique username. This is referred to above as an HD Robot.
[0075] The agent 314 is also responsible for transmitting the robot's status (e.g., periodically sending "heartbeat" messages to indicate that the robot is still functioning) and downloading necessary versions of packages to be fulfilled. In some embodiments, communication between the agent 314 and the conductor 340 is always initiated by the agent 314. In a notification scenario, the agent 314 may open a WebSocket channel that is later used by the conductor 330 to send commands (e.g., start, stop, etc.) to the robot.
[0076] The listener 330 monitors and records data regarding user interactions with the attended computing system and / or the operation of the unattended computing system on which the listener 330 resides. The listener 330 may be an RPA robot, part of an operating system, a downloadable application for the respective computing system, or any other software and / or hardware without departing from the scope of one or more embodiments herein. Indeed, in some embodiments, the logic of the listener is implemented partially or fully via physical hardware.
[0077] The server side 302 includes a presentation layer 333, a service layer 334, and a persistence layer 336, as well as a conductor 340. The presentation layer 333 may include a web application 342, an Open Data Protocol (OData) Representational State Transfer (REST) Application Programming Interface (API) endpoint 344, and a notification monitor 346. The service layer 334 may include an API implementation / business logic 348. The persistence layer 336 may include a database server 355, an AI / ML server 360, and an indexer server 370. For example, the conductor 340 includes a web application 342, an OData REST API endpoint 344, a notification monitor 346, and an API implementation / business logic 348. In some embodiments, most actions that a user performs in the conductor 340 interface (e.g., via the browser 320) are performed by calling various APIs. Such actions may include, but are not limited to, starting a job on a robot, adding / removing data in a queue, scheduling a job to run unattended, etc., without departing from the scope of one or more embodiments herein. The web application 342 may be a visual layer of a server platform. In this embodiment, the web application 342 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of one or more embodiments herein. A user interacts with web pages from the web application 342, in this embodiment, via a browser 320, to perform various actions to control the conductor 340. For example, a user may create robot groups, assign packages to robots, analyze logs per robot and / or per process, start and stop robots, etc.
[0078] In addition to the web application 342, the conductor 340 also includes a services layer 334 that exposes an OData REST API endpoint 344. However, other endpoints may be included without departing from the scope of one or more embodiments herein. The REST API is consumed by both the web application 342 and an agent 314, which in this embodiment is a supervisor of one or more robots on a client computer.
[0079] The REST API in this embodiment includes configuration, logging, monitoring, and queuing functionality (represented by at least arrow 349). In some embodiments, the configuration endpoint may be used to define and configure application users, permissions, robots, assets, releases, and environments. The logging REST endpoint may be used to log various information, such as errors, explicit messages sent by the robot, and other environment specific information. The deployment REST endpoint may be used by the robot to query the package version that should be committed when a start job command is used in the conductor 340. The queuing REST endpoint may be responsible for queue and queue item management, such as adding data to the queue, getting transactions from the queue, setting the status of transactions, etc.
[0080] Monitoring REST endpoints may monitor web application 342 and agents 314. Notification monitoring API 346 may be a REST endpoint used to register agents 314, deliver configuration settings to agents 314, and send and receive notifications from the server and agents 314. In some embodiments, notification monitoring API 346 may also use WebSocket communications. As shown in FIG. 3, one or more activities / actions described herein are represented by arrows 350 and 351.
[0081] The APIs in the service layer 334 can be accessed, in some embodiments, through configuration of appropriate API access paths based on, for example, whether the conductor 340 and the overall hyperautomation system have an on-premise or cloud-based deployment type. The APIs for the conductor 340 can provide custom ways to query statistics about various entities registered with the conductor 340. In some embodiments, each logic resource can be an OData entity. In such entities, components such as robots, processes, queues, etc. can have properties, relationships, and operations. In some embodiments, the conductor 340 APIs can be consumed by web applications 342 and / or agents 314 in two ways: by obtaining API access information from the conductor 340 or by registering an external application to use the OAuth flow.
[0082] In this embodiment, the persistence layer 336 includes three servers: a database server 355 (e.g., a SQL server), an AI / ML server 360 (e.g., a server that provides AI / ML models to provide services such as AI center functions), and an indexer server 370. The database server 355 in this embodiment stores configurations of robots, robot groups, associated processes, users, roles, schedules, etc. This information is managed through a web application 342 in some embodiments. The database server 355 can manage queues and queue items. In some embodiments, the database server 355 may store messages logged by the robots (in addition to or instead of the indexer server 370). The database server 355 may also store process mining, task mining, and / or task capture related data received, for example, from a listener 330 installed on the client side 301. In some embodiments, no arrows are shown between the listener 330 and the database 355, but it should be understood that the listener 330 can communicate with the database 355 and vice versa. This data may be stored in the form of PDDs, images, XAML files, etc. Listener 330 may be configured to intercept user actions, processes, tasks, and performance metrics on each computing system on which listener 330 resides. For example, listener 330 may record user actions (e.g., clicks, typed characters, location, application, active element, time, etc.) on its respective computing system and then convert these into a format suitable for being provided to and stored in database server 355.
[0083] AI / ML Server 360 makes it easy to incorporate AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options make such capabilities accessible even to non-data scientists. Deployed automation (e.g., RPA robots) can invoke AI / ML models from AI / ML Server 360. The performance of AI / ML models can be monitored, trained, and improved using human-validated data. AI / ML Server 360 can schedule and execute training jobs to train new versions of AI / ML models.
[0084] AI / ML server 360 may store data related to AI / ML models and ML packages for configuring various ML skills for users at development time. ML skills, as used herein, are pre-built and trained ML models for processes that may be used, for example, by automation. AI / ML server 460 may also store data related to document understanding techniques and frameworks, algorithms, and software packages for various AI / ML functions, including, but not limited to, intent analysis, natural language processing (NLP), speech analysis, different types of AI / ML models, etc.
[0085] Optionally in some embodiments, the indexer server 370 stores and indexes information logged by the robots. In certain embodiments, the indexer server 370 may be disabled through a configuration setting. In some embodiments, the indexer server 370 uses Elastic Search®, a full-text search engine from an open source project. Messages logged by the robots (e.g., using activities such as log messages or line writes) may be sent via logging REST endpoint(s) to the indexer server 370, where they are indexed for future use.
[0086] FIG. 4 is an architecture diagram illustrating the relationships between designer 410, activities 420, 430, 440, 450, drivers 460, APIs 470, and AI / ML models 480, according to one or more embodiments. As described herein, a developer uses designer 410 to develop a workflow to be performed by a robot. In some embodiments, various types of activities can be displayed to the developer. Designer 410 may be local to the user's computing system or remote (e.g., accessed via a VM or a local web browser that interacts with a remote web server). A workflow may include user-defined activities 420, API-driven activities 430, AI / ML activities 440, and / or UI automation activities 450. By way of example (as shown by the dotted lines), user-defined activities 420 and API-driven activities 440 interact with applications via their APIs. The user-defined activities 420 and / or AI / ML activities 440 may then invoke one or more AI / ML models 480, which in some embodiments may be located locally to and / or remotely from the computing system on which the robot is operating.
[0087] Some embodiments can identify non-text visual components in an image, referred to herein as CV. CV may be performed at least in part by AI / ML model(s) 480. Some CV activities related to such components may include, but are not limited to, extraction of text from segmented label data using OCR, fuzzy text matching, cropping segmented label data using ML, comparing extracted text in label data to ground truth data, etc. In some embodiments, there may be hundreds or thousands of activities that may be implemented in user-defined activities 420. However, any number and / or type of activities may be used without departing from the scope of one or more embodiments herein.
[0088] UI automation activities 450 are a subset of special low-level activities that are written in low-level code and facilitate interaction with a screen. UI automation activities 450 facilitate interaction via drivers 460 that enable the robot to interact with desired software. For example, drivers 460 may include operating system (OS) drivers 462, browser drivers 464, VM drivers 466, enterprise application drivers 468, etc. In some embodiments, one or more of AI / ML models 480 may be used by automation activities 450 to perform interactions with the computing system. In certain embodiments, AI / ML models 480 may augment drivers 460 or replace them entirely. Indeed, in certain embodiments, drivers 460 are not included.
[0089] Driver 460 may interact with the OS at a low level, such as looking for hooks and monitoring keys via OS driver 462. Driver 460 may facilitate integration with Chrome®, IE®, Citrix®, SAP®, etc. For example, a "click" activity plays the same role in these different applications via driver 460.
[0090] FIG. 5 is an architecture diagram illustrating a computing system 500 configured to provide a decision engine for automatically generating a graphic user interface heat map of most used system transactions with test automation details, according to one or more embodiments. In some embodiments, the computing system 500 may be one or more of the computing systems illustrated and / or described herein. In certain embodiments, the computing system 500 may be part of a hyper-automation system such as those illustrated in FIGS. 1 and 2. The computing system 500 includes a bus 505 or other communication mechanism for communicating information and a processor(s) 510 coupled to the bus 505 for processing information. The processor(s) 510 may be any type of general or application-specific processor, including a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor(s) 510 may also have multiple processing cores, at least some of which may be configured to perform specific functions. In some embodiments, multiple parallel processing may be used. In certain embodiments, at least one processor(s) 510 may be a neuromorphic circuit that includes processing elements that mimic biological neurons. In some embodiments, a neuromorphic circuit may not require typical components of a von Neumann computing architecture.
[0091] The computing system 500 further includes a memory 515 for storing information and instructions executed by the processor(s) 510. The memory 515 may be comprised of any combination of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage such as magnetic or optical disks, or any other type of non-transitory computer-readable medium, or any combination thereof. The non-transitory computer-readable medium may be any available medium that is accessible by the processor(s) 510 and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both.
[0092] Additionally, the computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communication network via wireless and / or wired connections. In some embodiments, the communication device 520 may be any of a variety of wireless technologies including Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiplexing (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile Communications (GSM) communications, General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), cdma2000, Wideband-CDMA (W-CDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), High Speed Packet Access (HSPA), Long Term Evolution (LTE), LTE Advanced (LTE-A), 802.11x, Wi-Fi, Zigbee, Ultra Wideband (UWB), 802.16x, 802.15, Home It may be configured to use Node-B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near Field Communication (NFC), Fifth Generation (5G) New Radio (NR), any combination thereof, and / or any other currently existing or future implemented communication standards and / or protocols without departing from the scope of one or more embodiments herein. In some embodiments, the communication device 520 may include one or more antennas that are a single antenna, an array antenna, a panel antenna, a phased antenna, a switched antenna, a beamforming antenna, a beam steering antenna, combinations thereof, and / or any other antenna configuration without departing from the scope of one or more embodiments herein.
[0093] The processor(s) 510 are further coupled via bus 505 to a display 525, such as a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, a Retina® display, an in-plane switching (IPS) display, or any other suitable display for displaying information to a user. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, or the like, using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse recognition, frustrated total internal reflection, or the like. Any suitable display device and haptic I / O may be used without departing from the scope of one or more embodiments herein.
[0094] A keyboard 530 and cursor control device 535, such as a computer mouse, touchpad, etc., are further coupled to the bus 505 to allow a user to interface with the computing system 500. However, in certain embodiments, a physical keyboard and mouse may not be present and the user may interact with the device solely through the display 525 and / or touchpad (not shown). Any type and combination of input devices may be used as a matter of design choice. In certain embodiments, there are no physical input devices and / or displays. For example, a user may interact with the computing system 500 remotely via another computing system that communicates with it, or the computing system 500 may operate autonomously.
[0095] The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. The modules include an operating system 540 for the computing system 500. The modules further include a module 545 (representing a decision engine including RPA and visualization software for automatically implementing / generating a graphic user interface heatmap with test automation details) configured to execute all or part of the processes described herein or derivatives thereof. The module 545 can include software code for generating the graphic user interface heatmap and providing therewith filters, mouse hover detection, configurable settings, test case view, test set view, and overall transaction view. The filters can provide additional possibilities to focus on the most relevant transactions and select relevant dates, and can further visualize the same within the graphic user interface heatmap. The mouse hover detection can provide additional information regarding the test case execution status, and can further visualize the same within the graphic user interface heatmap. The configurable settings can provide selectable options for the user to configure and modify the calculated values to determine test relevance or adjust said values as per the user's need. The Test Case View (for a selected transaction / process from the Graphic User Interface Heat Map) presents a view of all assigned test cases and allows creating new test cases or reviewing execution status. The Test Set View (for a selected transaction / process from the Graphic User Interface Heat Map) presents a view of all assigned test sets and allows immediate execution of the presented test sets either automatically or manually. The Overall Transaction View (for a selected transaction) can provide access to more detailed information of the selected transaction. Module 545 can be added to a software system or a standalone application.One or more technical effects, benefits, and advantages of module 545 (i.e., the decision engine) include providing analysis of test coverage and test relevance of actual usage transactions of a software system, reducing analysis time for analyzing usage data extracted using a collection program, and guiding new users to understand testing opportunities in a collection program through visualization of usage data. Thus, module 545 is necessarily rooted in the operation of computing system 500 to improve its functionality.
[0096] Computing system 500 may also include one or more additional functional modules 550 that contain additional functionality.
[0097] Those skilled in the art will appreciate that the "system" may be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device, or combination of devices, without departing from the scope of one or more embodiments herein. Presenting the above-described functions as being performed by a "system" is in no way intended to limit the scope of the embodiments herein, but is intended to provide an example of many embodiments. Indeed, the methods, systems, and apparatuses disclosed herein may be implemented in a localized and / or distributed form consistent with computing technologies, including cloud computing systems. The computing system may be part of or accessible by a local area network (LAN), a mobile communications network, a satellite communications network, the Internet, a public or private cloud, a hybrid cloud, a server farm, any combination thereof, and the like. Any local or distributed architecture may be used without departing from the scope of one or more embodiments herein.
[0098] It should be noted that some of the system features described herein have been presented as modules to more specifically emphasize their implementation independence. For example, a module may be implemented as a hardware circuit including custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, and the like.
[0099] Modules may also be implemented at least in part in software for execution by various types of processors. For example, an identified unit of executable code may include one or more physical or logical blocks of computer instructions organized as an object, procedure, or function. Nevertheless, executable files of identified modules need not be physically located together, but may include different instructions stored in different locations to accomplish the stated purpose of the modules, which when logically combined together comprise the modules. Furthermore, modules may be stored on a computer readable medium, which may be, for example, a hard disk drive, a flash device, RAM, tape, and / or any other non-transitory computer readable medium used to store data without departing from the scope of one or more embodiments herein.
[0100] Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed across several different code segments, different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within modules, and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set or distributed across different locations, including different storage devices, and may exist, at least in part, simply as electronic signals on a system or network.
[0101] Various types of AI / ML models can be trained and deployed without departing from the scope of one or more embodiments herein. For example, FIG. 6 illustrates an example of a neural network 600 trained to recognize graphical elements in an image, according to one or more embodiments. Here, neural network 600 receives pixels (represented by 610 columns) of a 1920×1080 screen screenshot image as input “neurons” 1 through I (represented by 620 columns) of the input layer. In this case, I is 2,073,600, which is the total number of pixels in the screenshot image.
[0102] Neural network 600 also includes several hidden layers (represented by columns 630 and 640). DLNNs and shallow learning neural networks (SLNNs) typically have multiple layers, although SLNNs may have only one or two layers in some cases and typically fewer layers than DLNNs. Typically, a neural network architecture includes an input layer, multiple intermediate layers (e.g., hidden layers), and an output layer (represented by column 650), as in the case of neural network 600.
[0103] DLNNs often have many layers (e.g., 10, 50, 200, etc.), with subsequent layers typically reusing features from previous layers to compute more complex and general functions. SLNNs, on the other hand, tend to have only a few layers and train relatively quickly because expert features are created in advance from raw data samples; however, feature extraction is laborious. DLNNs, on the other hand, usually do not require expert features, but they take longer to train and tend to have more layers.
[0104] For both approaches, the layers are trained simultaneously on the training set, usually with a check for overfitting on a separate cross-validation set. Both techniques can produce excellent results, and considerable enthusiasm is found for both approaches. The optimal size, shape, and amount of individual layers vary depending on the problem addressed by each neural network.
[0105] Returning to FIG. 6, the pixels provided as the input layer are fed as inputs to the J neurons of hidden layer 1. While in this example all pixels are fed to each neuron, a variety of architectures are possible, which may be used individually or in combination, including, but not limited to, feedforward networks, radial basis networks, deep feedforward networks, deep convolutional inverse graphics networks, convolutional neural networks, recurrent neural networks, artificial neural networks, long / short-term memory networks, gated recurrent unit networks, generative adversarial networks, liquid state machines, autoencoders, variational autoencoders, denoising autoencoders, sparse autoencoders, extreme learning machines, echo state networks, Markov chains, Hopfield networks, Boltzmann machines, restricted Boltzmann machines, deep residual networks, Kohonen networks, deep probability networks, deep convolutional networks, support vector machines, neural Turing machines, or any other suitable type or combination of neural networks without departing from the scope of one or more embodiments herein.
[0106] Hidden layer 2 (630) receives input from hidden layer 1 (620), hidden layer 3 receives input from hidden layer 2 (630), and so on for all hidden layers until the last hidden layer (represented by oval 655) provides its output as an input for the output layer. Note that the numbers of neurons I, J, K, and L are not necessarily equal, and thus any desired number of layers may be used in a given layer of neural network 600 without departing from the scope of one or more embodiments herein. Indeed, in certain embodiments, the types of neurons in a given layer may not all be the same.
[0107] Neural network 600 is trained to assign confidence scores to graphical elements believed to have been found in the image. To reduce matches with unacceptably low likelihood, in some embodiments, only results with confidence scores that meet or exceed a confidence threshold may be provided. For example, if the confidence threshold is 80%, outputs with confidence scores above this amount may be used and the rest may be ignored. In this case, the output layer indicates that two text fields (represented by outputs 661 and 662), a text label (represented by output 663), and a submit button (represented by output 665) were found. Neural network 600 may provide the location, dimensions, images, and / or confidence scores of these elements, which may then be used by the RPA robot or another process that uses this output for a given purpose, without departing from the scope of one or more embodiments herein.
[0108] Note that neural networks are probabilistic constructs that typically have a confidence score. This can be a score learned by the AI / ML model based on how often similar inputs were correctly identified during training. For example, text fields often have a rectangular shape and a white background. A neural network can learn to identify graphical elements with these characteristics with high confidence. Some common types of confidence scores include a decimal number between 0 and 1 (which can be interpreted as a percentage of confidence), a number between negative infinity and positive infinity, or a set of formulas (e.g., "low", "medium", "high"). Various post-processing calibration techniques can also be used to obtain more accurate confidence scores, such as temperature scaling, batch normalization, weight decay, and negative log-likelihood (NLL).
[0109] A "neuron" in a neural network is a mathematical function that is typically based on the function of a biological neuron. Neurons receive weighted inputs and have a summation and activation function that governs whether they pass on an output to the next layer. This activation function may be a non-linear thresholded activity function where nothing happens if the value is below the threshold, but the function responds linearly above the threshold (i.e., rectified linear unit (ReLU) non-linearity). Summation and ReLU functions are used in deep learning because real neurons can have roughly similar activity functions. Through linear transformations, information can be subtracted, added, etc. In essence, neurons act as gating functions that pass on an output to the next layer as governed by their underlying mathematical function. In some embodiments, different functions may be used for at least some neurons.
[0110] JPEG2024157530000001.jpg61170
[0111] JPEG2024157530000002.jpg55170
[0112] JPEG2024157530000003.jpg33170
[0113] In this case, the neuron 700 is a single-layer perceptron. However, any suitable neuron type or combination of neuron types may be used without departing from the scope of one or more embodiments herein. It should also be noted that the range of values of the weights and / or the output value(s) of the activation functions may vary in some embodiments without departing from the scope of one or more embodiments herein.
[0114] A goal or "reward function" is often used, such as in this case successfully identifying a graphical element in an image. The reward function guides the exploration of the state space, exploring intermediate transitions and steps with both short-term and long-term rewards in an attempt to achieve a goal (e.g., successfully identifying a graphical element, successfully identifying the next sequence of activities in an RPA workflow, etc.).
[0115] During training, various labeled data (in this case, images) are fed through the neural network 600. Successful identifications strengthen the input weights to the neurons, while failed identifications weaken the neurons. Using cost functions such as mean squared error (MSE) or gradient descent, slightly incorrect predictions can be handled much less often than very incorrect predictions. If the performance of an AI / ML model is not improving after a certain number of training iterations, a data scientist can modify the reward function, provide an indication of where the unidentified graphical elements are, provide compensation for incorrectly identified graphical elements, etc.
[0116] Backpropagation is a technique for optimizing synaptic weights in feedforward neural networks. Backpropagation can be used to "open the hood" on the hidden layers of a neural network to see how much loss every node contributes, and then update the weights to minimize the loss by giving lower weights to nodes with higher error rates, and vice versa. In other words, backpropagation allows data scientists to iteratively adjust the weights to minimize the difference between the actual output and the desired output.
[0117] The backpropagation algorithm is mathematically established in optimization theory. In supervised learning, training data with known outputs are passed through a neural network and an error is calculated from the known target output using a cost function, which gives the backpropagation error. An error is calculated at the output and this error is converted into a correction to the network weights that minimizes the error.
[0118] For supervised learning, an example of backpropagation is given below: A column vector input x is processed through a set of N nonlinear activity functions fi between each layer i=1,...,N of the network, and the output at a given layer is first multiplied by the synaptic matrix Wi and then a bias vector bi is added. The network output o is given by: o=f N (W N f N -1(W N-1 f N-2 (...f1(W1x+b1)...)+b N-1 )+b N )(4)
[0119] In some embodiments, o is compared to a target output t, and an error
number
[0120] An optimization in the form of a gradient descent procedure can be used to minimize the error by modifying the synaptic weights Wi of each layer. A gradient descent procedure requires the computation of an output o given an input x that corresponds to a known target output t, generating an error ot. This global error is then propagated backwards to provide a local error for computing weight updates that are similar, but not exactly the same, as those used for forward propagation. In particular, the backpropagation step is typically j (nj)=f j ´(n j ), where n j is the layer j (i.e., nj =W j o j-1 +b j ), where o j =f j( n j ), where the apostrophe ´ represents the derivative of the activity function f.
[0121] JPEG2024157530000005.jpg109147
[0122] where ○ denotes the Hadamard product (i.e., the element-wise product of two vectors), T represents the matrix transpose, and o j f j (W j o j-1 +b j ), where o0=x. Here, the learning rate η is chosen with respect to machine learning considerations. In the following, η is related to the neural Hebbian learning mechanism used in the neural implementation. Note that the synapses W and b can be combined into one large synaptic matrix, assuming the input vector is one-padded, with an extra column in W representing b synapses.
[0123] The AI / ML model can be trained over multiple epochs until it reaches a good level of accuracy (e.g., 97% or higher using an F2 or F4 threshold and about 2,000 epochs for detection). This accuracy level can be determined in some embodiments using an F1 score, an F2 score, an F4 score, or any other suitable technique without departing from the scope of one or more embodiments herein. Once trained on training data, the AI / ML model can be tested on a set of evaluation data that the AI / ML model has not encountered before. This helps ensure that the AI / ML model does not "overfit," such that it identifies graphical elements in the training data well but does not generalize well to other images.
[0124] In some embodiments, it may not be known what accuracy level the AI / ML model can achieve. Thus, if the accuracy of the AI / ML model starts to degrade when analyzing the evaluation data (i.e., the model is performing well on the training data but less well on the evaluation data), the AI / ML model may undergo more epochs of training on the training data (and / or new training data). In some embodiments, the AI / ML model is deployed only if the accuracy reaches a certain level or if the accuracy of the trained AI / ML model is better than the existing deployed AI / ML model.
[0125] In certain embodiments, a collection of trained AI / ML models can be used to accomplish tasks such as using an AI / ML model for each type of graphical element of interest, using an AI / ML model to perform OCR, deploying yet another AI / ML model to recognize proximity relationships between graphical elements, and using yet another AI / ML model to generate RPA workflows based on output from other AI / ML models, etc. This can, for example, collaboratively enable AI / ML models to enable semantic automation.
[0126] Some embodiments can use Transformer networks such as Sentence Transformers™, a Python™ framework for state-of-the-art sentence, text, and image embeddings. Such Transformer networks learn associations of words and phrases with both high and low scores. This trains an AI / ML model to determine which are close to the input and which are not, respectively. Transformer networks can also use field lengths and field types, rather than simply using word / phrase pairs.
[0127] FIG. 8 is a flow chart illustrating a process 800 for training an AI / ML model(s) according to one or more embodiments. It should be noted that the process 800 can also be applied to other UI learning operations such as NLP and chatbots. The process begins in block 810 with training data that provides labeled data as shown in FIG. 8, for example, labeled screens (e.g., where graphical elements and text are identified), words and phrases, a "thesaurus" of semantic associations between words and phrases such that similar words and phrases for a given word or phrase can be identified. The nature of the training data provided depends on the objectives that the AI / ML model is intended to achieve. The AI / ML model is then trained for multiple epochs in block 820, and the results are reviewed in block 830.
[0128] If, at decision block 840, the AI / ML model does not meet the desired confidence threshold (process 800 proceeds according to a NO arrow), then, at block 850, the training data is supplemented and / or the reward function is modified to help the AI / ML model better achieve its objective, and the process returns to block 820. If, at decision block 840, the AI / ML model meets the confidence threshold (process 800 proceeds according to a YES arrow), then, at block 860, the AI / ML model is tested against evaluation data to ensure that the AI / ML model generalizes well and does not overfit to the training data. The evaluation data may include screens, source data, etc., on which the AI / ML model has not previously been processed. If, at decision block 870, the confidence threshold is met for the evaluation data (process 800 proceeds according to a YES arrow), then, at block 880, the AI / ML model is deployed. If not (process 800 proceeds according to a NO arrow), the process returns to block 880 and the AI / ML model is further trained.
[0129] FIG. 9 is a flow chart illustrating a process 900 of a decision engine for automatically generating a graphic user interface heat map of most used system transactions with test automation details, according to one or more embodiments. The process 800 performed in FIG. 8 and the process 900 performed in FIG. 9 may be performed by a computer program encoding instructions for a processor(s), according to one or more embodiments. The computer program may be embodied on a non-transitory computer readable medium. The computer readable medium may be, but is not limited to, a hard disk drive, a flash device, RAM, tape, and / or any other such medium or combination of media used to store data. The computer program may include encoded instructions for controlling a processor(s) of a computing system (e.g., processor(s) 510 of computing system 500 of FIG. 5) to perform all or a portion of the process steps described in FIGS. 8-9, which may be stored on a computer readable medium.
[0130] According to one or more embodiments, the process 900 can be considered an example of an operation or method of a decision engine that obtains usage data of a software system and derives analytics from the usage data. The process 900 can be performed as add-on software or a standalone application of the software system. For example, the method can be applied to one or more of SAP (on-premise) ERP central component (ECC), SAP S / 4HANA, and SAP extended warehouse management (EWM). The process 900 can include utilizing the derived analytics to generate a graphic user interface heat map as a visualization of the usage of the software system.
[0131] According to one or more embodiments, the process 900 implemented by the decision engine can be applied to one or more use cases, such as existing user use cases (such as SAP systems) and new user use cases. For existing user use cases, the process 900 can enable a review of the current test portfolio, identify test case coverage based on actual usage of the SAP system, provide insights when automated tests cover transactions actually used in the SAP system, provide insights when manual tests cover transactions actually used in the SAP system, and identify gaps when transactions actually used do not have tests in the test portfolio. For new user use cases, the process 900 can enable projects to be started and established quickly in a short period of time, enable a quick jump start to a test project from scratch, and provide details on "what to test" and "where to start testing."
[0132] One or more technical effects, benefits, and advantages of method 900 include, but are not limited to, providing analysis for test coverage and test relevance of actual use transactions of a software system (e.g., providing analysis for test coverage and test relevance of actual use transactions of an SAP system), reducing analysis time for analyzing usage data extracted using a collection program (e.g., reducing analysis time for analyzing usage data extracted using an SAP collection program), and guiding new users to understand testing opportunities in collection programs through visualization of usage data (e.g., guiding new SAP users to understand testing opportunities in an SAP system through visualization of usage data). Another technical effect, benefit, and advantage of method 900 through the decision engine includes providing an easy to understand view of usage of a software system and answering questions regarding what to test and where to start testing.
[0133] Process 900 begins at block 910 with a decision engine performing data integration. Data integration can include aggregating data and normalizing the data into integrated data. For example, data integration by the decision engine can be a custom application or sub-code that performs analysis of transactional data of a software system to output integrated data.
[0134] According to one or more embodiments, in sub-block block 925, the decision engine performs data aggregation. For example, data aggregation can include a custom application or sub-code of the decision engine that performs or obtains transaction data (e.g., data generated by the ST03 system across all servers, users, and transactions, as well as data generated by SAP transactions, SAP Fiori applications, SAP Webdynpro, SAP WinGUI, SAP WebGUI, etc.) to generate aggregated data. Note that transaction data is a huge amount of data, such as over 10GB of technical data, that is outside the capacity of the human brain to process in a reasonable or cost-effective manner. According to one or more embodiments, the custom application or sub-code of the decision engine can be a collection program. The collection program can be run at time intervals, such as every minute, to aggregate the transaction data. The aggregated transaction data can be further accumulated at additional intervals, such as hourly, daily, weekly, etc.
[0135] In sub-block 935, a custom application or sub-code is performed to aggregate performance data metrics of the aggregated data (i.e., the cumulative transaction data generated in ST03). According to one or more embodiments, the custom application or sub-code aggregates the performance data metrics over a predetermined period of time, such as days, weeks, and / or months.
[0136] In sub-block 945, the aggregate data and performance data metrics are normalized by a custom application or sub-code to prepare the consolidated data. By way of example, the aggregate data and performance data metrics may undergo transformations, such as user-provided weights, to output the consolidated data.
[0137] According to one or more embodiments, the weights can be default weights. A user can also adjust the weights according to need / relevance. Examples of one or more categories of weights (e.g., SAP attributes) include, but are not limited to, user, dialog / completion, time / completion time, object / development object, and intensity / transaction type. As a further example, SAP attributes can be taken into account so that aggregate data can be transformed by the decision engine according to the provided weights.
[0138] The users as weight may include the number of users using the transaction. In this regard, a transaction may be given more weight depending on the number of users. For example, the more users that use or execute a transaction, the more importantly the transaction may be weighted (hence, a higher weight). For example, the number of users executing a transaction may be equal to the weight. The users as weight may be determined / calculated by a decision engine and / or may be selected from a range such as 0.0000 to 1.0000.
[0139] Dialogue / commitment as weight may include the number of times a transaction is performed. In this regard, a transaction can be given a higher weight depending on the number of times it is performed. For example, the more a transaction is performed, the more important it is that the transaction can be weighted (hence a higher weight). As a further example, the decision engine can determine how many "dialog steps" were performed per one dialog step of one screen in the SAP system. Dialogue / commitment as weight can be determined / calculated by the decision engine and / or can be selected from a range such as 0.0000 to 1.0000.
[0140] The time / performance time as weighting may include the overall processing time of the software system. In this regard, a transaction may be given a greater weight depending on the length of the transaction performance time. For example, the longer it takes to process / perform a transaction, the more it may be weighted. As a further example, the decision engine may determine the time the SAP system needs to process an action, which may include graphic user interface time, database time, network time, etc. The time / performance time as weighting may be determined / calculated by the decision engine and / or may be selected from a range, such as 0.0000 to 1.0000.
[0141] The object / development object as weight may include the number of development objects. In this regard, a greater weight may be given to an application depending on the number of objects. For example, the more objects an application has, the more its transactions may be weighted. As a further example, the decision engine may determine the development objects of individual parts of an Advanced Business Application Programming (ABAP) application, such as screens, functional modules, tables, domains, data elements, fields, classes, etc. The object / development object as weight may be determined / calculated by the decision engine and / or may be selected from a range, such as 0.0000 to 1.0000.
[0142] The strength / transaction type as weight may include the type of transaction (e.g., create, modify, view, test, etc.). In this regard, different transactions may be given more weight depending on the type. According to one or more embodiments, a create type transaction may have a higher importance than a modify type transaction. According to one or more embodiments, a view type transaction may have a lower importance than a test type transaction. According to one or more embodiments, a transaction that only allows viewing may be weighted as less important than a transaction that allows insert / update / edit actions. The strength / transaction type as weight may be determined / calculated by a decision engine and / or may be selected from a range such as 0.0000 to 1.0000.
[0143] At block 970, the decision engine performs data transformation and visualization, which may include custom applications or sub-code to calculate the final rating and generate the user interface, respectively.
[0144] For example, in subblock 985, the decision engine performs a data transformation by utilizing a mathematical algorithm to analyze the integrated data to "grade" the data therein. According to one or more embodiments, the mathematical algorithm generates a final rating by an average or weighted average of one or more categories for weights (e.g., SAP attributes, or user, dialog / execution, time / execution time, object / development object, and intensity / transaction type). The final rating can indicate the degree of relevance of the data (e.g., test relevance) or intensity of the transaction. The final rating can give guidance on "where to start automating". Once the integrated data is transformed into transformed data with a final rating (according to the weights provided), all test cases related to existing transactions in the transformed data are identified and executed by the decision engine. The test cases can be identified using the transaction label associated with it. For example, the test cases can be identified using the SAP transaction or SAP program label.
[0145] Further, in subblock 995, the decision engine generates a user interface that visually presents the transformed data. According to one or more embodiments, the user interface includes a heatmap having one or more elements, each including one or more qualifications. For example, the one or more elements can be shapes, such as a box in the heatmap. Furthermore, the one or more qualifications can include a size and / or a color of the shape. An exemplary qualification can be a final rating, which can be represented as a "size" of the box. The final rating or size can be derived by an average or weighted average of one or more categories for weights (e.g., SAP attributes, or users, dialogue / fulfillment, time / fulfillment time, objects / development objects, and intensity / transaction type). According to one or more embodiments, the final rating can give guidance on "where to start automating".
[0146] FIG. 10 illustrates an interface 1000 according to one or more embodiments. The interface 1000 is an example of a user interface generated by the decision engine in subblock 995. The interface 1000 can provide a visualization of the transformed data (which may originally be SAP usage data) via a heat map 1001 with test relevance. The interface 1000 includes a toolbar 1005 that can present one or more selectable elements, such as a task manager, a test menu, a profile icon, a search icon, etc. The interface 1000 includes a first panel 1010 that can present one or more selectable elements, such as a home icon 1011, an orchestrator icon 1012, a studio icon 1013, an action icon 1014, and an automation hub icon 1015, a process icon 1016, a test manager icon 1017, and an admin icon 1018. The interface 1000 includes a second panel 1020 that can present one or more selectable options based on which of the selectable elements of the first panel 1010 is selected. For example, when the test manager icon 1017 in the first panel 1010 is selected, the second panel 1020 may present a dashboard option 1021, a heat map option 1022, a requirements option 1023, a test case option 1024, a test set option 1025, a test result option 1026, and a project settings option 1027.
[0147] The interface 1000 includes a third panel 1030 that can present one or more menu items 1031 and 1032, and one or more selectable filters 1035, 1036, 1037 and 1038, presented alongside the heatmap 1001 and available for manipulating the heatmap 1001. For example, Figure 11 illustrates an interface 1100 according to one or more embodiments. Similar elements and identifiers from other figures are reused and will not be reintroduced for brevity.
[0148] The interface 1100 provides a filtering function that provides additional possibilities for selecting relevant dates to focus on the most relevant transactions. In this regard, one or more menu items 1031 and 1032 presented alongside the heat map 1001 are available for manipulating the heat map 1001. The menu item 1031 may be a drop-down menu 1101 that provides selectable menu options such as all transactions, top ten transactions, and top twenty (20) transactions. The menu item 1032 may be a drop-down menu 1102 that provides time intervals such as last day, last week, last month, etc. Other examples include, but are not limited to, last quarter, last year, last six months, etc.
[0149] Returning to Figure 10, one or more selectable filters 1035, 1036, 1037, and 1038 presented alongside the heat map 1001 can be utilized to manipulate the heat map 1001. For example, selectable filter 1035 can be a toggle checkbox to show all, selectable filter 1036 can be a toggle checkbox to show pass, selectable filter 1037 can be a toggle checkbox to show fail, and selectable filter 1038 can be a toggle checkbox to show no assigned test cases.
[0150] The heat map 1001 may include a first process 1040 having one or more elements or shapes 1041, 1042, 1043, 1044, and 1045 (e.g., squares) representing transactions and / or tests thereof. The heat map 1001 may include a second process 1050 having one or more elements or shapes 1051, 1052, and 1053 (e.g., squares) representing transactions and / or tests thereof. The heat map 1001 may include a third process 1060 having one or more elements or shapes 1061, 1062, and 1063 (e.g., squares) representing transactions and / or tests thereof. Thus, from the transformed data and test results, each element or shape of any process 1040, 1050, and 1060 (e.g., representing transactions / processes of an SAP system instance) includes one or more qualifications such as size (e.g., final rating) and color.
[0151] The final rating or size can be derived by an average or weighted average of one or more categories for weights (e.g., SAP attributes, or users, dialog / fulfillment, time / fulfillment time, object / development object, and intensity / transaction type). According to one or more embodiments, the final rating can provide guidance on "where to start automating".
[0152] The size of the shape may indicate test relevance, which may refer to the degree to which a test is relevant or important to achieving a particular goal. The test relevance or size is calculated and weighted by the decision engine based on one or more categories for weighting.
[0153] The color of the shape may indicate the test coverage. Test coverage may be a visualization that indicates the execution coverage of a transaction that is executed by a test case. That is, the identified test cases are executed to determine the test coverage of the transaction. One or more colors may be utilized. According to one or more embodiments, three colors are utilized by the decision engine in the heat map 1001. A first color (e.g., gray) indicates that a test case does not exist or was not executed. A second color (e.g., red) indicates that a test case exists and at least one execution has failed. A third color (e.g., green) indicates that a test case exists and all executions have passed.
[0154] As an example, the first process 1040 of the heat map 1001 of Figure 10 can be divided as follows: element 1041 can provide a general table display at 25% size, element 1042 can provide an example user interface design at 25% size, element 1043 can provide a demo of a development workbench at 12% size, element 1044 can provide a pause program block at 11% size, and element 1045 can provide an editor at 8% size, totaling 75% of the area of the heat map 1001. Additionally, although shading is not shown in Figure 10, elements 1041, 1042, and 1044 can be a second color, element 1043 can be a first color, and element 1045 can be a third color.
[0155] The second process 1050 of the heat map 1001 of Figure 10 can be divided as follows: element 1051 can provide purchase change orders at 6% of the size, element 1052 can provide cash journals at 3% of the size, and element 1053 can provide account statements at 3% of the size, totaling 12% of the area of the heat map 1001. Additionally, although shading is not shown in Figure 10, elements 1052 and 1053 can be a second color and element 1051 can be a third color.
[0156] Process 1060 of heat map 1001 of Figure 10 can be divided as follows: element 1061 can provide an invoice at 2% size, element 1062 can provide a financial report at 2% size, and element 1063 can provide a general table display at 14% size, totaling 18% of the area of heat map 1001. Additionally, although shading is not shown in Figure 10, element 1062 can be a first color, element 1061 can be a second color, and element 1063 can be a third color.
[0157] 12-13, interfaces 1200, 1300, and 1400 are shown in accordance with one or more embodiments. Interfaces 1200, 1300, and 1400 are examples of user interfaces generated by the decision engine in subblock 995. Similar elements and identifiers to other figures are reused and will not be reintroduced for brevity. For example, interface 1200 can include sub-interface 1201 having one or more interface elements 1221, 1222, and 1223 to provide selectable views.
[0158] When a transaction is selected and / or when an interface element 1221 is selected, the interface 1200 of FIG. 12 shows a test case view in a sub-interface 1201 of the selected transaction. That is, upon selecting any element or shape of any process 1040, 1050, and 1060 (e.g., representing a transaction / process of an SAP system instance) from the heat map 1001, a test case view can be provided by the interface 1200. Furthermore, the test case view can include one or more interface elements 1231, 1232, 1233, and 1234 to provide additional options such as a search option, a label selection, a latest result selection, and a studio project selection. Within the test case view, a user can create a new test case 1235 or review the execution status. Within the test case view, a table 1240 is provided. The table 1240 can provide all the test cases assigned to the selected transaction / process. The table 1240 can include sortable columns such as key, name, studio project, last modification, last result, and label. According to one or more embodiments, test case creation by selecting the create element 1235 is described with respect to sub-process 1290. Sub-process 1290 may be performed by a decision engine and includes blocks 1291, 1293, and 1295. In block 1291, any process 1040, 1050, and 1060 or elements therein may be selected. For example, an element may be selected that includes a first color (e.g., gray) indicating that no test cases exist. Next, in block 1293, the create element 1235 is selected to enable test case creation for that process 1040, 1050, and 1060 or elements therein. In this regard, a prompt may be displayed with one or more fields to at least accept input such as the name of the test case. In block 1295, the created test cases for the process 1040, 1050, and 1060 or elements therein are loaded into the heat map 1001.According to one or more embodiments, if any of processes 1040, 1050, and 1060 or elements therein have test cases, the decision engine creates additional test cases using sub-process 1290. According to one or more embodiments, if a test case is part of a test set (as shown in FIG. 13 ), the decision engine can execute the test case with the test set, either independently or via interface 1200.
[0159] When a transaction is selected and / or when an interface element 1222 is selected, the interface 1300 of FIG. 13 shows a test set view in the sub-interface 1201 of the selected transaction. That is, when any element or shape of any process 1040, 1050, and 1060 (e.g., representing a transaction / process of an SAP system instance) is selected from the heat map 1001, a test set view can be provided by the interface 1300. Furthermore, the test case view can include one or more interface elements 1331, 1332, and 1333 and can provide additional options such as a search option, a label selection, and a source selection. Within the test case view, a table 1340 is provided. The table 1340 can provide all assigned test sets and can enable immediate execution of one or more of the assigned test sets, either automatically or manually. For example, a popup 1341 provides selectable commands for the selected test set, such as execute manually, execute automatically, clone, and delete.
[0160] When a transaction is selected and / or when an interface element 1223 is selected, the interface 1400 of FIG. 14 shows an overview of the sub-interfaces 1201 of the selected transaction. The interface 1400 of FIG. 14 shows an overview of the transaction. That is, upon selection of any element or shape of any process 1040, 1050, and 1060 (e.g., representing a transaction / process of an SAP system instance) from the heat map 1001, an overview can be provided by the interface 1400. The overview can provide a table 1440. The table 1440 can provide access to more detailed information of the selected transaction. For example, the table 1440 can include transaction type, code, type, rating or final rating, and usage.
[0161] 15-16, interfaces 1500 and 1600 are shown in accordance with one or more embodiments. Interfaces 1500 and 1600 are examples of user interfaces generated by the decision engine in subblock 995. Similar elements and identifiers to other figures are reused and will not be reintroduced for brevity. Interface 1500 in FIG. 15 shows a mouse cursor 1501 hovering over element 1041. Hovering for a predefined time, such as immediately, 1 second, 2 seconds, etc., presents a panel 1503 that provides additional information about element 1041 (i.e., test case execution status). The additional information can include the result (pass or fail), the number of test cases, the rating or final rating, and the number of runs or executions.
[0162] The interface 1600 of FIG. 16 shows additional configurable settings. A right mouse click or hover to select the settings icon 1601 presents a panel 1602 offering one or more settings that the user can configure between 0% and 100%. The one or more settings allow the user to determine the modification of values that determine the test relevance according to the customer's needs. For example, one or more categories of weights can be modified as follows: the user weight 1611 can be modified to 40%, the performance weight 1612 can be modified to 10%, the time weight 1613 can be modified to 10%, the object weight 1614 can be modified to 10%, and the intensity weight 1615 can be modified to 30%. Thus, once one or more settings are configured, the cancel interface element 1616 and the save interface element 1617 can be used.
[0163] According to one or more embodiments, a method for transforming and visualizing integrated data is provided. The method is performed by a decision engine implemented as a computer program in a computing environment. The method includes performing data integration to normalize the aggregated data to integrated data by subjecting the aggregated data to a weight-wise transformation to output the integrated data. The method includes performing data transformation and visualization by determining a final rating for the integrated data and generating a user interface on a display including a heat map depicting the integrated data according to the final rating.
[0164] According to any of the one or more embodiments or method embodiments herein, the transaction data may include data generated by one or more servers, one or more users, or a software system across one or more transactions.
[0165] According to any of the one or more embodiments or method embodiments herein, the collection program of the decision engine may be performed at one or more time intervals to aggregate transaction data.
[0166] According to any of the one or more embodiments or method embodiments herein, the weights may include at least one or more of the following categories: a user category, a dialogue category, a time category, an object category, and an intensity category.
[0167] According to any of one or more embodiments or method embodiments herein, the dialog category may include a number of times a transaction is completed.
[0168] According to one or more embodiments or any of the method embodiments herein, the time category may include an overall processing time of the software system.
[0169] According to any of one or more embodiments or method embodiments herein, the object category may include a number of development objects.
[0170] According to any of one or more embodiments or method embodiments herein, the object category may include a type of transaction.
[0171] According to one or more embodiments or any of the method embodiments herein, the weights may be determined by a decision engine along a range of 0.0000 to 1.0000.
[0172] According to any of the one or more embodiments or method embodiments herein, the decision engine may perform data transformation by utilizing mathematical algorithms to analyze the integrated data to generate a final rating.
[0173] According to any of the one or more embodiments or method embodiments herein, the final rating may include an average or weighted average of one or more categories for weighting.
[0174] According to any of the one or more embodiments or method embodiments herein, the final rating may indicate a degree of relevance or strength of the aggregated data.
[0175] According to one or more embodiments or any method embodiment herein, the heat map may include one or more elements that include one or more qualifications.
[0176] According to any of one or more embodiments or method embodiments herein, the one or more elements may include a box and the one or more qualifications include a size of the box or a color of the box.
[0177] According to any of the one or more embodiments or method embodiments herein, the size of the box can correspond to the final rating.
[0178] According to any of the one or more embodiments or method embodiments herein, the color of the box can correspond to test coverage.
[0179] According to any of the one or more embodiments or method embodiments herein, the decision engine can be added to the software of a software system, or an independent application.
[0180] According to any of the one or more embodiments or method embodiments herein, the method implemented by the decision engine may be applicable to one or more of SAP ECC, SAP S / 4HANA, and SAP EWM.
[0181] According to any of the one or more embodiments or method embodiments herein, performing data integration may include aggregating transactional data of software systems to generate aggregated data.
[0182] According to one or more embodiments, a computing environment is provided. The computing environment includes at least one processor that executes a computer program for a decision engine. The computer program is stored in a memory of the computing environment. The decision engine is configured to perform transformation and visualization of the integrated data by having the computing environment perform data integration that normalizes the aggregate data to the integrated data by subjecting the aggregate data to a weight-wise transformation to output the integrated data. The decision engine is configured to perform transformation and visualization of the integrated data by having the computing environment perform data transformation and visualization by determining a final rating of the integrated data and generating a user interface on a display that includes a heat map that depicts the integrated data according to the final rating.
[0183] The computer program may be implemented in hardware, software, or a hybrid implementation. The computer program may be composed of modules designed to operatively communicate with each other and pass information or instructions for display. The computer program may be configured to operate on a general purpose computer, an ASIC, or any other suitable device.
[0184] It will be readily understood that the components of the various embodiments, as generally described and illustrated in the figures herein, could be arranged and designed in a wide variety of different configurations, and thus, the detailed description of the embodiments depicted in the accompanying drawings is not intended to limit scope as claimed, but is merely representative of selected embodiments.
[0185] Features, structures, or characteristics described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, references throughout this specification to "certain embodiments," "some embodiments," or similar language means that the particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. Thus, the appearances throughout this specification of "certain embodiments," "some embodiments," "other embodiments," or similar language do not necessarily all refer to the same group of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0186] It should be noted that references to features, advantages, or similar language throughout this specification do not imply that all of the features and advantages that may be realized should or are in any single embodiment. Rather, language referring to features and advantages is understood to mean that the particular feature, advantage, or characteristic described in connection with an embodiment is included in one or more embodiments. Thus, descriptions of features and advantages and similar language throughout this specification may, but do not necessarily, refer to the same embodiment.
[0187] Furthermore, the described features, advantages, and characteristics of one or more embodiments herein may be combined in any suitable manner. Those skilled in the art will recognize that the present disclosure may be practiced without one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments.
[0188] Those skilled in the art will readily appreciate that the present disclosure may be practiced with steps in a different order and / or hardware elements in a different configuration than that disclosed. Thus, while the present disclosure has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative constructions are apparent while remaining within the spirit and scope of the present disclosure. Therefore, reference should be made to the appended claims to determine the scope and boundaries of the present disclosure.
Claims
1. 1. A method for integrated data transformation and visualization, said method being performed by a decision engine implemented as a computer program in a computing environment, comprising: performing, by the decision engine, a data integration step of normalizing the aggregated data to the integrated data by subjecting the aggregated data to a weighted transformation to output integrated data; determining, by the decision engine, a final rating of the integrated data, and performing data transformation and visualization by generating a user interface on a display including a heat map depicting the integrated data according to the final rating.
2. The method of claim 1 , wherein the step of performing data integration comprises aggregating transactional data of software systems to generate the aggregated data.
3. 3. The method of claim 2, wherein the transaction data comprises data generated by the software system across one or more servers, one or more users, or one or more transactions.
4. The method of claim 1 , wherein the collection program of the decision engine executes at one or more time intervals to aggregate transaction data.
5. The method of claim 1 , wherein the weights include at least one or more of the following categories: a user category, a dialogue category, a time category, an object category, and an intensity category.
6. The method of claim 5 , wherein the dialog categories include a number of times a transaction is completed.
7. The method of claim 5 , wherein the time category comprises an overall processing time of a software system.
8. The method of claim 5 , wherein the object category comprises a number of development objects.
9. The method of claim 5 , wherein the object category comprises a type of transaction.
10. The method of claim 5 , wherein the weights are determined by the decision engine along a range from 0.0000 to 1.0000.
11. The method of claim 1 , wherein the decision engine performs the data transformation by utilizing a mathematical algorithm to analyze the combined data to generate the final rating.
12. The method of claim 1 , wherein the final rating comprises an average or weighted average of one or more categories for the weights.
13. The method of claim 1 , wherein the final rating indicates the relevance or strength of the aggregated data.
14. The method of claim 1 , wherein the heat map includes one or more elements that include one or more qualifications.
15. The method of claim 14 , wherein the one or more elements include a box, and the one or more qualifications include a size of the box or a color of the box.
16. The method of claim 15 , wherein the size of the box corresponds to the final rating.
17. The method of claim 15 , wherein the color of the box corresponds to test coverage.
18. The method of claim 1 , wherein the decision engine is added to a software system or a separate application.
19. The method of claim 1 , wherein the method implemented by the decision engine applies to one or more of SAP ECC, SAP S / 4HANA, and SAP EWM.
20. A computing environment including at least one processor executing a computer program for a decision engine, the computer program is stored in a memory of the computing environment; performing data integration to normalize the aggregated data to the integrated data by subjecting the aggregated data to a transformation by weights to output the integrated data; determining a final rating of the integrated data and performing data transformation and visualization by generating a user interface on a display including a heat map depicting the integrated data according to the final rating; causing the computing environment to the decision engine is configured to implement transformation and visualization of the integrated data; Computing environment.
