Automatic annotation and technical specification generation for robotic process automation workflows using artificial intelligence (AI)

Abstract

Automatic annotation and technical specification generation for robotic process automation (RPA) workflows using artificial intelligence (AI) is disclosed. The AI / ML model may enable an intelligent search of the workflow and automatically generate documents of the workflow, including descriptions of each activity, input / output parameters, and overall process interpretations. Annotations and documents may be automatically provided for the entire complex business as a sum of multiple workflows and applications. A process-defined document (PDD) where a traffic flow can be generated from the RPA workflow code itself when the RPA workflow code itself is not present. Other files, such as audit files, compliance files of legal or regulatory requirements, and the like, may be generated. The process may be iterative in which the generative AI model automatically converts text to RPA workflow code, produces runtime automation from the RPA workflow, and also generates other documents.

Description

Technical Field

[0001] The present invention generally relates to artificial intelligence (AI), and more particularly, to automatic annotation and technical specification generation for robotic process automation (RPA) workflows using AI. Background Art

[0002] When manually building, testing, releasing, and deploying RPA workflows, it can be difficult to understand what each stage of the automation / workflow is doing during troubleshooting / debugging due to the frequent lack of annotations and other documentation. Thus, improved and / or alternative methods may be beneficial. Summary of the Invention

[0003] Certain embodiments of the present invention may provide solutions to problems and needs in the art that have not been fully recognized, understood, or solved by current RPA technologies. For example, some embodiments of the present invention relate to automatic annotation and technical specification generation for RPA workflows using AI.

[0004] In one embodiment, a non-transitory computer-readable medium stores a computer program for providing automatic annotation for an RPA workflow. The computer program is configured to cause at least one processor to provide code for an RPA workflow or a process definition document (PDD) to a cognitive AI layer. The computer program is further configured to cause at least one processor to process the code for the RPA workflow or PDD by the cognitive AI layer. The computer program is further configured to cause at least one processor to provide, as output, annotations for the RPA workflow or RPA workflow code including the annotations by a generative AI layer.

[0005] In another embodiment, one or more computing systems include a memory for storing computer program instructions for providing automatic annotation for an RPA and at least one processor configured to execute the computer program instructions. The computer program instructions are configured to cause at least one processor to provide code for an RPA workflow or PDD to a cognitive AI layer, the cognitive AI layer including a generative AI model configured to generate the annotated RPA workflow, provide semantic associations between texts on a screen, logically group categories of RPA workflow activities, reason about subsequent activities to be added based on the context of the RPA workflow, convert the RPA workflow from the format of one RPA provider to the format of another RPA provider, or any combination thereof. The computer program instructions are further configured to cause at least one processor to process the code for the RPA workflow or PDD by the cognitive AI layer. The computer program instructions are further configured to cause at least one processor to provide, as output, annotations for the RPA workflow or RPA workflow code including the annotated RPA workflow. The annotations provide a description of the entire process of the RPA workflow, describe what each activity in the RPA workflow is doing, or both.

[0006] In yet another embodiment, a computer-implemented method for providing automated annotation of an RPA workflow includes providing, by an RPA designer application executing on a computing system, code of the RPA workflow to a cognitive AI layer or a PDD. The cognitive AI layer is configured to process the code of the RPA workflow and / or the PDD. The computer-implemented method further includes receiving, by the RPA designer application, an annotation of the RPA workflow or the RPA workflow code including the annotation from a generative AI layer. The computer-implemented method further includes displaying, by the RPA designer application, the annotated RPA workflow. The annotation provides a description of the overall process of the RPA workflow, a description of what each activity in the RPA workflow is doing, a description of the changes between the current version of the RPA workflow and one or more previous versions of the RPA workflow, or any combination thereof.

[0007] In yet another embodiment, a non-transitory computer-readable medium stores a computer program. The computer program is configured to cause at least one processor to provide a natural language description of a process to a cognitive AI layer. The computer program is further configured to cause at least one processor to process the natural language description by the cognitive AI layer. The computer program is further configured to cause at least one processor to generate an annotated RPA workflow by a generative AI layer.

[0008] In another embodiment, one or more computing systems include a memory storing computer program instructions and at least one processor configured to execute the computer program instructions. The computer program instructions are configured to cause at least one processor to provide a natural language description of a process to a cognitive AI layer. The computer program instructions are further configured to cause at least one processor to process the natural language description by the cognitive AI layer. The computer program instructions are further configured to cause at least one processor to generate an annotated robotic process automation (RPA) workflow by a generative AI layer.

[0009] In yet another embodiment, a computer-implemented method includes providing a natural language description of a process to a cognitive AI layer on one or more computing systems. The computer-implemented method further includes processing the natural language description by the cognitive artificial intelligence layer. The computer-implemented method further includes generating an annotated RPA workflow by a generative AI layer. The annotation provides a description of the overall process of the RPA workflow, a description of what each activity in the RPA workflow is doing, including a description of the changes between the current version of the RPA workflow and one or more previous versions of the RPA workflow, or any combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To facilitate an understanding of the advantages of certain embodiments of the present invention, a more specific description of the invention briefly described above will be presented by reference to specific embodiments shown in the accompanying drawings. While it should be understood that these drawings depict only typical embodiments of the invention and should not be considered as limiting its scope, the invention will be described and explained with additional features and details by using the drawings, wherein:

[0011] Figure 1 is an architectural diagram showing a hyper-automation system according to an embodiment of the present invention.

[0012] Figure 2 is an architectural diagram showing an RPA system according to an embodiment of the present invention.

[0013] Figure 3 is an architectural diagram showing a deployed RPA system according to an embodiment of the present invention.

[0014] Figure 4 is an architectural diagram showing the relationship between a designer, activities, and a driver according to an embodiment of the present invention.

[0015] Figure 5 is an architectural diagram showing a computing system configured to perform automatic annotation and technical specification generation for an RPA workflow using AI according to an embodiment of the present invention.

[0016] Figure 6A shows an example of a neural network that has been trained to use AI to supplement automatic annotation and technical specification generation for an RPA workflow according to an embodiment of the present invention.

[0017] Figure 6B shows an example of a neuron according to an embodiment of the present invention.

[0018] Figure 7 is a flowchart showing a process for training an AI / ML model according to an embodiment of the present invention.

[0019] Figure 8 shows an AI / ML model of a cognitive AI layer according to an embodiment of the present invention.

[0020] Figure 9 is a screenshot showing an annotated RPA workflow according to an embodiment of the present invention.

[0021] Figure 10 is a flowchart showing a process for providing automatic annotation and technical specification generation for an RPA workflow using AI for existing RPA automation according to an embodiment of the present invention.

[0022] Figure 11It is a flowchart showing a process for using AI to provide automatic annotation and technical specification generation for an RPA workflow when a developer creates an RPA workflow according to an embodiment of the present invention.

[0023] Figure 12 It is a flowchart showing a process for creating an RPA workflow using PDD or another description of a business process as input according to an embodiment of the present invention.

[0024] Unless otherwise specified, like reference numerals in all figures always denote corresponding features. Detailed Description

[0025] Some embodiments relate to automatic annotation and technical specification generation of RPA workflows using AI. Generally, annotations and other documents describing activities in RPA workflows are rarely or completely absent. Consider the case of an RPA workflow activity called "invoke code". In the activity, there are 10 lines of code. Testers or subsequent users of the automation may not know what the activity does. In another example, when an RPA workflow includes 30 or 40 activities with "use browser" as the parent activity, it may not be easy to understand what is happening or remember the automation flow, especially for non-technical users.

[0026] Some embodiments use artificial intelligence / machine learning (AI / ML) models to implement intelligent search of workflows and automatically generate documentation for the workflows, including descriptions of each activity, input / output parameters, and overall process descriptions. These embodiments can be used as part of a cognitive AI layer. Such embodiments can tell users what the activities, sequences, workflows, processes, and / or the entire automation are doing. The text can appear in the RPA workflow itself and / or be provided in a separate document (e.g., a process definition document (PDD)). In some embodiments, by automatically creating workflows and corresponding annotations and / or documentation, workflows from other RPA applications can be converted into workflows for a different RPA designer application as desired. In other words, previously created automations can be automated.

[0027] In some embodiments, software (e.g., an RPA designer application, such as the visual modeling tool UiPath Studio TM, monitor their operations during user development of RPA workflows (such as independent processes) and monitor the workflows when they create the workflows. When users develop applications, the software invokes (multiple) AI / ML models by providing RPA workflow information as input. For example, screenshots of the RPA designer application interface, the code of the RPA workflow (e.g., files or file streams in Extensible Markup Language (XML), Extensible Application Markup Language (XAML), Hypertext Markup Language (HTML), any other suitable format, etc.), which includes activities and their parameters (e.g., inputs, outputs, settings, UI descriptor types, etc.), business process documents, etc. can be provided to the (multiple) AI / ML models. The (multiple) AI / ML models process this information and provide recommended annotations, PDDs, etc. For example, the (multiple) AI / ML modules can provide a code version of the RPA workflow including annotations. The RPA designer application can then display this version of the RPA workflow to the user.

[0028] In some embodiments, annotations and documentation can be provided for an entire complex business automation, which is the sum of multiple workflows and applications, such as the entire recruitment process of a company, the company's procurement process, loan applications, etc. In such embodiments, the PDD can be used as a starting point. The PDD outlines the business processes selected for automation. Specifically, the PDD attempts to lay the foundation for ensuring that the process is a good candidate for automation. The PDD describes the sequence of steps executed as part of the business process, the conditions and rules of the process before automation, and how it works after the process is automated (whether partially or fully). The PDD can serve as a reference tool for developers and non-technical users, providing the detailed information required to apply RPA technology to the process.

[0029] In some embodiments, when the RPA workflow code itself does not exist, the PDD for the business process can be generated from the RPA workflow code itself. The (multiple) AI / ML models take the code of the RPA workflow, which includes activities and their parameters, and automatically generate the PDD as output from this input. In some embodiments, the (multiple) AI / ML models can also output an annotated version of the (multiple) RPA workflows. In some embodiments, other files can be generated, such as audit files, compliance files required by laws or regulations, etc.

[0030] In some embodiments, the process is iterative. The generative AI model can be used as part of the cognitive AI layer to automatically convert text into RPA workflow code. Runtime automation can be generated from this RPA workflow, and other documents, such as PDDs, can also be generated. In such embodiments, automation is created and documented as the business user envisions the business process to be automated, without the need for manual programming.

[0031] Figure 1 It is an architectural diagram showing the hyper-automation system 100 according to an embodiment of the present invention. As used herein, "hyper-automation" refers to an automation system that combines components for process automation, integration tools, and technologies that enhance the automation capabilities. For example, in some embodiments, RPA can be used at the core of the hyper-automation system, and in certain embodiments, the automation capabilities can be extended using AI / ML, process mining, analytics, and / or other advanced tools. As the hyper-automation system learns the process, trains the AI / ML model, and employs analytics, for example, more and more knowledge work can be automated, and computing systems in the organization, such as those used by individuals and those operating autonomously, can all participate in the hyper-automation process. The hyper-automation systems of some embodiments allow users and organizations to efficiently and effectively discover, understand, and scale automation.

[0032] The hyper-automation system 100 includes user computing systems, such as desktop computer 102, tablet computer 104, and smart phone 106. However, any desired user computing system can be used, including but not limited to smart watches, laptop computers, servers, Internet of Things (IoT) devices, etc. without departing from the scope of the present disclosure. Additionally, although Figure 1 three user computing systems are shown, any suitable number of user computing systems can be used without departing from the scope of the present invention. For example, in some embodiments, dozens, hundreds, thousands, or millions of user computing systems can be used. The user computing systems can be actively used by the user or operate automatically with little or no user input.

[0033] Each user computing system 102, 104, 106 has a corresponding (multiple) automation process 110, 112, 114 running thereon. In some embodiments, the automation processes are stored remotely (e.g., in server 130 or database 140 and accessed via network 120), and are loaded by RPA robots to achieve automation. The automation can exist as a script (e.g., XML, XAML, etc.) or be compiled into machine-readable code (e.g., as a digital link library).

[0034] (Multiple) automation processes 110, 112, 114 can include, but are not limited to, RPA robots, portions of an operating system, downloadable applications for corresponding computing systems, any other suitable software and / or hardware, or any combination thereof, without departing from the scope of the present invention. In some embodiments, one or more of the processes 110, 112, 114 can be listeners. A listener can be an RPA robot, a portion of an operating system, a downloadable application for a corresponding computing system, or any other software and / or hardware, without departing from the scope of the present invention. In fact, in some embodiments, the logic of the (multiple) listeners is implemented partially or fully via physical hardware.

[0035] Listeners monitor and record data related to user interactions with respective computing systems and / or operations of unattended computing systems, and send the data to the core hyper-automation system 120 via a network (e.g., local area network (LAN), mobile communication network, satellite communication network, Internet, any combination thereof, etc.). The data can include, but is not limited to, which buttons are clicked, where the mouse is moved, text entered in fields, one window minimized while another window is opened, the application associated with the window, and so on. In certain embodiments, data from the listeners can be sent periodically as part of a heartbeat message. In some embodiments, the data can be sent to the core hyper-automation system 120 once a predetermined amount of data has been collected, after a predetermined period of time has elapsed, or both. One or more servers, such as server 130, receive the data from the listeners and store it in a database such as database 140.

[0036] Automation processes can execute logic developed in a workflow at design time. In the case of RPA, the workflow can include a set of steps defined herein as "activities" that are executed in a sequence or some other logical flow. Each activity can include actions such as clicking a button, reading a file, writing to a log panel, and so on. In some embodiments, workflows can be nested or embedded.

[0037] In some embodiments, long-running workflows in RPA are the main items that support service orchestration, human intervention, and long-running transactions in an unattended environment. For example, see U.S. Patent No. 10,860,905, which is incorporated herein by reference in its entirety. Human intervention comes into play when certain processes require human input to handle exceptions, approvals, or validations before proceeding to the next step of an activity. In such cases, the process execution is paused, thereby releasing the RPA robot until the human task is completed.

[0038] Long-running workflows can support workflow fragmentation via persistent activities and can be combined with call processes and non-user interaction activities, with human tasks coordinated with RPA robot tasks. In some embodiments, multiple or many computing systems can participate in executing the logic of a long-running workflow. Long-running workflows can run in sessions to facilitate rapid execution. In some embodiments, long-running workflows can orchestrate background processes, which can include activities that execute API calls and run within a long-running workflow session. In some embodiments, these activities can be invoked by a call process. A process with user interaction activities running in a user session can be invoked by starting a job from a builder activity (the builder will be described in more detail later in this document). In some embodiments, the user can interact via a task that requires filling out a form in the builder. Activities can include having the RPA robot wait for the form task to complete and then resuming the long-running workflow.

[0039] One or more of the (multiple) automation processes 110, 112, 114 communicate with the core hyper-automation system 120. In some embodiments, the core hyper-automation system 120 can run a builder application on one or more servers (e.g., server 130). Although one server 130 is shown for illustrative purposes, multiple or many servers that are close to each other or in a distributed architecture can be employed without departing from the scope of the present invention. For example, one or more servers can be provided for builder functionality, AI / ML model serving, authentication, governance, and / or any other suitable functionality without departing from the scope of the present invention. In some embodiments, the core hyper-automation system 120 can alternatively be part of a public cloud architecture, private cloud architecture, hybrid cloud architecture, etc. In certain embodiments, the core hyper-automation system 120 can host multiple software-based servers on one or more computing systems (e.g., server 130). In some embodiments, one or more servers of the core hyper-automation system 120, such as server 130, can be implemented via one or more virtual machines (VMs).

[0040] In some embodiments, one or more of the automation processes 110, 112, 114 may call one or more AI / ML models 132 deployed on or accessible by the core hyper-automation system 120 and trained to perform various tasks. For example, the AI / ML models 132 may include models trained to find various application versions, perform CV, perform OCR, UI descriptors, provide suggestions for the next activity or sequence of activities in an RPA workflow, etc. The AI / ML models may be trained using labeled data, which includes but is not limited to elements from data sources (e.g., web pages, forms, scanned documents, application interfaces, screens, etc.), previously created RPA workflows, screenshots of various application screens of various versions and their corresponding UI elements, libraries of UI objects, etc. The AI / ML models 132 may be trained to achieve a desired confidence threshold without overfitting to a given training dataset.

[0041] The AI / ML models 132 may be trained for any suitable purpose without departing from the scope of the invention, which will be discussed in more detail below. In some embodiments, two or more of the AI / ML models 132 may be linked (e.g., in series, in parallel, or a combination thereof) such that they jointly provide a collaborative output. The AI / ML models 132 may perform or assist with CV, OCR, document processing and / or understanding, semantic learning and / or analysis, analytical prediction, process discovery, task mining, testing, automated RPA workflow generation, sequence extraction, cluster detection, audio-to-text translation, any combination thereof, etc. However, any desired number and / or type of AI / ML models may be used without departing from the scope of the invention. For example, using multiple AI / ML models may enable the system to develop a global picture of what is happening on a given computing system. For example, one AI / ML model may perform OCR, another model may detect buttons, and another model may compare sequences, etc. The determination may be made by a single AI / ML model alone or by multiple AI / ML models jointly. In certain embodiments, one or more of the AI / ML models are locally deployed on at least one of the computing systems 102, 104, 106.

[0042] In some embodiments, multiple AI / ML models 132 can be used. Each AI / ML model 132 is an algorithm (or model) that runs on data, and for example, the AI / ML model itself can be a deep learning neural network (DLNN) of trained artificial "neurons" that are trained on training data. In some embodiments, the AI / ML model 132 can have multiple layers that perform various functions, such as statistical modeling (e.g., Hidden Markov Model (HMM)), and utilize deep learning techniques (e.g., Long Short-Term Memory (LSTM) deep learning, encoding of previous hidden states, etc.) to perform the desired functions.

[0043] In some embodiments, the hyper-automation system 100 can provide four main groups of functions: (1) discovery; (2) build automation; (3) management; and (4) engagement. In some embodiments, the automation (e.g., running on a user computing system, server, etc.) can be run by software robots, such as RPA robots. For example, attended robots, unattended robots, and / or test robots can be used. Attended robots work with users to help them complete tasks (e.g., via UiPath Assistant TM ). Unattended robots work independently of users and can run in the background without the users' knowledge. Test robots are unattended robots that are used to run test cases against applications or RPA workflows. In some embodiments, test robots can run in parallel on multiple computing systems.

[0044] The discovery function can discover and provide automated recommendations for different opportunities for business process automation. Such a function can be implemented by one or more servers, such as server 130. In some embodiments, the discovery function can include providing an automation hub, process mining, task mining, and / or task capture. The automation hub (e.g., UiPath automation Hub TM ) can provide a mechanism for managing the automation showcase with visibility and control. For example, ideas for automation can be crowdsourced from employees via a submission form. The feasibility and ROI calculations for automating these ideas can be provided, documentation for future automation can be collected, and collaboration can be provided to move faster from automation discovery to build.

[0045] Process mining (e.g., via UiPath Automation Cloud TM and / or UiPath AI Center TM) refers to the process of collecting and analyzing data from applications (such as enterprise resource planning (ERP) applications, customer relationship management (CRM) applications, email applications, call center applications, etc.) to identify what end-to-end processes exist in an organization, how to effectively automate these processes, and what the impact of the automation will be. For example, this data can be collected by listeners from user computing systems 102, 104, 106 and processed by a server such as server 130. In some embodiments, one or more AI / ML models 132 can be used for this purpose. This information can be output to an automation hub to accelerate implementation and avoid manual information transfer. The goal of process mining can be to increase business value by automating processes within an organization. Some examples of process mining goals include, but are not limited to, increasing profits, improving customer satisfaction, complying with regulations and / or contracts, improving employee efficiency, etc.

[0046] Task mining (e.g., via UiPath Automation Cloud TM and / or UiPath AI Center TM ) identifies and aggregates workflows (such as employee workflows), then applies artificial intelligence to reveal patterns and variations in daily tasks, scores these tasks for ease of automation and potential savings (such as time and / or cost savings). One or more AI / ML models 132 can be employed to reveal repetitive task patterns in the data. Then, repetitive tasks suitable for automation can be identified. In some embodiments, this information can initially be provided by a listener and analyzed on a server of the core hyperautomation system 120, such as server 130. The results of task mining (such as XAML process data) can be output to a process documentation or designer application (such as UiPath Studio TM ) to create and deploy automation more quickly. In some embodiments, task mining can include taking screenshots of user actions (such as mouse click locations, keyboard inputs, application windows and graphical elements the user is interacting with, timestamps of interactions, etc.), collecting statistical data (such as execution times, number of actions, text entries, etc.), editing and annotating screenshots, specifying the types of actions to record, etc.

[0047] Task capture (e.g., via UiPath Automation Cloud TM and / or UiPath AI Center TM ) automatically records the processes involved when a user is working or provides a framework for unattended processes. Such documentation may include operations that capture each part of the process in PDD, skeleton workflows, record user actions, and automatically generate a comprehensive workflow diagram, including each step, Microsoft Details of documents, XAML files, etc. In some embodiments, the build-ready workflow can be directly output to a designer application, such as UiPath Studio TM . Task capture can simplify the requirements gathering process for subject matter experts who interpret processes and members of a Center of Excellence (CoE) that provides production-grade automation.

[0048] Build automation can be achieved via a designer application (e.g., UiPath Studio TM , UiPath StudioX TM or UiPath Studio Web TM ). For example, an RPA developer in the RPA development facility 150 can use the RPA designer application 154 of the computing system 152 to build and test automations for various applications and environments, such as web, mobile, and virtual desktops. API integration can be provided for various applications, technologies, and platforms. Predefined activities, drag-and-drop modeling, and workflow recorders can make automation easier with minimal coding. Document understanding capabilities can be provided via drag-and-drop AI skills for data extraction and interpretation that calls one or more AI / ML models 132. Such automation can handle almost any document type and format, including tables, checkboxes, signatures, and handwriting. When data is validated or exceptions are handled, this information can be used to retrain the corresponding AI / ML models, improving their accuracy over time.

[0049] The RPA designer application 152 can be designed to call one or more trained AI / ML models 132 on the server 130 and / or a generative AI model 172 in a cloud environment via a network 120 (e.g., local area network (LAN), mobile communication network, satellite communication network, internet, any combination thereof, etc.) to assist in the RPA automation development process. In some embodiments, one or more AI / ML models can be encapsulated with the RPA designer application 152 or otherwise stored on the local computing system 150.

[0050] In some embodiments, one or more of the RPA Designer application 152 and the AI / ML models 132 may be configured to use an object repository stored in the database 140. For example, see U.S. Patent 11,748,069, the entire content of which is incorporated herein by reference. The object repository may include a UI object library, and these UI objects may be used to develop RPA workflows via the RPA Designer application 152. The object repository may be used to add UI descriptors to activities in the workflows of the RPA Designer application 152 for UI automation. In some embodiments, one or more of the AI / ML models 132 may generate new UI descriptors and add them to the object repository in the database 140. Once automation is completed in the Designer application 152, they may be published on the server 130 and rolled out to the computing systems 102, 104, 106, etc.

[0051] For example, an integration service may allow developers to seamlessly combine UI automation with API automation. For example, an automation that requires an API or traverses API and non-API applications and systems may be built. A repository (e.g., UiPath Object repository TM ) or a marketplace (e.g., UiPath marketplace TM ) may be provided to allow developers to automate various processes more quickly. Thus, when building an automation, the hyperautomation system 100 may provide a user interface, a development environment, API integration, pre-built and / or customized AI / ML models, development templates, an integrated development environment (IDE), and advanced AI capabilities. In some embodiments, the hyperautomation system 100 implements the development, deployment, management, configuration, monitoring, debugging, and maintenance of RPA robots, which may provide automation for the hyperautomation system 100.

[0052] In some embodiments, components of the hyperautomation system 100, such as the (multiple) designer applications and / or external rule engines, provide support for managing and enforcing governance policies to control the various functions provided by the hyperautomation system 100. Governance is the ability of an organization to establish policies to prevent users from developing automation (e.g., RPA robots) that can take actions that may harm the organization, such as violating the EU General Data Protection Regulation (GDPR), the US Health Insurance Portability and Accountability Act (HIPAA), the terms of service of third-party applications, etc. Since developers may create automation that violates privacy laws, terms of service, etc. when executing their automation, some embodiments implement access control and governance restrictions at the robot and / or robot design application levels. In some embodiments, this can provide an additional level of security and compliance to the automation process development pipeline by preventing developers from relying on unapproved software libraries that may introduce security risks or operate in a manner that violates policies, regulations, privacy laws, and / or privacy policies. For example, see U.S. Patent 11733668, the entire content of which is incorporated herein by reference.

[0053] The management function can provide automated management, deployment, and optimization across the organization. In some embodiments, the management function can include orchestration, test management, AI capabilities, and / or insights. The management function of the hyperautomation system 100 can also act as an integration point for third-party solutions and applications with the automation applications and / or RPA robots. The management capabilities of the hyperautomation system 100 can include, but are not limited to, facilitating the provisioning, deployment, configuration, queuing, monitoring, logging, and interconnection of RPA robots, etc.

[0054] Builder applications, such as UiPath Orchestrator TM (In certain embodiments, it can be provided as part of UiPathAutomation Cloud TM or locally, in virtual machines, in private or public clouds, in Linux TM virtual machines, or via the UiPath automation suite TM provided as a cloud-native single-container suite), provide orchestration capabilities for deploying, monitoring, optimizing, scaling, and securing the deployment of RPA robots. The test suite (e.g., UiPath test suite TM ) can provide test management to monitor the quality of the deployed automation. The test suite can facilitate test planning and execution, requirements fulfillment, and defect traceability. The test suite may include comprehensive test reports.

[0055] Analytics software (e.g., UiPath Insights TM)It can track, measure, and manage the performance of deployed automation. The analytics software can align automated operations with the organization's specific key performance indicators (KPIs) and strategic outcomes. The analytics software can present the results in a dashboard format for better understanding by human users.

[0056] For example, data services (e.g., UiPath data service TM ) can be stored in database 140 and bring the data to a single scalable and secure place using a drag-and-drop storage interface. Some embodiments can provide low-code or no-code data modeling and storage for automation while ensuring seamless access to data, enterprise-level security, and scalability. The artificial intelligence capabilities can be provided by an artificial intelligence hub (e.g., UiPath AI center TM ) which helps incorporate AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options can make these capabilities accessible even to non-data scientists. The deployed automation (e.g., RPA robots) can call AI / ML models, such as AI / ML module 132, from the AI hub. The performance of the AI / ML models can be monitored, trained, and improved using human-verified data such as that provided by data review center 160. Human reviewers can provide labeled data to the core hyperautomation system 120 via the review application 152 on the computing system 154. For example, human reviewers can verify that the predictions of the AI / ML model 132 and / or generative AI model 172 are accurate or otherwise provide corrections. This dynamic input can then be saved as training data for retraining the AI / ML model 132 and / or generative AI model 172 and can be stored in a database such as database 140. The AI hub can then schedule and execute training jobs to train a new version of the AI / ML model using the training data. Both positive and negative examples can be stored and used for retraining the AI / ML model 132 and / or generative AI model 172.

[0057] The engagement function enables seamless collaboration between humans and automation as a team in the desired process. Low-code applications (e.g., via UiPath Apps TM ) can be built to connect browser tabs and legacy software even in some embodiments where APIs are lacking. For example, a web browser can be used to quickly create an application through a rich library of drag-and-drop controls. The application can be connected to a single automation or multiple automations.

[0058] The action center (e.g., UiPath action center TM) provides a simple and effective mechanism to hand off the automation process to humans and vice versa. Humans can provide approvals or escalations, handle exceptions, etc. Then, the automation can perform the automated functions of a given workflow.

[0059] The local assistant can serve as a launchpad for users to initiate automation (e.g., UiPath assistant TM ). For example, this functionality can be provided in a tray offered by the operating system and can allow users to interact with RPA robots and RPA robot-driven applications on their computing systems. The interface can list the automations approved for a given user and allow the user to run them. These can include off-the-shelf automations from an automation marketplace, an automation store within the automation hub, etc. When the automations run, they can run in parallel as local instances with other processes on the computing system, so that the user can use the computing system while the automation performs its actions. In some embodiments, the assistant is integrated with a task capture feature so that users can record the processes they are about to automate from the assistant launchpad.

[0060] Chatbots (e.g., UiPath Chatbots TM ), social messaging apps, and / or voice commands can enable users to run automations. This can simplify access to the information, tools, and resources needed for users to interact with customers or perform other activities. As with other processes, conversations between people can potentially be automated. RPA robots triggered in this way can potentially execute using plain language commands for operations such as checking order status, posting data in the CRM, etc.

[0061] In some embodiments, the hyperautomation system 100 can provide end-to-end measurement and management of automation programs of any scale. According to the above, analytics can be used to understand the performance of the automation (e.g., via UiPath Insights TM ). Data modeling and analysis using any combination of available business metrics and operational insights can be used for various automation processes. Custom-designed and pre-built dashboards allow visualizing data across desired metrics, discovering new analytical insights, tracking performance metrics, discovering the ROI of the automation, performing telemetry monitoring on the user computing system, detecting errors and exceptions, and debugging the automation. An automation management console (e.g., UiPath automation Ops TM ) can be provided to manage the automation throughout its lifecycle. Organizations can manage how the automations are built, what users can do with them, and which automations users can access.

[0062] In some embodiments, the hyper-automation system 100 provides an iterative platform. Processes can be discovered, automation can be built, tested, and deployed, performance can be measured, the use of automation can be readily provided to users, feedback can be obtained, AI / ML models can be trained and retrained, and the process can repeat itself. This facilitates the implementation of a more robust and efficient automation suite.

[0063] In some embodiments, generative AI models are used. Generative AI can generate various types of content, such as text, images, audio, and synthetic data. Various types of generative AI models can be used, including but not limited to large language models (LLMs), generative adversarial networks (GANs), variational autoencoders (VAEs), transformers, etc. These models can be part of the AI / ML model 132 hosted on the server 130. For example, a generative AI model can be trained on a large corpus of text information to perform semantic understanding, understand the nature of the content present on a screen from text, automatically generate code, and so on. In certain embodiments, a generative AI model 172 provided by an existing cloud ML service provider (such as etc.) can be used and trained to provide such functionality. In generative AI embodiments where the (multiple) generative AI models 172 are remotely hosted, the server 130 can be configured to integrate with a third-party API that allows the server 130 to send requests including the necessary input information to the generative AI model and receive a response in return (e.g., semantic matching of fields between application versions, classification of the type of application on a screen, etc.). Such embodiments can provide a more advanced and refined user experience and provide access to state-of-the-art natural language processing (NLP) and other ML capabilities offered by these companies.

[0064] In some embodiments, one aspect of generative AI models is the use of transfer learning. In transfer learning, a pre-trained generative AI model, such as an LLM, is fine-tuned for a specific task or domain. This enables the LLM to leverage the knowledge it has already learned during its initial training and adapt it to a specific application. In the case of an LLM, the pre-training phase involves training the LLM on a large corpus of text typically consisting of billions of words. During this phase, the LLM learns the relationships between words and phrases, which enables the LLM to generate coherent, human-like responses to text-based inputs. The output of this pre-training phase is an LLM with a high-level understanding of the underlying patterns in natural language.

[0065] During the fine-tuning phase, the pre-trained LLM is adapted to a specific task or domain by training it on a smaller task-specific dataset. For example, in some embodiments, the LLM can be trained to analyze one or more types of data sources to improve its accuracy regarding their content. Such information can be provided as part of the training data, and the LLM can learn to focus on these areas and more accurately identify data elements therein. Fine-tuning allows the LLM to learn the nuances of the task or domain, such as the specific vocabulary and grammar used in the domain, without requiring as much data as would be needed to train the LLM from scratch. By leveraging the knowledge learned during the pre-training phase, the fine-tuned LLM can achieve state-of-the-art performance on a specific task with a relatively small amount of training data.

[0066] The LLM can be trained using a vector database. The vector database indexes, stores, and provides access to structured or unstructured data (e.g., text, images, time series data, etc.) and their vector embeddings. Data such as text can be tokenized, where individual letters, words, or sequences of words are parsed from the text into tokens. These tokens are then "embedded" into vector embeddings, which are numerical representations of the data. The vector database allows users to quickly and at scale find and retrieve similar objects in a production environment.

[0067] AI and ML allow for the digital representation of unstructured data without losing its semantics in the vector embedding. A vector embedding is a long string of numbers, each number describing a feature of the data object that the vector embedding represents. Similar objects are grouped together in the vector space. In other words, the more similar the objects, the closer the vector embeddings representing the objects will be. Similar objects can be found using vector search, similarity search, or semantic search. The distance between vector embeddings can be calculated using various techniques, including but not limited to squared Euclidean or L2 squared distance, Manhattan or L1 distance, cosine similarity, dot product, Hamming distance, etc. It may be beneficial to choose the same metric used to train the AI / ML model.

[0068] Vector indexes can be used to organize vector embeddings for efficient data retrieval. If there are a large number of data points, it can be computationally expensive to calculate the distance between a vector embedding and all other vector embeddings in a vector database using the k-nearest neighbor (kNN) algorithm because the required computations increase linearly with the dimensionality and number of data points (O(n)). It is more efficient to use approximate nearest neighbor (ANN) methods to find similar objects. The distances between vector embeddings are pre-computed, and similar vectors are organized and stored close to each other (e.g., in clusters or graphs), allowing similar objects to be found more quickly. This process is called "vector indexing". In some embodiments, ANN algorithms that can be used include, but are not limited to, cluster-based indexing, proximity graph-based indexing, tree-based indexing, hash-based indexing, compression-based indexing, etc.

[0069] Figure 2 is an architectural diagram of the RPA system 200 according to an embodiment of the present invention. In some embodiments, the RPA system 200 is Figure 1 part of the hyper-automation system 100. The RPA system 200 includes a designer 210 that allows developers to design and implement workflows. The designer 210 can provide solutions for application integration as well as automating third-party applications, managing information technology (IT) tasks, and business IT processes. The designer 210 can facilitate the development of automation projects, which are graphical representations of business processes. In short, the designer 210 facilitates the development and deployment of workflows and robots. In some embodiments, the designer 210 can be an application running on a user's desktop, an application running remotely in a VM, a web application, etc.

[0070] Automation projects enable rule-based process automation by giving developers control over the execution order and the relationships between the set of custom steps developed in a workflow, which are defined as "activities" herein according to the above definition. A business example of an embodiment of the designer 210 is UiPath Studio TM . Each activity can include actions such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows can be nested or embedded.

[0071] Certain types of workflows can include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suitable for linear processes, enabling the flow from one activity to another without cluttering the workflow. Flowcharts may be particularly suitable for more complex business logic, supporting the integration of decisions and connecting activities in more diverse ways through multiple branching logical operators. FSMs may be particularly suitable for large workflows. FSMs can use a finite number of states in their execution, which are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suitable for determining workflow behavior and debugging processes when execution errors are encountered.

[0072] Once a workflow is developed in the designer 210, the execution of the business process is orchestrated by the builder 220, which orchestrates the execution of one or more robots 230 that develop the workflow in the designer 210. One business example of an embodiment of the builder 220 is UiPath Orchestrator TM . The builder 220 facilitates the creation, monitoring, and deployment of resources in the management environment. The builder 220 can act as an integration point with third - party solutions and applications. According to the above, in some embodiments, the builder 220 can be Figure 1 part of the core hyper - automation system 120.

[0073] The builder 220 can manage the robot 230 queue, connecting and executing robots 230 from a centralized point. The types of robots 230 that can be managed include, but are not limited to, attended robots 232, unattended robots 234, development robots (similar to unattended robots 234 but for development and testing purposes), and non - production robots (similar to attended robots 232 but for research, development, and testing purposes). Attended robots 232 are triggered by user events and operate on the same computing system as humans. Attended robots 232 can be used with the builder 220 for centralized deployment and recording media. Attended robots 232 can assist human users in completing various tasks and can be triggered by user events. In some embodiments, a process cannot be started from the builder 220 on this type of robot, and / or they cannot run under a locked screen. In certain embodiments, attended robots 232 can only be started from the robot tray or command prompt. In some embodiments, attended robots 232 should run under human supervision.

[0074] The unattended robot 234 operates unattended in a virtual environment and is capable of automating multiple processes. The unattended robot 234 can be responsible for remote execution, monitoring, scheduling, and providing support for work queues. In some embodiments, debugging of all robot types can be run in the designer 210. Both attended and unattended robots can automate various systems and applications, including but not limited to mainframes, web applications, VMs, enterprise applications (e.g., generated by etc.) and computing system applications (such as desktop and laptop computer applications, mobile device applications, wearable computer applications, etc.).

[0075] The builder 220 can have various capabilities, including but not limited to provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning can include creating and maintaining a connection between the robot 230 and the builder 220 (e.g., a web application). Deployment can include delivering the correct software package version to the assigned robot 230 for execution. Configuration can include the maintenance and delivery of the robot environment and process configurations. Queuing can include providing management of queues and queue items. Monitoring can include tracking robot identification data and maintaining user permissions. Logging can include storing and indexing logs into a database (e.g., a Structured Query Language (SQL) database or a "Not only" SQL (NoSQL) database) and / or another storage mechanism (e.g., which provides the ability to store and quickly query large datasets). The builder 220 can provide interconnectivity by acting as a centralized communication point for third-party solutions and / or applications.

[0076] The robot 230 is an execution agent for implementing the workflows built in the designer 210. A business example of some embodiments of the robot 230 is UiPath Robots TM . In some embodiments, the robot 230 installs the Microsoft Service Control Manager (SCM) - management service by default. Thus, such a robot 230 can open an interactive session under the local system account and has permissions for the service.

[0077] In some embodiments, the robot 230 can be installed in user mode. For such a robot 230, this means they have the same rights as the user on whose machine the given robot 230 is installed. This feature can also be used for high-density (HD) robots to ensure maximum utilization of each machine's potential. In some embodiments, any type of robot 230 can be configured in an HD environment.

[0078] In some embodiments, the robot 230 is divided into several components, each dedicated to a specific automation task. In some embodiments, the robot components include, but are not limited to, the SCM-managed robot service, the user-mode robot service, the actuator, the agent, and the command line. The SCM-managed robot service manages and monitors and acts as an agent between the builder 220 and the execution host (i.e., the computing system on which the robot 230 executes). These services are trusted by the robot 230 and manage its certificates. The console application is launched by the SCM under the local system.

[0079] In some embodiments, the user-mode robot service manages and monitors sessions and acts as an agent between the builder 220 and the execution host. The user-mode robot service can be trusted by the robot 230 and manage its certificates. If the SCM-managed robot service is not installed, the application can be automatically launched.

[0080] The actuator can run a given job session (i.e., they can execute a workflow) under . The actuator may know the dots per inch (DPI) setting of each monitor. The agent can be a Presentation Foundation (WPF) application that displays available jobs in the system taskbar window. The agent can be a client of the service. The agent may request to start or stop a job and change settings. The command line is a client of the service. The command line is a console application that can request to start a job and wait for its output.

[0081] As described above, splitting the components of the robot 230 helps developers, support users, and computing systems more easily run, identify, and track what each component is doing. Special behaviors can be configured for each component in this way, such as setting different firewall rules for the actuator and the service. In some embodiments, the actuator can always know the DPI setting of each monitor. Therefore, the workflow can be executed at any DPI and is independent of the configuration of the computing system on which the workflow is created. In some embodiments, the projects from the designer 210 can also be independent of the browser zoom level. For applications that do not know the DPI or intentionally mark the DPI as unknown, the DPI can be disabled in some embodiments.

[0082] The RPA system 200 in this embodiment is part of a hyper-automation system. Developers can use the designer 210 to build and test RPA bots that utilize AI / ML models deployed in the core hyper-automation system 240 (e.g., as part of its AI hub). Such RPA bots can send inputs for the execution of the AI / ML models and receive outputs therefrom via the core hyper-automation system 240.

[0083] As described above, one or more bots 230 can be listeners. These listeners can provide information to the core hyper-automation system 240 about what the user is doing while using their computing system. The core hyper-automation system can then use this information for process mining, task mining, task capture, etc.

[0084] An assistant / chatbot 250 can be provided on the user computing system to allow the user to launch an RPA local bot. For example, the assistant can be located in the system tray. The chatbot can have a user interface so that the user can see text in the chatbot. Or, the chatbot may not have a user interface but run in the background, listening to the user's voice using the computing system's microphone.

[0085] In some embodiments, data tagging can be performed by the user of the computing system on which the bot executes or on another computing system to which the bot provides information. For example, if a bot invokes an AI / ML model that performs CV on an image of a VM user, but the AI / ML model does not correctly identify a button on the screen, the user can draw a rectangle around the mis-identified or un-identified component and may provide text with the correct identification. This information can be provided to the core hyper-automation system 240 and then used later to train a new version of the AI / ML model.

[0086] Figure 3 is an architecture diagram showing the deployment of an RPA system 300 according to an embodiment of the present invention. In some embodiments, the RPA system 300 can be Figure 2 part of the RPA system 200 and / or Figure 1 part of the hyper-automation system 100. The deployed RPA system 300 can be an enterprise-level, user-level, or device-level automation solution based on a cloud system, on-premises system, desktop system, etc., providing automation for different computing processes.

[0087] It should be noted that the client, the server, or both can include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robotic application 310 includes an actuator 312, an agent 314, and a designer 316. However, in some embodiments, the designer 316 may not run on the same computing system as the actuator 312 and the agent 314. The actuator 312 is in the running process. Several business items can run simultaneously, such as Figure 3 as shown. The agent 314 (e.g., Figure 2 a service) is the single point of contact for all actuators 312 in this embodiment. All messages in this embodiment are logged into the builder 340, and the builder 340 also processes them via the database server 350, the AI / ML server 360, the indexer server 370, or any combination thereof. As discussed above regarding

[0088] Server2012), multiple robots can run simultaneously, each using a unique username to run in a separate session. This is referred to as an HD robot above.

[0089] The agent 314 is also responsible for sending the status of the robot (e.g., periodically sending a "heartbeat" message indicating that the robot is still running), and downloading the required version of the package to be executed. In some embodiments, the communication between the agent 314 and the builder 340 is always initiated by the agent 314. In a notification scenario, the agent 314 can open a WebSocket channel, which is later used by the builder 340 to send commands (e.g., start, stop, etc.) to the robot.

[0090] The listener 330 monitors and records data related to the user's interaction with the attended computing system and / or the operation of the unattended computing system where the listener 330 is located. The listener 330 can be an RPA robot, a part of the operating system, a downloadable application for the corresponding computing system, or any other software and / or hardware without departing from the scope of the present invention. In fact, in some embodiments, the logical part of the listener is implemented partially or fully via physical hardware.

[0091] ​On the server side, it includes a presentation layer (web application 342, Open Data Protocol (oData), Representational State Transfer (REST), Application Programming Interface (API) endpoints 344, and notification and monitoring 346), a service layer (API implementation / business logic 348), and a persistence layer (database server 350, AI / ML server 360, and indexer server 370). Builder 340 includes web application 342, oData REST API endpoints 344, notification and monitoring 346, and API implementation / business logic 348. In some embodiments, most of the actions performed by the user in the interface of builder 340 (e.g., via browser 320) are executed by calling various APIs. Such actions may include, but are not limited to, starting a job on a robot, adding / removing data in a queue, scheduling a job to run unattended, etc. without departing from the scope of the present invention. Web application 342 is the visual layer of the server platform. In this embodiment, web application 342 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the present invention. In this embodiment, the user interacts with the web pages from web application 342 via browser 320 to perform various actions to control builder 340. For example, the user can create a group of robots, assign packages to robots, analyze the logs of each robot and / or each process, start and stop robots, etc.

[0092] In addition to web application 342, builder 340 also includes a service layer that exposes oData REST API endpoints 344. However, other endpoints may be included without departing from the scope of the present invention. The REST API is used by both web application 342 and proxy 314. In this embodiment, proxy 314 is a manager for one or more robots on the client computer.

[0093] The REST API in this embodiment includes configuration, logging, monitoring, and queuing functions. In some embodiments, the configuration endpoints can be used to define and configure application users, permissions, robots, assets, releases, and environments. The logging REST endpoints can be used to log different information, such as errors, explicit messages sent by robots, and other environment-specific information. If the start job command is used in builder 340, the robot can use the deployment REST endpoint to query the software package version that should be executed. The queuing REST endpoints may be responsible for queue and queue item management, such as adding data to the queue, retrieving transactions from the queue, setting the status of transactions, etc.

[0094] The monitoring REST endpoints can monitor the web application 342 and the agent 314. The notification and monitoring API 346 can be a REST endpoint for registering the agent 314, passing configuration settings to the agent 314, and for sending / receiving notifications from the server and the agent 314. In some embodiments, the notification and monitoring API 346 can also use WebSocket communication.

[0095] In some embodiments, the APIs in the service layer can be accessed by configuring appropriate API access paths, e.g., based on whether the builder 340 and the overall hyper-automation system have an on-premises deployment type or a cloud-based deployment type. The APIs for the builder 340 can provide customized methods for querying statistics about the various entities registered in the builder 340. In some embodiments, each logical resource can be an oData entity. In such an entity, components such as robots, processes, queues, etc. can have properties, relationships, and operations. In some embodiments, the web application 342 and / or the agent 314 can use the APIs of the builder 340 in two ways: by obtaining API access information from the builder 340, or by registering an external application to use the oAuth flow.

[0096] In this embodiment, the persistence layer includes three servers - a database server 350 (e.g., an SQL server), an AI / ML server 360 (e.g., a server providing AI / ML model services, e.g., the AI hub function), and an indexer server 370. The database server 350 in this embodiment stores configurations such as robots, robot groups, associated processes, users, roles, schedules, etc. In some embodiments, this information is managed through the web application 342. The database server 350 can manage queues and queue items. In some embodiments, the database server 350 can store the messages recorded by the robots (in addition to or instead of the indexer server 370). The database server 350 can also store, for example, process mining, task mining, and / or task capture-related data received from the listener 330 installed on the client. Although no arrow is shown between the listener 330 and the database 350, it should be understood that in some embodiments, the listener 330 is capable of communicating with the database 350 and vice versa. This data can be stored in the form of PDD, images, XAML files, etc. The listener 330 can be configured to intercept user actions, processes, tasks, and performance metrics on the respective computing systems where the listener 330 is located. For example, the listener 330 can record user actions (e.g., clicks, typed characters, locations, applications, active elements, time, etc.) on its corresponding computing system, and then convert these actions into a suitable format for providing and storing in the database server 350.

[0097] The AI / ML Server 360 facilitates the incorporation of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options enable access to these capabilities even for those who are not data scientists. Deployment automation (e.g., RPA robots) can call AI / ML models from the AI / ML Server 360. The performance of the AI / Machine Learning models can be monitored and trained and improved using human-verified data. The AI / ML Server 360 can schedule and execute training jobs to train new versions of the AI / ML models.

[0098] The AI / ML Server 360 can store data related to AI / ML models and ML packages for configuring various ML skills for users during development. As used herein, an ML skill is an ML model pre-built and trained for a process, e.g., which can be used by automation. The AI / ML Server 360 can also store data related to document understanding techniques and frameworks, algorithms, and software packages for various AI / ML capabilities, including but not limited to intent analysis, NLP, speech analysis, different types of AI / ML models, etc.

[0099] The Indexer Server 370 is optional in some embodiments and stores and indexes information logged by robots. In certain embodiments, the Indexer Server 370 can be disabled via a configuration setting. In some embodiments, the Indexer Server 370 uses This is an open-source project full-text search engine. Messages logged by robots (e.g., activities such as using log messages or write lines) can be sent to the Indexer Server 370 via a log REST endpoint where they are indexed for future use.

[0100] Figure 4 is an architectural diagram that illustrates the relationship 400 between the Designer 410, Activities 420, 430, 440, 450, Driver 460, API 470, and AI / ML Model 480 in accordance with an embodiment of the present invention. As described above, developers use the Designer 410 to develop workflows to be executed by robots. In some embodiments, various types of activities can be presented to the developer. The Designer 410 can be a user's local or remote computing system (e.g., accessed via a VM or a local web browser interacting with a remote web server). The workflow can include user-defined activities 420, API-driven activities 430, AI / ML activities 440, and / or UI automation activities 450. The user-defined activities 420 and API-driven activities 440 interact with applications via their APIs. In some embodiments, the user-defined activities 420 and / or AI / ML activities 440 can call one or more AI / ML models 480, which can be local and / or remote to the computing system on which the robot operates.

[0101] Some embodiments are capable of identifying non-text visual components in an image, referred to herein as CVs. However, it should be noted that in some embodiments, CVs incorporate OCR. CVs can be performed at least in part by the AI / ML model 480. Some CV activities associated with these components can include, but are not limited to, using OCR to extract text from segmented labeled data, fuzzy text matching, using ML to crop segmented labeled data, comparing the text extracted from the labeled data with ground truth data, etc. In some embodiments, hundreds or even thousands of activities can be implemented in the user-defined activity 420. However, any number and / or type of activities can be used without departing from the scope of the present invention.

[0102] UI automation activities 450 are a subset of special lower-level activities written in a lower-level code that facilitate interaction with the screen. UI automation activities 450 facilitate these interactions that allow a robot to interact with the desired software via the driver 460. For example, in some embodiments, the driver 460 can include an operating system (OS) driver 462, a browser driver 464, a VM driver 466, an enterprise application driver 468, etc. UI automation activities 450 can use one or more of the AI / ML models 480 in order to perform interactions with the computing system. In certain embodiments, the AI / ML models 480 can augment the driver 460 or replace them entirely. In fact, in certain embodiments, the driver 460 is not included.

[0103] The driver 460 can interact with the operating system at a low level via the operating system driver 462, looking for hooks, monitoring keys, etc. The driver 460 can facilitate integration with etc. For example, the "click" activity performs the same role in these different applications via the driver 460.

[0104] Figure 5 is an architectural diagram of a computing system 500 configured to perform automatic annotation and technical specification generation of an RPA workflow using AI according to an embodiment of the present invention. In some embodiments, the computing system 500 can be one or more of the computing systems depicted and / or described herein. In certain embodiments, the computing system 500 can be part of a hyper-automation system, such as Figure 1 and Figure 2As shown. Computing system 500 includes a bus 505 or other communication mechanism for communicating information, and (multiple) processors 510 coupled to the bus 505 for processing information. The (multiple) processors 510 can be any type of general-purpose or special-purpose processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The (multiple) processors 510 can also have multiple processing cores, and at least some of the cores can be configured to perform specific functions. Multiple parallel processing can be used in some embodiments. In certain embodiments, at least one of the (multiple) processors 510 can be a neuromorphic circuit that includes processing elements that simulate biological neurons. In some embodiments, the neuromorphic circuit may not require the typical components of a von Neumann computing architecture.

[0105] Computing system 500 also includes a memory 515 for storing information and instructions executed by the (multiple) processors 510. The memory 515 can be composed of random access memory (RAM), read-only memory (ROM), flash memory, cache, static memory such as a magnetic disk or optical disk, or any other type of non-transitory computer-readable medium or any combination thereof. The non-transitory computer-readable medium can be any available medium accessible by the (multiple) processors 510, and can include volatile media, non-volatile media, or both. The medium can also be removable, non-removable, or both. Computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communication network via wireless and / or wired connections. In some embodiments, the communication device 520 can include one or more antennas without departing from the scope of the present invention, and these antennas can be single, arrayed, phased, switched, beamforming, beam steering, combinations thereof, and / or any other antenna configuration.

[0106] (Multiple) processors 510 are also coupled to a display 525 via the bus 505. Any suitable display device and tactile I / O can be used. A keyboard 530 and a cursor control device 535, such as a computer mouse, a touchpad, etc., without departing from the scope of the present invention, are also coupled to the bus 505 to enable a user to interface with the computing system 500. However, in certain embodiments, there may be no physical keyboard and mouse, and the user can interact with the device only through the display 525 and / or a touchpad (not shown). Any type and combination of input devices can be used as a design choice. In certain embodiments, there is no physical input device and / or display. For example, a user can interact remotely with the computing system 500 via another computing system communicating with it, or the computing system 500 can operate autonomously.

[0107] The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. These modules include an operating system 540 for the computing system 500. The modules also include an auto-generation module 545 that is configured to perform all or part of the AI / ML processes described herein or derivatives thereof. The computing system 500 may include one or more additional function modules 550 that include additional functionality.

[0108] Those skilled in the art will understand that, without departing from the scope of the present invention, a "system" can be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a smart watch, a quantum computing system, or any other suitable computing device or combination of devices. Presenting the above functions as being performed by a "system" is not intended to limit the scope of the present invention in any way, but rather to provide an example of many embodiments of the present invention. In fact, the methods, systems, and devices disclosed herein can be implemented in localized and distributed forms consistent with computing technologies, including cloud computing systems. The computing system can be part of a LAN, a mobile communication network, a satellite communication network, the Internet, a public or private cloud, a hybrid cloud, a server farm, any combination thereof, etc., or otherwise accessible. Any localized or distributed architecture can be used without departing from the scope of the present invention.

[0109] It should be noted that some system functions described in this specification have been presented as modules to more particularly emphasize their implementation independence. For example, a module can be implemented as a hardware circuit that includes custom very large scale integration (VLSI) or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module can also be implemented in a programmable hardware device such as a field programmable gate array, a programmable array logic, a programmable logic device, a graphics processing unit, etc.

[0110] A module can also be implemented at least partially in software for execution by various types of processors. For example, the identified executable code units can include one or more physical or logical blocks of computer instructions. For example, the computer instructions can be organized as objects, procedures, or functions. However, the executable programs of the identified modules do not need to be physically located together, but rather can include different instructions stored in different locations that, when logically connected together, include the module and implement the stated purpose of the module. In addition, a module can be stored on a computer-readable medium without departing from the scope of the present invention, which can be, for example, a hard disk drive, a flash device, a RAM, a magnetic tape, and / or any other such non-transitory computer-readable medium for storing data.

[0111] In fact, an executable code module can be a single instruction, or multiple instructions, and can even be distributed across several different code segments, different programs, and across several memory devices. Similarly, operational data can be identified and shown within the module, and can be embodied in any suitable form and organized within any suitable type of data structure. The operational data can be collected as a single data set, or can be distributed across different locations, including different storage devices, and can exist at least partially only as electronic signals on a system or network.

[0112] Various types of AI / ML models can be trained and deployed without departing from the scope of the present invention. For example, Figure 6A FIG. 600 shows an example of a neural network 600 according to an embodiment of the present invention, which has been trained to use AI to supplement the automatic annotation and technical specification generation of an RPA workflow. The neural network 600 includes a plurality of hidden layers. Both deep learning neural networks (DLNNs) and shallow learning neural networks (SLNNs) typically have multiple layers, although in some cases, an (SLNN) may have only one or two layers and generally fewer than a DLNN. Generally, a neural network architecture includes an input layer, a plurality of intermediate layers, and an output layer, as is the case with the neural network 600.

[0113] A DLNN typically has many layers (e.g., 10, 50, 200, etc.), and subsequent layers typically reuse the features of the previous layer to compute more complex general functions. On the other hand, an SLNN tends to have only a few layers and is relatively fast to train because the specialized features are created in advance from the raw data samples. However, feature extraction is laborious. On the other hand, a DLNN typically does not require specialized features but tends to take longer to train and has more layers.

[0114] For both methods, these layers are trained simultaneously on a training set, and overfitting is typically checked on an isolated cross-validation set. Both techniques can produce excellent results, and there is considerable enthusiasm for both methods. The optimal size, shape, and number of each layer vary according to the problem solved by each neural network.

[0115] Returning to Figure 6A, RPA workflows, PDDs, and / or other documents, UI screenshots, RPA workflows, and annotations, etc. are provided as the input layer and fed as input into the J neurons of hidden layer 1. Various other inputs are possible, including but not limited to computing system status information, published automations, business rules, information about RPA workflows and / or tasks, initial qualifications of automations, process automation documents, etc. Although all these inputs are fed into each neuron in this example, various architectures are possible, either alone or in combination, including but not limited to feedforward networks, radial basis networks, deep feedforward networks, deep convolutional inverse graphics networks, convolutional neural networks, recurrent neural networks, artificial neural networks, long / short-term memory networks, gated recurrent unit networks, generative adversarial networks, liquid state machines, autoencoders, variational autoencoders, denoising autoencoders, sparse autoencoders, extreme learning machines, echo state networks, Markov chains, Hopfield networks, Boltzmann machines, restricted Boltzmann machines, deep residual networks, Kohonen networks, deep belief networks, deep convolutional networks, support vector machines, neural Turing machines, or any other suitable type or combination of neural networks, without departing from the scope of the present invention.

[0116] Hidden layer 2 receives input from hidden layer 1, hidden layer 3 receives input from hidden layer 2, and so on, until the last hidden layer provides its output as the input to the output layer. Although multiple suggestions are shown as outputs here, in some embodiments, only a single output suggestion is provided. In certain embodiments, the suggestions are ranked based on confidence scores.

[0117] It should be noted that the numbers of neurons I, J, K, and L are not necessarily equal. Thus, any desired number of layers can be used for a given layer of neural network 600 without departing from the scope of the present invention. In fact, in certain embodiments, the types of neurons in a given layer may not all be the same.

[0118] The neural network 600 is trained to assign (a) confidence score(s) to appropriate outputs. To reduce inaccurate predictions, in some embodiments, only those results with confidence scores that meet or exceed a confidence threshold may be provided. For example, if the confidence threshold is 80%, outputs with confidence scores exceeding that amount may be used, and the rest may be ignored.

[0119] A neural network is a probabilistic structure that typically has a confidence score. This can be a score learned by an AI / ML model based on the frequency of similar inputs correctly identified during training. Some common types of confidence scores include decimal numbers between 0 and 1 (which can also be interpreted as a confidence percentage), numbers between negative infinity and positive infinity, sets of expressions (e.g., "low", "medium", and "high"), etc. To obtain a more accurate confidence score, various post-processing calibration techniques can also be employed, such as temperature scaling, batch normalization, weight decay, negative log-likelihood (NLL), etc.

[0120] "Neurons" in a neural network are algorithmically implemented as mathematical functions, typically based on the functionality of biological neurons. Neurons receive weighted inputs and have a summation and activation function that controls whether they pass the output to the next layer. This activation function can be a non-linear threshold activation function, where nothing happens if the value is below the threshold, but above the threshold the function then responds linearly (i.e., the rectified linear unit (ReLU) non-linearity). The summation function and the ReLU function are used in deep learning because real neurons can have approximately similar activity functions. Via linear transformation, information can be subtracted, added, etc. Essentially, the neuron acts as a gating function, passing the output to the next layer, controlled by its underlying mathematical function. In some embodiments, at least some neurons can use different functions.

[0121] In Figure 6B an example of a neuron 610 is shown. Inputs x1, x2, …, x n from the previous layer are assigned corresponding weights w1, w2, …, w n . Thus, the collective input from the previous neuron 1 is w1x1. These weighted inputs are used in the summation function of the neuron modified by a bias, e.g.:

[0122]

[0123] This sum is compared with the activation function f(x) to determine whether the neuron "fires". For example, f(x) can be given by:

[0124]

[0125] Thus, the output y of the neuron 610 can be given by:

[0126]

[0127] In this case, neuron 610 is a single-layer perceptron. However, any suitable type of neuron or combination of neuron types may be used without departing from the scope of the present invention. It should also be noted that in some embodiments, the weights of the activation function and / or the range of the (multiple) output values may vary without departing from the scope of the present invention.

[0128] A target or "reward function" is typically employed. The reward function explores intermediate transitions and steps with short-term and long-term rewards to guide the search of the state space and attempts to achieve the goal (e.g., finding the most accurate answer to a user query based on relevant metrics). During training, various labeled data are fed through the neural network 600. Successful identifications enhance the weights of the neuron inputs, while unsuccessful identifications weaken them. A cost function, such as mean squared error (MSE) or gradient descent, can be used to penalize slightly incorrect predictions, much less than very incorrect predictions. If the performance of the AI / ML model does not improve after a certain number of training iterations, the data scientist can modify the reward function, provide corrections for incorrect predictions, etc.

[0129] Backpropagation is a technique for optimizing the synaptic weights in a feedforward neural network. Backpropagation can be used to "uncover" the hidden layers of a neural network to see how much loss each node is responsible for, and then update the weights to minimize the loss by giving nodes with higher error rates lower weights and vice versa. In other words, backpropagation allows the data scientist to repeatedly adjust the weights to minimize the difference between the actual output and the desired output.

[0130] The backpropagation algorithm is mathematically based in optimization theory. In supervised learning, training data with known outputs are passed through the neural network, and an error is calculated using a cost function with the known target output, which provides the error for backpropagation. The error is calculated at the output and converted into a correction of the network weights to minimize the error.

[0131] In the case of supervised learning, an example of backpropagation is provided below. A column vector input x is processed through a series of N non-linear activation functions f between each layer i = 1,..., N of the network i where the output of a given layer is first multiplied by the synaptic matrix W i , and the bias vector b i is added. The network output o is given by

[0132] o = f N (W N f N-1 (W N-1 f N-2 (... f1(W1x + b1)... ) + b N-1 ) + b N ) (4)

[0133] In some embodiments, comparing o with the target output t results in an error that needs to be minimized.

[0134] Optimization in the form of a gradient descent procedure can be used to minimize the error by modifying the synaptic weights W of each layer i The gradient descent procedure requires computing the output o given an input x corresponding to a known target output t, and producing the error o – t. The global error is then backpropagated, giving a local error for weight updates, which is computed similarly to but not exactly the same as the computation used for forward propagation. In particular, the backpropagation step typically requires an activation function of the form p j (n j ) = f′ j (n j ) where n j is the network activity at layer j (i.e., n j = W j o j-1 + b j ), where o j = f j (n j ) and the prime ′ denotes the derivative of the activation function f.

[0135] The weight updates can be computed via the following formula:

[0136]

[0137]

[0138]

[0139]

[0140]

[0141] where ° denotes the Hadamard product (i.e., the element-wise product of two vectors), T denotes matrix transpose, o j denotes f j (W j o j-1 + b j ), where o0 = x. Here, the learning rate η is chosen based on machine learning considerations. Below, η is related to the neural Hebbian learning mechanism used in the neural implementation. Note that the synapses W and b can be combined into a large synaptic matrix, where it is assumed that the input vector has additional synapses, and the additional columns representing the b synapses are subsumed into W.

[0142] AI / ML models can be trained over multiple epochs until they reach a good accuracy level (e.g., detection using F2 or F4 thresholds and approximately 2000 epochs to reach 97% or better). In some embodiments, the F1 score, F2 score, F4 score, or any other suitable technique can be used to determine this accuracy level without departing from the scope of the invention. Once trained on the training data, the AI / ML model can be tested on an evaluation data set that the AI / ML model has not encountered before. This helps ensure that the AI / ML model does not "overfit," meaning it performs well on the training data but poorly on other data.

[0143] In some embodiments, it may not be known what accuracy level the AI / ML model may reach. Thus, if the accuracy of the AI / ML model starts to decline when analyzing the evaluation data (i.e., the model performs well on the training data but poorly on the evaluation data), the AI / ML model can undergo more training epochs on the training data (and / or new training data). In some embodiments, the AI / ML model is only deployed when the accuracy reaches a certain level or if the accuracy of the trained AI / ML model is better than an existing deployed AI / ML model. In certain embodiments, a collection of trained AI / ML models can be used to complete a task. For example, one model can be trained to recognize images, another model can recognize text, and yet another model can recognize semantic and / or ontological associations, and so on.

[0144] Some embodiments can use a transformer network, such as SentenceTransformers TM , which is a Python TM framework for state-of-the-art sentence, text, and image embeddings. Such a transformer network learns associations of words and phrases with high and low scores. This trains the AI / ML model to determine which are close to the input and which are not close to the input, respectively. The transformer network can also use field lengths and field types, not just paired words / phrases.

[0145] As mentioned above, NLP techniques such as word2vec, BERT, GPT-3, ChatGPT, other LLMs, etc., can be used in some embodiments to facilitate semantic understanding and provide more accurate and human-like answers. Other techniques, such as clustering algorithms, can be used to discover similarities between groups of elements. Clustering algorithms can include but are not limited to density-based algorithms, distribution-based algorithms, centroid-based algorithms, hierarchical-based algorithms. K-means clustering algorithm, DBSCAN clustering algorithm, Gaussian mixture model (GMM) algorithm, balanced iterative reducing and clustering using hierarchies (BIRCH) algorithm, etc. These techniques can also help with classification.

[0146] Figure 7 is a flowchart showing process 700 for training an AI / ML model(s) according to an embodiment of the present invention. In some embodiments, as described above, the AI / ML model(s) can be generative AI models. The neural network architecture of an AI / ML model typically includes multiple layers of neurons, including input, output, and hidden layers. For example, see Figure 6A and Figure 6B . The intermediate hidden layers process the input data and generate an intermediate representation of the input for generating the output. These hidden layers can include various types of neurons, such as convolutional neurons, recurrent neurons, and / or transform neurons.

[0147] At 710, the training process begins by providing an RPA workflow, PDD, and / or other documents, screenshots, annotated RPA workflow activities, etc., whether labeled or unlabeled. Then at 720, the AI / ML model is trained over multiple epochs, and at 730, the results are reviewed. Although various types of AI / ML models can be used, LLM and other generative AI models are typically trained using a process called "supervised learning," which was also discussed above. Supervised learning involves providing the model with a large dataset that the model uses to learn the relationship between the input and the output. During the training process, the model adjusts the weights and biases of the neurons in the neural network to minimize the difference between the predicted output and the actual output in the training dataset.

[0148] In some embodiments, one aspect of the model is the use of transfer learning. For example, transfer learning can utilize a pre-trained model, such as ChatGPT, to be fine-tuned for a specific task or domain in step 720. This allows the model to leverage the knowledge already learned from the pre-training phase and adapt it to a specific application via the training phase in step 720.

[0149] The pre-training phase involves training the model on an initial training dataset that may be more general. In this phase, the model learns the relationships in the data. In the fine-tuning phase (e.g., in some embodiments, if the pre-trained model is used as the initial basis for the final model, it is performed during step 720 in addition to or instead of the initial training phase), the pre-trained model is adapted to a specific task or domain by training the model on a smaller task-specific dataset. For example, in some embodiments, the model can focus on certain types of data sources. This can help the model identify data elements therein more accurately than a generative AI model pre-trained alone. Fine-tuning allows the model to learn the nuances of the source, such as specific vocabulary and grammar, certain graphical features, certain data formats, etc., without requiring as much data as would be needed to train the model from scratch. By leveraging the knowledge learned in the pre-training phase, the fine-tuned model can achieve state-of-the-art performance on a specific task with relatively little additional training data.

[0150] If the AI / ML model fails to meet the desired confidence threshold at 740, additional training data and / or the reward function is modified at 750 to help the AI / ML model better achieve its goal, and the process returns to step 720. If the AI / ML model meets the confidence threshold at 740, the AI / ML module is tested on the evaluation data at 760 to ensure that the AI / ML model generalizes well and the AI / ML model is not overfitted with respect to the training data. The evaluation data includes information that the AI / ML model has not processed before. If the confidence threshold for the evaluation data is met at 770, the AI / ML model is deployed at 780. If not, the process returns to step 750 and the AI / ML model is further trained.

[0151] Figure 8 An AI / ML model of the cognitive AI layer 800 according to an embodiment of the present invention is shown. In some embodiments, the model of the cognitive AI layer 800 can use Figure 7It is trained by process 700. The generative AI model 810 provides the results to other cognitive AI layer models in series configuration 840, parallel configuration 842, or a combination of series and parallel configurations 844. For example, the generative AI model can generate an annotated RPA workflow, generate PDDs and / or other documents, provide semantic associations between on-screen texts, logically group categories of RPA workflow activities, infer subsequent activities to add based on the context of the RPA workflow under development, convert the RPA workflow from the format of one RPA provider to that of another RPA provider, generate automation from PDDs or business process descriptions provided by the user, and so on. The CV model 820 and the OCR model also provide the detected graphical elements and recognized texts to the generative AI model 810 and other cognitive AI layer models in configurations 840, 842, or 844 respectively.

[0152] Other cognitive AI layer models in configurations 840, 842, or 844 use the outputs of the generative AI model 810, CV model 820, and / or OCR model 830 to provide the automatic annotation and technical specification generation discussed in detail herein. The cognitive AI layer 800 can facilitate understanding what code to generate for the RPA workflow, what text to generate for the PDD, etc. The output (i.e., result 850) from other cognitive AI layer models in configurations 840, 842, or 844 can include RPA workflows, PDDs, other documents, etc.

[0153] Figure 9 Figure 900 is a screenshot showing an annotated RPA workflow according to an embodiment of the present invention. Field 910 lists the project name, and text box 920 includes a project description generated by the generative AI model of the cognitive AI layer. For example, the generative AI model can be an LLM trained in a fine-tuning phase to understand the nature and interrelationships of RPA workflow activities. In Figure 9 the example, the generative AI model can understand the existing activities to create a new row in the provider table, add a new row in , notify individual teams via , and send email confirmations to these individuals via . More complex workflows are possible and may be annotated by the cognitive AI layer.

[0154] Figure 10 Figure 1000 is a flowchart showing a process 1000 for using AI to provide automatic annotation and technical specification generation of an RPA workflow for an existing RPA automation according to an embodiment of the present invention. In some embodiments, Figure 10 process 1000 can be part of Figure 11 process 1100. For example, in some embodiments, Figure 10Steps 1010 to 1040 can correspond to Figure 11 steps 1130, 1140.

[0155] In some embodiments, the process begins with generating a PDD that describes a process for automation using a generative AI model of a cognitive AI layer or another generative AI model (e.g., a model trained separately from and operating independently of the cognitive AI layer). The PDD can be generated by the generative AI model using various sources. For example, the PDD can be generated using process logs (e.g., from or other applications), manual tracking, diagrams, etc.

[0156] At 1020, the RPA workflow code or PDD is provided as an input to the cognitive AI layer. For example, an XAML file for the RPA workflow can be fed into an initial AI / ML model of the cognitive AI layer, such as an LLM. In some embodiments, (multiple) screenshots can be provided such that a CV and / or OCR model can identify the images and / or text therein. The cognitive AI layer then processes the RPA workflow code or PDD (and other input information, if provided) at 1030 and provides annotations for the RPA workflow or code for the RPA workflow including the annotations at 1040. For example, the generative AI model of the cognitive AI layer can be trained to annotate the RPA workflow and / or generate a PDD from the content of the RPA workflow. In some embodiments, the RPA workflow annotations can provide a description of the entire process of the RPA workflow, describing what each activity in the RPA workflow is doing, or both. In certain embodiments, the annotations include a description of the changes between the current version of the RPA workflow and (multiple) previous versions of the RPA workflow. For example, the description of an activity can be partitioned by what it did in a previous version (if any) and what has changed relative to the current version. At 1050, in embodiments where an annotated RPA workflow is generated, the annotated RPA workflow can be displayed in an RPA designer application.

[0157] Figure 11It is a flowchart showing process 1100 for using AI to provide automatic annotation and technical specification generation for an RPA workflow when a developer creates an RPA workflow according to an embodiment of the present invention. The process starts with monitoring the RPA workflow development in the RPA designer application at 1110. This monitoring can be performed by the RPA designer application itself or by another process (e.g., a task mining process, a listener process, etc.). Then, the RPA designer application or other process determines at 1120 that an activity of the RPA workflow is added or modified. This determination can be made when an activity is completed and the user advances to the next activity, when any changes are made periodically (e.g., every minute, every ten minutes, etc.), when the user saves the RPA workflow, when the user clicks a button for annotating the RPA workflow in the RPA designer application, and so on.

[0158] At 1130, the RPA workflow code is provided to and processed by the cognitive AI layer, and the cognitive AI layer provides the annotated RPA workflow. For example, an XAML file for the RPA workflow can be fed into an initial AI / ML model of the cognitive AI layer, such as an LLM. In some embodiments, screenshots can be provided so that CV and / or OCR models can identify images and / or text therein. At 1140, the RPA designer application uses this output to update the RPA workflow and display the annotated version to the user. In certain embodiments, the annotation includes a description of the changes between the current version and a previous version of the RPA workflow. If the user is still working on the RPA workflow, the process returns to step 1110. Otherwise, the process ends.

[0159] Figure 12 It is a flowchart showing a process for creating an RPA workflow using a PDD or another natural language description of a business process as input according to an embodiment of the present invention. The process starts at 1210 by providing a PDD or other business process description to the cognitive AI layer. For example, a user can type a natural language describing the desired business process, use speech-to-text to dictate what he or she wants to automate, etc. Then, the cognitive AI layer processes the PDD or other business process description at 1220 and generates the annotated RPA workflow at 1230. For example, the LLM of the cognitive AI layer can be trained during a fine-tuning phase to understand the nature and interrelationships of RPA workflow activities. In some embodiments, the cognitive AI layer or the RPA designer application can use the RPA workflow to generate runtime automation at 1240, which can be deployed in a production environment. In certain embodiments, documents such as a PDD (if it was not provided as input to the cognitive AI layer), compliance documents, audit documents, etc. can be generated at 1250.

[0160] According to an embodiment of the present invention, Figure 7and Figures 10 to 12 The process steps performed in can be executed by a computer program that encodes instructions for at least part of the process described in Figure 7 and Figures 10 to 12 The computer program can be embodied on a non-transitory computer-readable medium. The computer-readable medium can be, but is not limited to, a hard disk drive, a flash device, RAM, magnetic tape, and / or any other such medium or combination of media for storing data. The computer program can include instructions for controlling a processor of a computing system (e.g., Figure 5 processor 510 of computing system 500) of to implement Figure 7 and Figures 10 to 12 All or part of the encoded instructions for the process steps described in, which can also be stored on the computer-readable medium.

[0161] The computer program can be implemented in hardware, software, or a hybrid implementation. The computer program can consist of modules that communicate effectively with each other, and these modules are designed to transmit information or instructions for display. The computer program can be configured to operate on a general-purpose computer, an ASIC, or any other suitable device.

[0162] It is readily understood that, as generally described and illustrated in the accompanying drawings herein, the components of the various embodiments of the present invention can be arranged and designed in a variety of different configurations. Thus, the detailed description of the embodiments of the present invention as shown in the drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention.

[0163] The features, structures, or characteristics of the present invention described throughout this specification can be combined in any suitable manner in one or more embodiments. For example, throughout the specification, references to "certain embodiments", "some embodiments", or similar language mean that the particular features, structures, or characteristics described in connection with the embodiments are included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in certain embodiments", "in some embodiments", "in other embodiments", or similar language throughout this specification do not necessarily all refer to the same set of embodiments, and the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0164] It should be noted that references to features, advantages, or similar language throughout this specification do not mean that all features and advantages that can be achieved by the present invention should be or exist in any single embodiment of the present invention. On the contrary, the language referring to features and advantages is understood to mean that the particular features, advantages, or characteristics described in connection with the embodiments are included in at least one embodiment of the present invention. Thus, the discussions of features and advantages and similar language throughout this specification can, but do not necessarily, refer to the same embodiments.

[0165] In addition, the features, advantages, and characteristics of the present invention may be combined in any suitable manner in one or more embodiments. Those skilled in the relevant art will recognize that the present invention may be practiced without one or more specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not exist in all embodiments of the present invention.

[0166] Those of ordinary skill in the art will readily understand that the present invention as described above may be practiced with steps in a different order and / or with hardware elements in a configuration different from that disclosed. Accordingly, although the present invention has been described based on these preferred embodiments, certain modifications, variations, and alternative configurations will be apparent to those skilled in the art while remaining within the spirit and scope of the present invention. Therefore, reference should be made to the appended claims to determine the scope and limitations of the present invention.

Claims

1. A non-transitory computer readable medium storing a computer program for providing automatic annotation for a Robotic Process Automation (RPA) workflow, the computer program being configured to cause at least one processor to: Provide the cognitive artificial intelligence AI layer with code for RPA workflow or process definition document PDD; The code for the RPA workflow or the PDD is processed by the cognitive AI layer; and The generative AI layer provides annotations for the RPA workflow or RPA workflow code including the annotations as output.

2. The non-transitory computer readable medium of claim 1 , wherein the computer program is configured to cause the at least one processor to: The PDD describing the process to be automated is generated using the generative AI model of the cognitive AI layer or another generative AI model.

3. The non-transitory computer-readable medium of claim 1, wherein the annotation comprises a description of changes between a current version of the RPA workflow and one or more previous versions of the RPA workflow.

4. The non-transitory computer readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to: The annotated RPA workflow is displayed in a user interface of an RPA designer application.

5. The non-transitory computer-readable medium of claim 1, wherein the generative AI layer comprises a large language model (LLM) that is fine-tuned during training to understand the nature of RPA workflow activities and the interrelationships between the RPA workflow activities.

6. The non-transitory computer readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to: One or more screenshots are provided to a computer visualization CV model and / or an optical character recognition OCR model of the cognitive AI layer to identify text and / or images therein, wherein the one or more screenshots include one or more visual representations of at least a portion of the RPA workflow.

7. The non-transitory computer-readable medium of claim 1, wherein the annotation provides a description of the overall process of the RPA workflow, describes what each activity in the RPA workflow is doing, or both.

8. The non-transitory computer readable medium of claim 1, wherein the computer program is further configured to cause the at least one processor to: Monitoring the development of the RPA workflow in the RPA designer application; determining that one or more activities are added to the RPA workflow and / or modified in the RPA workflow; and In response to the determination that the one or more activities are added to the RPA workflow and / or modified in the RPA workflow, providing the code for the RPA workflow to the cognitive AI layer.

9. The non-transitory computer-readable medium of claim 8, wherein the determination that the one or more activities are added to the RPA workflow and / or modified in the RPA workflow comprises: Determining that an activity of the RPA workflow is completed and the user advances to the next activity, periodically checking for changes to the RPA workflow, determining that the user has saved the RPA workflow, and determining that the user has clicked a button for annotating the RPA workflow in an RPA designer application.

10. The non-transitory computer readable medium of claim 8, wherein the computer program is further configured to cause the at least one processor to: After the generative AI layer provides the annotations for the RPA workflow or the RPA workflow code including the annotations as output, repeating the steps of claim 8.

11. The non-transitory computer readable medium of claim 1 , wherein the cognitive AI layer comprises: A generative AI model configured to generate annotated RPA workflows, provide semantic associations between text on the screen, logically group categories of RPA workflow activities, infer subsequent activities to be added based on the context of the RPA workflow, convert RPA workflows from the format of one RPA provider to the format of another RPA provider, or any combination thereof.

12. One or more computing systems, including: A memory storing computer program instructions for providing automatic annotation for a Robotic Process Automation (RPA) workflow; as well as at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to: providing code for an RPA workflow or process definition document (PDD) to a cognitive artificial intelligence (AI) layer, the cognitive artificial intelligence layer including a generative AI model configured to generate annotated RPA workflows, provide semantic associations between text on the screen, logically group categories of RPA workflow activities, reason about subsequent activities to be added based on the context of the RPA workflow, convert RPA workflows from a format of one RPA supplier to a format of another RPA supplier, or any combination thereof, the code for the RPA workflow or the PDD is processed by the cognitive AI layer, and The generative AI layer provides as output an annotation for the RPA workflow or an RPA workflow code including the annotation, wherein The annotations provide a description of the overall process of the RPA workflow, describe what each activity in the RPA workflow is doing, or both.

13. The one or more computing systems of claim 12, wherein the annotation comprises a description of changes between a current version of the RPA workflow and one or more previous versions of the RPA workflow.

14. The one or more computing systems of claim 12, wherein the computer program instructions are further configured to cause the at least one processor to: The annotated RPA workflow is displayed in a user interface of an RPA designer application.

15. The one or more computing systems of claim 12, wherein the generative AI layer comprises a large language model (LLM) that is fine-tuned during training to understand the nature of RPA workflow activities and the interrelationships between the RPA workflow activities.

16. The one or more computing systems of claim 12, wherein the computer program instructions are further configured to cause the at least one processor to: providing one or more screenshots to a computer visualization (CV) model and / or an optical character recognition (OCR) model of the cognitive AI layer to identify text and / or images therein, the one or more screenshots comprising one or more visual representations of at least a portion of the RPA workflow, wherein The CV model and / or the OCR model are configured to provide output to the generative AI model of the cognitive AI layer.

17. The one or more computing systems of claim 12, wherein the computer program instructions are further configured to cause the at least one processor to: Monitoring the development of the RPA workflow in the RPA designer application; determining that one or more activities are added to the RPA workflow and / or modified in the RPA workflow; and In response to the determination that the one or more activities are added to the RPA workflow and / or modified in the RPA workflow, providing the code for the RPA workflow to the cognitive AI layer.

18. The one or more computing systems of claim 17, wherein the determination that the one or more activities are added to the RPA workflow and / or modified in the RPA workflow comprises: Determining that an activity of the RPA workflow is completed and the user advances to the next activity, periodically checking for changes to the RPA workflow, determining that the user saved the RPA workflow, or determining that the user clicked a button for annotating the RPA workflow in an RPA designer application.

19. A computer-implemented method for providing automatic annotation for a Robotic Process Automation (RPA) workflow, comprising: providing, by an RPA designer application executed on a computing system, a code for an RPA workflow or a process definition document PDD to a cognitive artificial intelligence AI layer, the cognitive AI layer being configured to process the code for the RPA workflow and / or the PDD; Receiving, by the RPA designer application from the generative AI layer, annotations for the RPA workflow or RPA workflow code including the annotations; as well as The annotated RPA workflow is displayed by the RPA designer application, wherein The annotations provide a description of the overall process of the RPA workflow, a description of what each activity in the RPA workflow is doing, a description of changes between a current version of the RPA workflow and one or more previous versions of the RPA workflow, or any combination thereof.

20. The computer-implemented method of claim 19, wherein the generative AI layer comprises a large language model (LLM) that is fine-tuned during training to understand the nature of RPA workflow activities and the interrelationships between the RPA workflow activities.

21. The computer-implemented method of claim 19, further comprising: The RPA designer application provides one or more screenshots to the computer visualization CV model and / or optical character recognition OCR model of the cognitive AI layer to identify text and / or images therein, wherein the one or more screenshots include one or more visual representations of at least a portion of the RPA workflow, wherein The CV model and / or the OCR model are configured to provide output to the generative AI model of the cognitive AI layer.

22. The computer-implemented method of claim 19, further comprising: Monitoring the development of the RPA workflow by the RPA designer application; determining, by the RPA designer application, that one or more activities are added to the RPA workflow and / or modified in the RPA workflow; as well as In response to the determination that the one or more activities are added to the RPA workflow and / or modified in the RPA workflow, providing, by the RPA designer application, the code for the RPA workflow to the cognitive AI layer, wherein The determination that one or more activities are added to the RPA workflow and / or modified in the RPA workflow includes: determining that an activity of the RPA workflow is completed and the user advances to the next activity, periodically checking for changes to the RPA workflow, determining that the user saved the RPA workflow, or determining that the user clicked a button for annotating the RPA workflow in an RPA designer application.

23. The computer-implemented method of claim 22, further comprising: After the generative AI layer provides the annotations for the RPA workflow or the RPA workflow code including the annotations as output, the steps of claim 22 are repeated by the RPA designer application.

24. The computer-implemented method of claim 19, wherein the cognitive AI layer comprises: A generative AI model configured to generate annotated RPA workflows, provide semantic associations between text on the screen, logically group categories of RPA workflow activities, infer subsequent activities to be added based on the context of the RPA workflow, convert RPA workflows from the format of one RPA supplier to the format of another RPA supplier, or any combination thereof.

Images