Artificial intelligence / machine learning model drift detection and correction for robotic process automation

Abstract

This paper discloses a method for detecting and correcting AI / ML model drift in Robotic Process Automation (RPA). It analyzes information related to the input data of the AI / ML model to determine if data drift has occurred, analyzes information related to the results of the AI / ML model's execution to determine if model drift has occurred, or both. Based on the analysis of this information, the AI / ML model is retrained when changes in conditions are detected, when a change threshold is met or exceeded, or both. The retrained AI / ML model can then be deployed to provide better predictions for real-world data.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. nonprovisional patent application number 16 / 934,369, filed July 21, 2020. The subject matter of that earlier application is incorporated herein by reference in its entirety. Technical Field

[0003] This invention relates generally to robotic process automation (RPA), and more specifically to drift detection and correction of artificial intelligence (AI) / machine learning (ML) models for RPA. Background Technology

[0004] Trained AI / ML models can be deployed to provide probabilistic solutions for a variety of tasks, such as image recognition and document processing. However, predictions made by AI / ML models can change or drift over time. In the context of RPA, where software bots can leverage AI / ML models to perform tasks with or without user knowledge, or perhaps without any user involvement at all, AI / ML model drift can be particularly problematic. Therefore, improved techniques for detecting and / or correcting AI / ML model drift can be beneficial. Summary of the Invention

[0005] Certain embodiments of the present invention may provide solutions to problems and needs that have not yet been fully identified, understood, or resolved by current RPA technologies. For example, some embodiments of the present invention relate to AI / ML model drift detection and correction for RPA.

[0006] In one embodiment, a computer program for performing AI / ML model drift detection and correction for RPA is embodied on a non-transitory computer-readable medium. The computer program is configured to cause at least one processor to: analyze information about input data related to the AI / ML model to determine whether data drift has occurred, analyze information about the results of execution from the AI / ML model to determine whether model drift has occurred, or both. Based on the analysis of the information, the computer program is further configured to cause at least one processor to retrain the AI / ML model upon detection of a change in conditions, upon satisfaction or exceeding of a change threshold, or both.

[0007] In another embodiment, a computer-implemented method for performing AI / ML model drift detection and correction for RPA includes: analyzing information related to input data of the AI / ML model to determine whether data drift has occurred, analyzing information related to the execution results of the AI / ML model to determine whether model drift has occurred, or both. Based on the analysis of the information, upon detection of a change in conditions, upon meeting or exceeding a change threshold, or both, the method includes retraining the AI / ML model, verifying whether the retrained AI / ML model meets one or more performance thresholds, and deploying the retrained AI / ML model when the retrained AI / ML model meets one or more performance thresholds.

[0008] In another embodiment, a computer-implemented method for performing AI / ML model drift detection and correction for RPA includes analyzing information comprising one or more statistical distributions including predictions made by the AI / ML model. This information relates to input data to the AI / ML model used to determine whether data drift has occurred, and to the results of execution from the AI / ML model used to determine whether model drift has occurred, or both. The method also includes mapping the one or more statistical distributions to corresponding actions taken by multiple RPA robots. Based on the analysis of the information, the method further includes retraining the AI / ML model when a change condition is detected for at least a period of time, a change threshold is met or exceeded for at least a period of time, or both. This action is taken in the corresponding activity of the RPA workflow of the respective RPA robot. Attached Figure Description

[0009] To facilitate understanding of the advantages of certain embodiments of the invention, a more specific description of the invention, which has been briefly described above, will be presented with reference to specific embodiments shown in the accompanying drawings. It should be understood that these drawings depict only exemplary embodiments of the invention and should not be considered as limiting its scope. The invention will be described and explained with additional features and details using the drawings, wherein:

[0010] Figure 1 This is an architecture diagram illustrating a robotic process automation (RPA) system according to an embodiment of the present invention.

[0011] Figure 2 This is an architecture diagram illustrating an RPA system deployed according to an embodiment of the present invention.

[0012] Figure 3 This is an architecture diagram illustrating the relationship between the designer, activities, and drivers according to an embodiment of the present invention.

[0013] Figure 4 This is an architecture diagram illustrating an RPA system according to an embodiment of the present invention.

[0014] Figure 5 This is an architecture diagram illustrating a computational system configured to perform AI / ML model drift detection and correction for RPA according to an embodiment of the present invention.

[0015] Figure 6 This is an architecture diagram illustrating a system configured to perform AI / ML model drift detection and correction for RPA according to an embodiment of the present invention.

[0016] Figure 7A and Figure 7B This is a flowchart illustrating the process of performing AI / ML model drift detection and correction for RPA according to an embodiment of the present invention.

[0017] Unless otherwise stated, similar reference numerals in all figures always denote corresponding features. Detailed Implementation

[0018] Some embodiments involve AI / ML model drift detection and correction for RPA. Some embodiments store key performance indicators (KPIs), such as how many predictions the AI / ML model made and how many were actually used, and one or more statistical distributions of the predictions. Warnings or retraining triggers can be generated when a threshold is met or exceeded. For example, the threshold could be a deviation from historical performance by at least a certain amount, potentially over at least a certain time period. The AI / ML model can then potentially be trained using (updated) new data provided by one or more users and / or bots, and a new version or entirely new model of the AI / ML model can be deployed that provides better predictions and addresses the drift problem.

[0019] AI / ML model drift can occur due to data drift, model drift, or both. Data drift occurs when the characteristics of the input data to an AI / ML model change over time from the data used to train the model. This can lead to less accurate predictions made by the AI / ML model. Model drift is the change over time of an underlying problem in the relationship between the data input to and output from an AI / ML model. The occurrence of one or both of these forms of drift can impair the performance of an AI / ML model, or even make it inaccurate for the problem it was trained to solve. Data drift and model drift are collectively referred to as “concept drift” in this paper.

[0020] In some embodiments, statistical distributions are mapped to actions taken by RPA robots in their workflows. In some embodiments, statistical distributions may be matched to business outcomes. For example, consider a scale of 1 to 5 for a particular business outcome, and the outcome typically falls within a standard distribution (e.g., 2 to 4) on a scale of 1 to 5. If 1 and 5 start to become more prevalent (e.g., occurring more frequently than one standard deviation), anomalies can be automatically triggered. Human reviewers can examine the performance characteristics of the AI / ML model to determine how the AI / ML model is drifting. Human validation can be performed to improve the model, thereby improving business outcomes. The root cause of drift can be data drift or model drift. In some embodiments, a dashboard may be presented to users showing these statistical distributions for each business outcome.

[0021] In the context of RPA, drift detection can be particularly important. For example, when an RPA bot is invoking an AI / ML model to make predictions, there are currently no existing methods to check the quality of the model in these scenarios without human supervision. RPA bots are consumers of AI / ML models, and they can operate autonomously. Users may lack control over what actions the bot takes based on these AI / ML model predictions, and this autonomy of RPA bots can lead to unintended or even dangerous results (e.g., in the case of mission-critical systems).

[0022] Some embodiments employ one or more algorithms to measure data drift and model drift. For example, techniques for reducing the dimensionality of a dataset (e.g., principal component analysis (PCA)), density estimation (i.e., evaluating the continuous properties of density in a high-dimensional space), combinations thereof, etc., can be used. However, it should be noted that any suitable technique(s) can be used to measure drift without departing from the scope of the invention.

[0023] In some embodiments, an automation platform provides and implements a framework for detecting and handling AI / ML model drift. For example, if an AI / ML model is deployed to production, it may have an input pattern common to other deployed AI / ML models within the automation platform. In some embodiments, the extent to which the pattern definition is implemented may depend on the AI / ML model itself and the AI / ML model pipeline. The input pattern describes what the AI / ML model accepts as input. In some embodiments, this may be a comma-separated value (CSV) file with predefined columns and specific data types. Certain field requirements and / or values ​​may be implemented.

[0024] For example, if an email body is passed as input to an AI / ML model in serialized JavaScript Object Notation (JSON), then the email body can be considered input. However, CSV files with numerical data may not be accepted because the AI / ML model may not know how to interpret the data. Therefore, this schema can help prevent unwanted or unauthorized data formats from being fed into the AI / ML model.

[0025] When a deployed AI / ML model is run and analytics are performed, representations of the input data and AI / ML model predictions can be built up over time. For example, the distance between a one-dimensional distribution of the original dataset and a new dataset can be measured to account for covariate drift. In some embodiments, the analytics include determining whether statistical moments have changed. In some embodiments, statistical moments include mean, variance, skewness, kurtosis, covariance, or combinations thereof. Moreover, the distance between the residual drift of two models can be examined. For example, if two variants of an AI / ML model (e.g., a new version of the model and a previous version of the model) are trained on different datasets or different parts of the dataset, the distance between the distributions of the models can be used to attempt to determine why drift occurs in one model and may not occur (or at least less) in the other.

[0026] Consider the example above again, where the AI / ML model predicts that the input data falls into one of five classes ranging from 1 to 5. The AI / ML model has been deployed for 6 months, and 95% of its predictions are for class 2, 3, or 4. However, the AI / ML model then begins making more predictions for class 1 and 5. It can then be determined whether the characteristics of the input data (e.g., its distribution) have changed, causing the change in the predictions, or whether the change in the AI / ML model's predictions occurred even without significant changes in the input data.

[0027] In some embodiments, activities in an RPA workflow can detect whether drift is occurring (and, in some cases, what type). For example, an RPA designer application may provide a set of activities that allow developers to consider data or model drift. These activities can be configured based on a certain percentage of drift occurring, exceeding the standard deviation, or unusual predictions becoming significantly more common.

[0028] When drift is identified, triggers can be generated. For example, a trigger could send and store predictions and / or input data made by the AI / ML model for analysis by a human reviewer. An action center queue can allow humans to validate and automatically retrain a new version of the model. The human reviewer can then review the AI / ML model's predictions based on the input data and determine if there are issues with the model's accuracy. When the retrained AI / ML model meets certain thresholds (e.g., a confidence threshold, some form of random sampling, other trial-and-error methods, etc.), it can be automatically deployed alongside the existing AI / ML model, allowing for side-by-side measurement of model drift to see if the new version performs better on real-world data. If the model falls below the threshold, a human review can be conducted to determine why it doesn't meet the threshold, and attempts can be made to train the AI / ML model to meet or exceed it.

[0029] In some embodiments, when training an AI / ML model, a portion of the training data is separated for later testing. A set of metrics can be used to create a high-level metric to evaluate the model. An example is the F1 score for a document understanding model. This is a derivative of precision and recall, with a trade-off between the two. Cross-validation can be performed to separate another segment of data, mix it, and use the mixed data to test the model. However, if the separated data segment does not represent real-world data, the model may overfit the training data and not perform well for real-world applications.

[0030] In some embodiments, the process can be automated. For example, if the F1 score exceeds 90%, a new version of the AI / ML model can be deployed. The drift of this new version can then be checked and / or manually verified.

[0031] Some implementations can be used for robotic process automation (RPA). Figure 1 This is an architectural diagram illustrating an RPA system 100 according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that allows developers to design and implement workflows. The designer 110 can provide solutions for application integration and automation of third-party applications, management of information technology (IT) tasks, and business IT processes. The designer 110 can facilitate the development of automation projects, which are graphical representations of business processes. Simply put, the designer 110 facilitates the development and deployment of workflows and robots.

[0032] Automation projects automate rule-based processes by giving developers control over the order of execution and the relationships between custom sets of steps developed within a workflow (defined here as "activities"). A commercial example of an embodiment of Designer 110 is UiPath Studio. TM Each activity may include actions such as clicking a button, reading a file, or writing to a log panel. In some embodiments, workflows may be nested or embedded.

[0033] Some types of workflows can include, but are not limited to, sequences, flowcharts, flow charts (FSMs), and / or global exception handlers. Sequences are particularly well-suited for linear processes, enabling flow from one activity to another without disrupting the workflow. Flowcharts are particularly suitable for more complex business logic, allowing for the integration of decisions and connecting activities in more diverse ways through multiple branching logic operators. FSMs are especially suitable for large workflows. FSMs can utilize a limited number of states in their execution, triggered by conditions (i.e., transitions) or activities. Global exception handlers are particularly well-suited for identifying workflow behavior and debugging processes when execution errors are encountered.

[0034] Once a workflow is developed in designer 110, the execution of the business process is coordinated by commander 120. Commander 120 coordinates one or more robots 130 that execute the workflow developed in designer 110. A commercial example of a commander 120 is the UiPath Orchestrator. TM Commander 120 helps manage the creation, monitoring, and deployment of resources within the environment. Commander 120 can serve as one of the integration or aggregation points for third-party solutions and applications.

[0035] Commander 120 can manage a fleet of robots 130, connecting and executing robots 130 from a central point. The types of manageable robots 130 include, but are not limited to, manned robots 132, unmanned robots 134, development robots (similar to unmanned robots 134 but used for development and testing purposes), and non-production robots (similar to manned robots 132 but used for development and testing purposes). Manned robots 132 are triggered by user events and operate alongside humans on the same computing system. Manned robots 132 can be used with commander 120 for centralized process deployment and recording media. Manned robots 132 can assist human users in completing various tasks and can be triggered by user events. In some embodiments, processes cannot be started from commander 120 on this type of robot and / or they cannot run under a locked screen. In some embodiments, manned robots 132 can only be started from a robot tray or from a command prompt. In some embodiments, manned robots 132 should operate under human supervision.

[0036] Unattended robot 134 operates unattended in a virtual environment and can automate many processes. Unattended robot 134 can be responsible for remote execution, monitoring, scheduling, and support of work queues. In some embodiments, debugging of all robot types can be run in designer 110. Both manned and unattended robots can automate various systems and applications, including but not limited to mainframes, web applications, VMs, and enterprise applications (e.g., by...). Those generated by [etc.] and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computing applications, etc.).

[0037] Commander 120 may have various capabilities, including but not limited to provisioning, deployment, version control, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining a connection between robot 130 and commander 120 (e.g., a web application). Deployment may include ensuring that package versions are correctly delivered to assigned robots 130 for execution. In some embodiments, version control may include the management of unique instances of some processes or configurations. Configuration may include the maintenance and delivery of robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs to a database (e.g., an SQL database) and / or another storage mechanism (e.g., a web application). It provides the ability to store and quickly query large datasets. The Commander 120 can provide interconnectivity by acting as a centralized communication point for third-party solutions and / or applications.

[0038] Robot 130 is the execution agent that runs the workflows constructed in designer 110. A commercial example of some embodiments of (multiple) robots 130 is UiPath Robots. TM In some embodiments, robot 130 has Microsoft installed by default. Services managed by the Service Control Manager (SCM). As a result, this robot 130 can open interactive windows under the local system account. Conversation, and has Service permissions.

[0039] In some embodiments, robots 130 can be installed in user mode. For such robots 130, this means they have the same permissions as the user who installed the given robot 130. This feature can also be used for high-density (HD) robots, which ensure full utilization of each machine at its maximum potential. In some embodiments, any type of robot 130 can be configured in an HD environment.

[0040] In some embodiments, robot 130 is divided into several components, each dedicated to a specific automation task. In some embodiments, robot components include, but are not limited to, SCM-managed robot services, user-mode robot services, actuators, agents, and command lines. SCM-managed robot service management and monitoring... The session acts as a proxy between the controller 120 and the execution host (i.e., the computing system on which the robot 130 is executed). These services are trusted and manage the credentials of the robot 130. The console application is launched by the SCM under the local system.

[0041] In some embodiments, user-mode robot service management and monitoring The session acts as a proxy between the command center 120 and the execution host. The user-mode robot service trusts and manages credentials for robot 130. If an SCM-managed robot service is not installed, it can be started automatically. application.

[0042] The actuator can be A given job runs within a session (i.e., they can execute workflows). The executor can be aware of the dots per inch (DPI) setting. The agent can display available jobs in a system tray window. This demonstrates a basic (WPF) application. A proxy can be a client of a service. A proxy can request to start or stop a job and change settings. The command line is a client of a service. The command line is a console application that can request to start a job and wait for its output.

[0043] As described above, separating the components of robot 130 makes it easier for developers, support users, and the computing system to run, identify, and track what each component is doing. This allows for configuring specific behaviors for each component, such as establishing different firewall rules for executors and services. In some embodiments, the executor can always know the DPI setting for each monitor. As a result, workflows can be executed at any DPI, regardless of the configuration of the computing system on which the workflow is created. In some embodiments, projects from designer 110 can also be independent of browser scaling levels. For applications that do not know the DPI or are intentionally marked as unknown, DPI can be disabled in some embodiments.

[0044] Figure 2 This is an architectural diagram illustrating an RPA system 200 deployed according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be... Figure 1 The RPA system 100 or can be Figure 1This is part of the RPA system 100. It should be noted that the client side, server side, or both can include any desired number of computing systems without departing from the scope of the invention. On the client side, the robot application 210 includes an actuator 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not run on the computing system 210. The actuator 212 is running a process. Figure 2 As shown, several business projects can run simultaneously. In this embodiment, agent 214 (e.g., The service is a single contact point for all actuators 212. All messages in this embodiment are logged in the controller 230, which further processes them via a database server 240, an indexer server 250, or both. (See above regarding...) Figure 1 The actuator 212 may be a robot component.

[0045] In some embodiments, a robot represents an association between a machine name and a username. A robot can manage multiple actuators simultaneously. In computing systems that support multiple concurrent interactive sessions (e.g., On a server (2012), multiple bots can run simultaneously, each using a unique username on a separate server. Running in a session. This is referred to above as the HD robot.

[0046] Agent 214 is also responsible for sending the robot's status (e.g., periodically sending "heartbeat" messages indicating that the robot is still working) and downloading the required version of the package to be executed. In some embodiments, communication between agent 214 and controller 230 is always initiated by agent 214. In notification scenarios, agent 214 may open a WebSocket channel, which controller 230 later uses to send commands to the robot (e.g., start, stop, etc.).

[0047] On the server side, there are presentation layers (web application 232, Open Data Protocol (OData) representing State Transfer (REST) ​​Application Programming Interface (API) endpoint 234, and notification and monitoring 236), service layers (API implementation / business logic 238), and persistence layers (database server 240 and indexer server 250). The controller 230 includes web application 232, OData REST API endpoint 234, notification and monitoring 236, and API implementation / business logic 238. In some embodiments, most actions performed by the user in the interface of the controller 230 (e.g., via browser 220) are performed by calling various APIs. Such actions may include, but are not limited to, starting a job on a robot, adding / removing data from a queue, scheduling a job to run unattended, etc., without departing from the scope of the invention. Web application 232 is the visual layer of the server platform. In this embodiment, web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used. In this embodiment, the user interacts with a webpage from web application 232 via browser 220 to perform various actions to control the controller 230. For example, the user can create robot groups, assign packages to robots, analyze logs for each robot and / or each process, start and stop robots, etc.

[0048] In addition to web application 232, the controller 230 also includes a service layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the invention. The REST API is used by both web application 232 and agent 214. In this embodiment, agent 214 is a manager for one or more robots on a client computer.

[0049] The REST API in this embodiment includes configuration, logging, monitoring, and queuing functions. In some embodiments, the configuration endpoint can be used to define and configure application users, licenses, bots, assets, publications, and environments. The logging REST endpoint can be used to log various information, such as errors, explicit messages sent by the bot, and other environment-specific information. The deployment REST endpoint can be used by the bot to query the package version that should be executed if the start job command is used in command 230. Queuing the REST endpoint can handle queue and queue item management, such as adding data to the queue, retrieving transactions from the queue, and setting the status of transactions.

[0050] The monitoring REST endpoint can monitor web application 232 and agent 214. The notification and monitoring API 236 can be a REST endpoint used to register agent 214, deliver configuration settings to agent 214, and send / receive notifications from the server and agent 214. In some embodiments, the notification and monitoring API 236 can also use WebSocket communication.

[0051] In this embodiment, the persistence layer includes a pair of servers—a database server 240 (e.g., an SQL server) and an indexer server 250. In this embodiment, the database server 240 stores robot configurations, robot groups, associated processes, users, roles, schedules, etc. In some embodiments, this information is managed via a web application 232. The database server 240 can manage queues and queue items. In some embodiments, the database server 240 can store messages recorded by the robots (in addition to or instead of the indexer server 250).

[0052] In some embodiments, an optional indexer server 250 stores and indexes information recorded by the robot. In some embodiments, the indexer server 250 can be disabled via configuration settings. In some embodiments, the indexer server 250 uses... Should It is an open-source full-text search engine project. Messages logged by the bot (e.g., using activities such as log messages or write lines) can be sent to the indexer server 250 via logging (multiple) REST endpoints, where they are indexed for future use.

[0053] Figure 3This is an architecture diagram illustrating the relationship 300 between the designer 310, activities 320, 330, and driver 340 according to an embodiment of the present invention. As described above, the developer uses the designer 310 to develop workflows executed by a robot. The workflow may include user-defined activities 320 and UI automation activities 330. Some embodiments are capable of identifying non-textual visual portions in an image, referred to herein as computer vision (CV). Some CV activities associated with such portions may include, but are not limited to, clicking, typing, getting text, hovering, element presence, refreshing range, highlighting, etc. In some embodiments, clicking identifies an element and clicks it, for example using CV, optical character recognition (OCR), fuzzy text matching, and multi-anchors. Typing can identify an element using the types described above and elements. Getting text can identify the location of specific text and scan it using OCR. Hovering can identify an element and hover over it. Element presence can use the techniques described above to check if an element is present on the screen. In some embodiments, hundreds or thousands of activities can be implemented in the designer 310. However, any number and / or type of activities may be used without departing from the scope of the invention.

[0054] UI automation activities 330 are a subset of specialized low-level activities (e.g., CV activities) written in low-level code, designed to facilitate interaction with the screen. UI automation activities 330 facilitate these interactions through drivers 340 that allow the robot to interact with the desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc.

[0055] Driver 340 can interact with the OS at a lower level, find hooks, monitor keys, etc. They can facilitate... Integration of features such as "click" activities. For example, the "click" activity performs the same role in these different applications via driver 340.

[0056] Figure 4 This is an architectural diagram illustrating an RPA system 400 according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be... Figure 1 and / or Figure 2 RPA systems 100 and / or 200 or including Figure 1 and / or Figure 2 The RPA system 100 and / or 200. The RPA system 400 includes multiple client computing systems 410 that operate the robot. The computing systems 410 are capable of communicating with the command computing system 420 via a web application running thereon. The command computing system 420 is also capable of communicating with a database server 430 and an optional indexer server 440.

[0057] about Figure 1 and Figure 3It should be noted that although web applications are used in these embodiments, any suitable client / server software can be used without departing from the scope of the invention. For example, the controller can run a server-side application that communicates with a non-web-based client software application on a client computing system.

[0058] Figure 5 This is an architectural diagram illustrating a computing system 500 configured to perform AI / ML model drift detection and correction for RPA according to an embodiment of the present invention. In some embodiments, the computing system 500 may be one or more computing systems depicted and / or described herein. The computing system 500 includes a bus 505 or other communication mechanism for transmitting information, and multiple processors 510 coupled to the bus 505 for processing information. The multiple processors 510 may be any type of general-purpose or special-purpose processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The multiple processors 510 may also have multiple processing cores, and at least some cores may be configured to perform specific functions. In some embodiments, multi-parallel processing may be used. In some embodiments, at least one of the multiple processors 510 may be a neuromorphic circuit including processing elements that simulate biological neurons. In some embodiments, the neuromorphic circuit may not require typical components of a von Neumann computing architecture.

[0059] The computing system 500 also includes a memory 515 for storing information and instructions to be executed by the processor(s) 510. The memory 515 may consist of random access memory (RAM), read-only memory (ROM), flash memory, cache memory, static storage devices such as magnetic disks or optical disks, or any other type of non-transitory computer-readable medium or any combination thereof. The non-transitory computer-readable medium may be any available medium accessible by the processor(s) 510 and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both.

[0060] Additionally, the computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communication network via a wireless and / or wired connection. In some embodiments, the communication device 520 may be configured to use Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiplexing (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), CDMA2000, Wideband CDMA (W-CDMA), High-Speed ​​Downlink Packet Access (HSDPA), and High-Speed ​​Uplink. The communication device 520 may include HSUPA, HSPA, LTE, LTE-A, 802.11x, Wi-Fi, Zigbee, UWB, 802.16x, 802.15, HnB, Bluetooth, RFID, IrDA, NFC, 5G, NR, and / or any other existing or future communication standards and / or protocols, without departing from the scope of the invention. In some embodiments, the communication device 520 may include one or more antennas, which may be single, arrayed, phased, switched, beamformed, beamcontrolled, combinations thereof, and / or any other antenna configuration, without departing from the scope of the invention.

[0061] Multiple processors 510 are also coupled to a display 525 via a bus 505, such as a plasma display, liquid crystal display (LCD), light-emitting diode (LED) display, field emission display (FED), organic light-emitting diode (OLED) display, flexible OLED display, flexible substrate display, projection display, 4K display, high-definition display, etc. The display 525 may be a touch (haptic) display, an in-plane switching (IPS) display, or any other suitable display for displaying information to a user. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, etc., using resistive, capacitive, surface acoustic wave (SAW) capacitor, infrared, optical imaging, dispersive signal technology, acoustic impulse recognition, suppressed total internal reflection, etc. Any suitable display device and haptic I / O may be used without departing from the scope of the invention.

[0062] Keyboard 530 and cursor control devices 535, such as a computer mouse and touchpad, are also coupled to bus 505 to enable a user to interface with computing system 500. However, in some embodiments, a physical keyboard and mouse may be absent, and the user may interact with the device solely through display 525 and / or touchpad (not shown). Any type and combination of input devices can be considered as a design choice. In some embodiments, there are no physical input devices and / or displays. For example, a user may interact remotely with computing system 500 via another computing system with which it communicates, or computing system 500 may operate autonomously.

[0063] Memory 515 stores software modules that provide functionality when executed by processor(s) 510. These modules include an operating system 540 for computing system 500. These modules also include an AI / ML model drift detection and correction module 545, which is configured to perform all or part of the processes described herein or derivatives thereof. Computing system 500 may include one or more additional function modules 550 that include additional functionality.

[0064] Those skilled in the art will understand that the "system" can be implemented as a server, embedded computing system, personal computer, console, personal digital assistant (PDA), cellular phone, tablet computing device, quantum computing system, or any other suitable computing device or combination of devices without departing from the scope of the invention. Presenting the foregoing functionality as being performed by the "system" is not intended to limit the scope of the invention in any way, but rather to provide one example of many embodiments of the invention. In fact, the methods, systems, and apparatuses disclosed herein can be implemented in localized and distributed forms consistent with computing technologies, including cloud computing systems.

[0065] It should be noted that some system features described in this specification have been presented as modules to more specifically emphasize their implementation independence. For example, modules can be implemented as hardware circuitry including custom-designed very large-scale integration (VLSI) circuitry or gate arrays, such as logic chips, transistors, or other discrete components in off-the-shelf semi-controllers. Modules can also be implemented in programmable hardware devices such as field-programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, etc.

[0066] Modules can also be implemented, at least partially, as software executed by various types of processors. Identifying units of executable code can, for example, comprise one or more physical or logical blocks of computer instructions, which can be organized, for example, as objects, procedures, or functions. However, the executable program of the identified module need not be physically located together, but can include entirely different instructions stored in different locations, which, when logically combined, comprise the module and achieve the module's purpose. Furthermore, modules can be stored on computer-readable media, such as hard disk drives, flash memory devices, RAM, magnetic tape, and / or any other such non-transitory computer-readable media for storing data, without departing from the scope of the invention.

[0067] In practice, an executable code module can be a single instruction or multiple instructions, and can even be distributed across several different code segments, different programs, and across several storage devices. Similarly, operational data can be identified and represented within the module herein, and can be materialized in any suitable form and organized within any suitable type of data structure. Operational data can be collected as a single dataset, or it can be distributed across different locations including different storage devices, and can exist at least in part as electrical signals within the system or network.

[0068] In some embodiments, information about the performance of the AI / ML model is obtained. This information can be obtained from a single computing system of a local AI / ML model tailored for that user or computing system, or from multiple or many computing systems of an AI / ML model designed for broader application (e.g., providing group-level, company-level, global-level, etc. solutions), from the AI / ML model itself, from applications or other processes monitoring the AI / ML model, or from any combination thereof. This information can help determine how well the AI / ML model performs over time. For example, this information may include, but is not limited to, what predictions were made (e.g., the output from the AI / ML model), how many predictions the AI / ML model made and how many were actually used, one or more statistical distributions of the predictions (e.g., normal, binomial, or any other statistical distribution), the input data provided to the AI / ML model, any combination thereof, etc. In some embodiments, the statistical distribution is mapped to actions taken by the RPA robot in its workflow (e.g., mapped to activities).

[0069] After collecting this information, it is analyzed to determine whether data drift, model drift, or both have occurred. An alert or retraining trigger is generated when a change condition (e.g., predicted or input data falling outside historical range) and / or a change threshold (e.g., model statistical performance deviating from historical performance by at least a certain amount) is detected. In some embodiments, the change condition or change threshold must be met for at least a certain time period or number of instances before the AI / ML model performance is considered to have drifted. In some embodiments, the distance between the one-dimensional distribution of the original dataset and the new dataset can be measured for covariate drift. Furthermore, if a version of the AI / ML model already exists, the distance between remaining drifts can be examined.

[0070] The collected information and / or other data can then be used to retrain the AI / ML model in an attempt to improve its performance. In some embodiments, when the retrained AI / ML model meets certain thresholds, it can be automatically deployed alongside the existing AI / ML model, and model drift can be measured side-by-side to see if the new version of the AI / ML model performs better on real-world data. In some embodiments, the new version of the AI / ML model is not deployed unless it meets these thresholds.

[0071] In some embodiments, canary testing is used. For example, a new version of the AI / ML model can be used by a relatively small number of users who may not be aware of the changes to the AI / ML model (e.g., 10% of users each time, based on geographic location or some other filter). If the new version of the AI / ML model does not perform as well as the previous version, the new version can be rolled back and retraining can be performed.

[0072] In some embodiments, a champion / challenger technique can be used, where input data is sent to both a new version and a previous version of the AI / ML model, but only one set of outputs is used. This can help analyze how the new and previous versions of the AI / ML model perform on a similar-to-similar basis based on some fundamental facts. This technique can help determine which AI / ML model version is correct or closer to being correct more often. Furthermore, this technique can provide information helpful for troubleshooting regarding why the previous version of the AI / ML model exhibited certain behaviors.

[0073] In some embodiments, the automation platform provides and implements a general framework that addresses what information should be fed to AI / ML models invoked by RPA bots from workflow activities and / or executed by RPA bots within workflow activities. This framework helps ensure that the AI / ML models used in the RPA platform consume the expected data. It also helps ensure that the AI / ML models receive input in an appropriate format to enforce certain field requirements and / or values.

[0074] Figure 6 This is an architectural diagram illustrating a system 600 configured to perform AI / ML model drift detection and correction for RPA according to an embodiment of the present invention. System 600 includes user computing systems such as desktop computers 602, 603, tablets 604, and smartphones 606. However, any desired computing system can be used without departing from the scope of the invention, including but not limited to smartwatches, laptops, Internet of Things (IoT) devices, vehicle computing systems, etc. Furthermore, although in Figure 6 Three user computing systems are shown, but any suitable number of computing systems can be used without departing from the scope of the invention. For example, in some embodiments, tens, hundreds, thousands, or millions of computing systems may be used.

[0075] Each computing system 602, 604, 606 has an RPA robot 610 running thereon, which invokes and / or executes ML models (e.g., a model in AI / ML model 632 of server 630, a local ML model on computing systems 602, 604, 606, and ML models located in any other location, etc.). In some embodiments, RPA robot 610 may be a robot generated via an RPA designer application. In this embodiment, RPA robot 610 sends requests and possible input data to AI / ML model 632. However, in some embodiments, in addition to or in lieu of input data from RPA robot 610, AI / ML model 632 may obtain input data from one or more other sources (e.g., database 640, another computing system, etc.).

[0076] RPA robot 610 sends AI / ML model requests to AI / ML model 632 on server 630 via network 620 (e.g., local area network (LAN), mobile communication network, satellite communication network, Internet, any combination thereof, etc.) and receives AI / ML model execution results from it. Server 630 collects information about model drift for AI / ML model 632 and stores it in database 640. This information may relate to data drift, model drift, or both.

[0077] When a change condition is detected in one of the AI / ML models 632 (e.g., predicted or input data falls outside historical range) and / or a change threshold is met or exceeded (e.g., model statistical performance deviates from historical performance by at least a certain amount), in some embodiments, server 630 sends an alert or retraining trigger to application 652 running on reviewer computing system 650 (e.g., as part of a queue for review and labeling). In some embodiments, application 652 may enable reviewers to view information related to AI / ML model drift and correct results (e.g., for model drift) and / or identify that the nature of the input data has changed (e.g., data drift). In some embodiments, application 652 may enable ML operators who "own" the ML model in production to see model drift in a two-dimensional graph. For example, PCA can be used to perform dimensionality reduction to visually display certain anomalies. When one of these anomalies occurs, certain alerts may also be created to label it.

[0078] In some embodiments, application 652 can have further layers of abstraction, where data is presented to business users in a way that the business users can understand (e.g., a 2-D graph showing how business users change their analysis of enterprise data). This can be presented as a supplement to or alternative to the view for technically more skilled ML operators. For example, a word cloud mechanism can be used to display results that appear more frequently in larger fonts and make it easier for humans to see which results are most popular.

[0079] In some embodiments, significant changes can be highlighted. For example, business users might see that the AI / ML model used for loan approval has jumped from approving 5 loans per day to approving 20 loans per day. If the problem is persistent, business users can then define rules to attempt to correct the model behavior at their end until retraining occurs. Business users can also determine if there are anomalies not caused by the performance of the AI / ML model (e.g., the number of loan applications has increased, more qualified applicants are applying, etc.). Humans can also use Application 652 to identify intrinsic changes that cause bias but are not detected by the automatic drift detection mechanism. In some embodiments, there can be two layers—one for technically skilled users and another for business users.

[0080] In some embodiments, multiple or numerous computing systems and review applications, including computing system 650 and application 652, can be used to jointly review and correct the results. In some embodiments, these results are retrieved from a common queue. These corrections can then be provided back to server 630 and stored in database 640 for retraining the drifting AI / ML model 632. However, in some embodiments, server 630 can automatically detect AI / ML model drift occurring due to changes in input data and / or changes in AI / ML model predictions.

[0081] After AI / ML data and / or model drift is detected by server 630 (and in some embodiments, after manual correction), the drifting AI / ML model 632 is retrained. In some embodiments, retraining is performed by server 630. However, in some embodiments, retraining may be performed by one or more other servers besides or instead of server 630. The collected information and / or other information can then be used to retrain the drifting AI / ML model 632 in an attempt to improve AI / ML model performance.

[0082] After a new version of AI / ML model 632 is properly trained (e.g., when the retrained AI / ML model 632 meets certain thresholds), server 630 may store the new version of AI / ML model 632 in database 640, potentially replacing a previous version of the AI / ML model. However, in some embodiments, multiple versions of AI / ML model 632 may be stored. Server 630 then deploys the retrained AI / ML model 632 for invocation by RPA robot 610. However, in some embodiments, the retrained AI / ML model 632 is deployed along with one or more previous versions of AI / ML model 632. The drift of each version of AI / ML model 632 is then measured side-by-side to see if the new version of AI / ML model 632 performs better on real-world data. If it is determined that this is a time-varying phenomenon, the new version of AI / ML model 632 may replace the previous version of AI / ML model 632. However, if not, the new version of AI / ML model may be discarded, or further retraining may be performed.

[0083] Figure 7A and Figure 7BThis is a flowchart illustrating a process 700 of performing AI / ML model drift detection and correction for RPA according to an embodiment of the present invention. At 705, the process receives a request from the RPA robot for executing an AI / ML model and input information for the AI / ML model. At 710, verification is performed to determine whether the input information conforms to the framework for the AI / ML model. If the input information does not conform to the framework at 715, an error indicating that the input information does not conform to the framework is provided to the RPA robot at 720. At 725, if the input information conforms to the framework at 715, the AI / ML model is executed using the input information, and the execution result of the AI / ML model is returned to the RPA robot. In some embodiments, the framework is implemented for multiple AI / ML models.

[0084] At 730, information related to the input data for the AI / ML model is analyzed to determine whether data drift has occurred, information related to the execution results of the AI / ML model to determine whether model drift has occurred, or both. In some embodiments, this analysis includes measuring one or more distances between one-dimensional distributions of the original dataset and the new dataset for covariate drift. In some embodiments, this analysis includes determining whether statistical moments have changed. In some embodiments, statistical moments include mean, variance, skewness, kurtosis, covariance, or combinations thereof. In some embodiments, this information includes what predictions the AI / ML model makes, how many predictions the AI / ML model makes, and how many predictions are used by the RPA robot, one or more statistical distributions of the predictions made by the AI / ML model, the input data provided to the AI / ML model, or combinations thereof. In some embodiments, the analysis of information (e.g., via joint learning) is performed across multiple RPA processes executed by multiple respective RPA robots.

[0085] In some embodiments, the information includes one or more statistical distributions predicted by an AI / ML model. In such an embodiment, at 735, the one or more statistical distributions can be mapped to corresponding actions taken by multiple RPA robots. In some embodiments, actions are taken in corresponding activities of the RPA workflow of the respective RPA robots.

[0086] At 740, a dimensionality reduction technique is performed on the information to generate a two-dimensional representation of the information, and the two-dimensional representation of the information is displayed. At 745, based on the analysis of the information, verification is performed to determine whether a change condition has been detected, a change threshold has been met or exceeded, or both. In some embodiments, verification includes verifying that the change condition, the change threshold, or both have been met or exceeded for at least a predetermined time period.

[0087] If, at point 750, analysis of the information reveals a change in conditions, a change threshold being met or exceeded, or both, the AI / ML model is retrained at point 755. Then, at point 760, the retrained AI / ML model is verified to meet one or more performance thresholds. If one or more performance thresholds are not met at point 765, the process returns to retraining the AI / ML model at point 755. However, if the retrained AI / ML model meets one or more performance thresholds at point 765, the retrained AI / ML model is deployed at point 770. In some embodiments, deploying the AI / ML model may include modifying one or more activities of one or more RPA bots to invoke the new AI / ML model, replacing the previous version of the AI / ML model with the retrained AI / ML model. In some embodiments, canary testing or champion / challenger testing techniques are used to deploy the AI / ML model.

[0088] At 775, model drift is measured using real data for both the retrained AI / ML model and a previous version of the AI / ML model. At 780, if the retrained AI / ML model does not perform better on the real data than the previous version of the AI / ML model based on the measured model drift, the retrained AI / ML model is discarded at 785 and / or the process returns to step 755 for further retraining. However, if at 780, based on the measured model drift, the retrained AI / ML model performs better on the real data than the previous version of the AI / ML model, the retrained AI / ML model is used at 790 in addition to or in place of the previous version of the AI / ML model. At 795, the retrained AI / ML model or a previous version of the AI / ML model is selected and invoked based on the input data and / or use case. In some embodiments, the selection is performed in a hierarchical manner.

[0089] In some embodiments, model drift detection can be performed on each process or group of processes. For example, it is preferable to use a specific version of the AI / ML model for a specific type of processing. It can be observed that when an AI / ML model is trained on input data with specific characteristics, it becomes less accurate for another type of input data. For example, consider an AI / ML model trained using European date formats (e.g., November 17, 2019 or 17 / 11 / 2019) for processing invoices. If more US date formats (e.g., November 17, 2019 or 11 / 17 / 2019) are used during training, the AI / ML model may begin to fail for European date format invoices.

[0090] Therefore, in some embodiments, different versions of the AI / ML model can be trained to adapt to different situations, and drift detection can be used to select the AI / ML model version that works best for a given data. In some embodiments, AI / ML model versions can be applied in a hierarchical manner based on drift detection at the RPA workflow level. Logic can be developed to determine which AI / ML model version to use based on the type of input, or multiple AI / ML models can be tried in a hierarchical manner, and the result can be selected if certain criteria are met. This can also provide insight into how drift occurs among multiple AI / ML model versions. This can help determine whether an AI / ML model is bad for one use case and not bad for others. In some embodiments, an AI / ML model chaining process can be constructed, where deterministic logic manages the AI / ML model versions used in the hierarchical structure. In some embodiments, another AI / ML model can determine which AI / ML model version to use. In some embodiments, federated learning can be used, where multiple computing systems collaboratively learn a shared predictive model while keeping the training data on local devices. This can be particularly helpful in ensuring compliance with RPA management requirements (e.g., the EU General Data Protection Regulation (GDPR), the US Health Insurance Portability and Accountability Act (HIPAA), third-party service terms, etc.).

[0091] According to an embodiment of the present invention, in Figure 7A and Figure 7B The processing steps performed in the process can be executed by a computer program that encodes instructions for (multiple) processors to execute in the process. Figure 7A and Figure 7B The computer program may be contained on at least a portion of the processing described herein. The computer program may be contained on a non-transitory computer-readable medium. The computer-readable medium may be, but is not limited to, hard disk drives, flash memory devices, RAM, magnetic tape, and / or any other such medium or combination of media for storing data. The computer program may include processor(s) for controlling a computing system (e.g., ...). Figure 5 The computing system 500 has (multiple) processors 510 to achieve Figure 7A and Figure 7B The coded instructions for all or part of the processing steps described herein may also be stored on a computer-readable medium.

[0092] Computer programs can be implemented in hardware, software, or a hybrid approach. A computer program can consist of modules that can operatively communicate with each other, designed to pass information or instructions to a display. A computer program can be configured to operate on a general-purpose computer, an ASIC, or any other suitable device.

[0093] It is readily understood that the components of the various embodiments of the present invention, as generally described and illustrated in the accompanying drawings, can be arranged and designed in a wide variety of different configurations. Therefore, the detailed description of the embodiments of the invention shown in the drawings is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the invention.

[0094] The features, structures, or characteristics of the invention described throughout this specification can be combined in any suitable manner in one or more embodiments. For example, references to "certain embodiments," "some embodiments," or similar language throughout the specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. Therefore, the appearance of the phrases "in some embodiments," "in some embodiments," "in other embodiments," or similar language throughout this specification does not necessarily refer to the same group of embodiments, and the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0095] It should be noted that references to features, advantages, or similar language throughout the specification do not imply that all features and advantages achievable with the invention should be or be present in any single embodiment of the invention. Rather, references to features and advantages should be understood as indicating that a particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Therefore, the discussion of features and advantages, as well as similar language, throughout the specification may, but does not necessarily, refer to the same embodiments.

[0096] Furthermore, the features, advantages, and characteristics described in this invention can be combined in one or more embodiments in any suitable manner. Those skilled in the art will recognize that the invention can be practiced without one or more specific features or advantages of a particular embodiment. In other instances, additional features and advantages that may not be present in all embodiments of the invention may be recognized in certain embodiments.

[0097] It will be readily understood by those skilled in the art that the present invention as described above can be implemented with steps in a different order and / or with hardware elements in a configuration different from the disclosed configuration. Therefore, although the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative constructions will be readily apparent while remaining within the spirit and scope of the invention. Therefore, reference should be made to the appended claims to determine the limits and scope of the invention.

Claims

1. A computer program, embodied on a non-transitory computer-readable medium, for performing artificial intelligence (AI) / machine learning (ML) model drift detection and correction for robotic process automation (RPA), the computer program configured to cause at least one processor to: analyze information related to input data to an AI / ML model executed by one or more RPA robots to determine whether data drift has occurred, analyze information related to results from execution of the AI / ML model to determine whether model drift has occurred, or both, wherein the information comprises one or more statistical distributions of predictions made by the AI / ML model; map the one or more statistical distributions of the predictions made by the AI / ML model to respective actions taken by the one or more RPA robots in respective activities of an RPA workflow; and retrain the AI / ML model based on the analysis of the information when a change condition is found, when a change threshold is met or exceeded, or both.

2. The computer program of claim 1, wherein the program is further configured to cause the at least one processor to: verify whether the change condition, the change threshold, or both have been met or exceeded for at least a predetermined period of time.

3. The computer program of claim 1, wherein the analysis comprises determining whether a statistical moment changed.

4. The computer program of claim 3, wherein the statistical moment comprises a mean, a variance, a skew, a kurtosis, a covariance, or a combination thereof.

5. The computer program of claim 1, wherein the information comprises what predictions were made by the AI / ML model, how many predictions were made by the AI / ML model, and how many predictions were used by RPA robots, one or more statistical distributions of the predictions made by the AI / ML model, input data provided to the AI / ML model, or a combination thereof.

7. The computer program of claim 1, the computer program further configured to cause the at least one processor to: verify whether the retrained AI / ML model meets one or more performance thresholds; and deploy the retrained AI / ML model when the retrained AI / ML model meets the one or more performance thresholds.

3. The computer program of claim 1, wherein the analysis comprises:

8. The computer program of claim 7, wherein the retrained AI / ML model is deployed with a previous version of the AI / ML model, and the program is further configured to cause the at least one processor to: measure model drift of the retrained AI / ML model and the previous version of the AI / ML model using actual data; and use the retrained AI / ML model instead of the previous version of the AI / ML model when the retrained AI / ML model performs better than the previous version of the AI / ML model on the actual data based on the measured model drift. ​ ​ 6. The computer program of claim 1, wherein the information comprises: ​ ​ ​ ​ ​ ​ ​ 9. The computer program of claim 8, wherein, when the AI / ML model retrained based on the measured model drift does not perform better than the previous version of the AI / ML model on the actual data, the computer program is further configured to cause the at least one processor to: discard the AI / ML model retrained or perform further retraining.

10. The computer program of claim 7, wherein the AI / ML model retrained is deployed using a canary testing technique or a champion / challenger testing technique.

11. The computer program of claim 1, wherein the computer program is further configured to cause the at least one processor to: receive, from an RPA robot, a request to execute the AI / ML model and input information for the AI / ML model; verify whether the input information complies with a framework for the AI / ML model; and when the input information complies with the framework: execute the AI / ML model using the input information, and return results of the execution of the AI / ML model to the RPA robot.

12. The computer program of claim 11, wherein, when the input information does not comply with the framework, the computer program is further configured to cause the at least one processor to: provide an error to the RPA robot indicating that the input information does not comply with the framework.

13. The computer program of claim 11, wherein the framework is implemented for a plurality of AI / ML models.

14. The computer program of claim 1, wherein the computer program is further configured to cause the at least one processor to: perform a dimensionality reduction technique on the information to generate a two- dimensional representation of the information; and display the two-dimensional representation of the information.

15. The computer program of claim 1, wherein the analysis of the information is performed across a plurality of RPA processes performed by a plurality of respective RPA robots.

16. The computer program of claim 1, wherein the computer program is further configured to cause the at least one processor to: based on the input data, select the AI / ML model retrained or a previous version of the AI / ML model; and invoke the selected version of the AI / ML model.

17. The computer program of claim 16, wherein the selection is performed in a hierarchical manner.

18. A computer-implemented method for performing artificial intelligence (AI) / machine learning (ML) model drift detection and correction for robot process automation (RPA), comprising: analyzing information related to input data for an AI / ML model executed by one or more RPA robots to determine whether data drift has occurred, analyzing information related to results from the execution of the AI / ML model to determine whether model drift has occurred, or both, wherein the information comprises one or more statistical distributions of predictions made by the AI / ML model; mapping the one or more statistical distributions of the predictions made by the AI / ML model to respective actions taken by the one or more RPA robots in respective activities of an RPA workflow; and based on the analysis of the information, when a change condition is found, when a change threshold is met or exceeded, or both: re-training the AI / ML model, verifying whether the re-trained AI / ML model meets one or more performance thresholds, and deploying the re-trained AI / ML model when the re-trained AI / ML model meets the one or more performance thresholds.

19. The computer-implemented method of claim 18, further comprising: verifying whether the change condition, the change threshold, or both have been met or exceeded for at least a predetermined period of time.

20. The computer-implemented method of claim 18, wherein the re-trained AI / ML model is deployed with a previous version of the AI / ML model, and the method further comprises: measuring model drift of the re-trained AI / ML model and the previous version of the AI / ML model using actual data; and using the re-trained AI / ML model in place of the previous version of the AI / ML model when the re-trained AI / ML model performs better than the previous version of the AI / ML model on the actual data based on the measured model drift.

21. The computer-implemented method of claim 18, further comprising: receiving, from an RPA robot, a request to execute the AI / ML model and input information for the AI / ML model; verifying whether the input information conforms to a framework for the AI / ML model; and when the input information conforms to the framework: executing the AI / ML model using the input information, and returning results of the execution of the AI / ML model to the RPA robot, wherein the framework is implemented for a plurality of AI / ML models.

22. The computer-implemented method of claim 21, wherein when the input information does not conform to the framework, the method further comprises: providing an error to the RPA robot indicating that the input information does not conform to the framework.

23. The computer-implemented method of claim 18, further comprising: performing a dimensionality reduction technique on the information to generate a two- dimensional representation of the information; and displaying the two-dimensional representation of the information.

24. The computer-implemented method of claim 18, wherein the analysis of the information is performed across a plurality of RPA processes performed by a plurality of respective RPA robots.

25. The computer-implemented method of claim 18, further comprising: based on the input data, selecting the re-trained AI / ML model or a previous version of the AI / ML model; and invoking the selected version of the AI / ML model.

26. A computer-implemented method for performing artificial intelligence (AI) / machine learning (ML) model drift detection and correction for robotic process automation (RPA), comprising: analyzing information comprising one or more statistical distributions of predictions made by an AI / ML model executed by one or more RPA robots, the information relating to input data to the AI / ML model to determine whether data drift has occurred, results from execution of the AI / ML model to determine whether model drift has occurred, or both, wherein the information comprises one or more statistical distributions of predictions made by the AI / ML model; mapping the one or more statistical distributions of the predictions made by the AI / ML model to respective actions taken by the one or more RPA robots in respective activities of an RPA workflow; and based on the analysis of the information, retraining the AI / ML model when a change condition is found for at least a time period, when a change threshold is met or exceeded for at least a time period, or both.

27. The computer-implemented method of claim 26, wherein the information comprises how many predictions the AI / ML model makes and how many predictions are used by the plurality of RPA robots.

28. The computer-implemented method of claim 26, further comprising: receiving, from an RPA robot of the plurality of RPA robots, a request to execute the AI / ML model and input information for the AI / ML model; verifying whether the input information conforms to a framework for the AI / ML model; and when the input information conforms to the framework: executing the AI / ML model using the input information, and returning results of the execution of the AI / ML model to the RPA robot, wherein the framework is implemented for a plurality of AI / ML models.

Images