Machine learning model retraining pipeline for robotic process automation

Abstract

A machine learning (ML) model retraining pipeline for robotic process automation (RPA) is disclosed. When an ML model is deployed in a production or development environment, an RPA robot sends requests to the ML model while executing its workflow. When a confidence level of the ML model falls below a certain confidence, training data is potentially collected from a large number of computing systems. The ML model is then trained at least in part using the collected training data, and a new version of the ML model is deployed.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. nonprovisional patent application No. 16 / 864,000, filed April 30, 2020. The subject matter of that earlier application is incorporated herein by reference in its entirety. Technical Field

[0003] This invention generally relates to robotic process automation (RPA), and more specifically to a machine learning (ML) model retraining pipeline for RPA. Background Technology

[0004] Currently, there is no standard mechanism or process for automating feedback loops to retrain ML models. Therefore, improved solutions can be beneficial. Summary of the Invention

[0005] Certain embodiments of the present invention may provide solutions to problems and needs in the art that have not yet been fully identified, understood, or resolved by current RPA technologies. For example, some embodiments of the present invention relate to an ML model retraining pipeline for RPA.

[0006] In one embodiment, a computer-implemented method for implementing an ML model retraining pipeline for RPA includes: an RPA robot invoking an ML model; and the RPA robot receiving the execution results from the ML model. The computer-implemented method further includes: the RPA robot determining whether one or more trigger conditions are met for labeling data for the ML model. When one or more trigger conditions are met, the computer-implemented method further includes: the RPA robot prompting a user to provide labeled data for training or retraining the ML model, and sending the labeled data to a server for training or retraining the ML model; or the RPA robot adding information related to the results from the ML model to a queue for subsequent labeling.

[0007] In another embodiment, a computer-implemented method for implementing an ML model retraining pipeline for RPA includes: an RPA robot determining whether one or more trigger conditions are met for labeling data for the ML model. When one or more trigger conditions are met, the computer-implemented method further includes: prompting a user to provide labeled data for training or retraining the ML model, and sending the labeled data to a server for training or retraining the ML model; or the RPA robot adding information related to the results from the ML model to a queue for subsequent labeling.

[0008] In another embodiment, a computer-implemented method for implementing an ML model retraining pipeline for RPA includes: an RPA robot invoking an ML model; and the RPA robot receiving the results of execution from the ML model. The computer-implemented method further includes: the RPA robot determining whether one or more trigger conditions are met for labeling data for the ML model. When one or more trigger conditions are met, the computer-implemented method further includes: the RPA robot prompting a user to provide labeled data for training or retraining the ML model, and sending the labeled data to a server for training or retraining the ML model; or the RPA robot adding information related to the results from the ML model to a queue for subsequent labeling. When one or more training criteria are met, the computer-implemented method further includes: the server using the labeled data to train the ML model to produce a new version of the ML model; and the server deploying the new version of the ML model. Attached Figure Description

[0009] To facilitate understanding of the advantages of certain embodiments of the invention, a more specific description of the invention briefly described above will be presented with reference to specific embodiments illustrated in the accompanying drawings. While it should be understood that these drawings depict only exemplary embodiments of the invention and are therefore not intended to be limiting of the scope of the invention, the invention will be described and explained with additional specificity and detail through the use of the drawings, in which:

[0010] Figure 1 This is an illustration of the architecture of an RPA system according to an embodiment of the present invention.

[0011] Figure 2 This is an illustration of the architecture of an RPA system deployed according to an embodiment of the present invention.

[0012] Figure 3 This is an architecture diagram illustrating the relationship between the designer, activities, and drivers according to an embodiment of the present invention.

[0013] Figure 4 This is an illustration of the architecture of an RPA system according to an embodiment of the present invention.

[0014] Figure 5 This is an illustration of the architecture of a computing system configured to implement part of an ML model retraining pipeline for RPA according to an embodiment of the present invention.

[0015] Figure 6 This is an illustration of the architecture of a system configured to implement an ML model retraining pipeline for RPA according to an embodiment of the present invention.

[0016] Figure 7This is a flowchart illustrating a process for a pipeline for retraining an ML model for RPA according to an embodiment of the present invention. Detailed Implementation

[0017] Some embodiments involve ML model retraining pipelines for RPA. When an ML model is deployed in a production (i.e., runtime) or development environment, an RPA bot can send requests to the ML model as it executes its workflow. However, ML model performance can degrade over time and / or deviate from expected performance characteristics as various conditions change. For example, consider the scenario where an RPA bot invokes an ML model trained to identify dogs. The ML model initially has a 99% confidence threshold, but as the ML model is used more widely in more dog images and new breeds, the confidence threshold drops to 95%.

[0018] Some embodiments employ one or more triggers to initiate the collection of labeled data for retraining. Without departing from the scope of the invention, such triggers may include, but are not limited to: ML model performance falling below a confidence threshold, ML model results deviating from a statistical distribution (e.g., ML model predictions no longer falling within a normal distribution relative to a set of users), ML model results falling outside the previously observed range, human users (e.g., RPA developers) noticing undesirable performance characteristics and initiating the collection of labeled data for retraining, and / or any other suitable trigger. Unlike deterministic logic, other types of mechanisms should be employed where there is always a specific logical path that can provide “anomalies” for probabilistic ML models.

[0019] Therefore, in some embodiments, the timing of triggering labeled data collection (i.e., "human in the loop") is trigger-based. Returning to the example above of the ML model identifying the dog, human user input can be requested when the confidence threshold drops below 95%. For example, an interface or option for correction can be presented to the human user of the computing system on which the RPA robot is executing. For instance, the human user can review the labeling made by the ML model (e.g., "not a dog") and can make corrections if the labeling is incorrect. The incorrect image and the user's corrections can be sent as training data to a server (e.g., a cloud-based system, data center, corporate server, multiple servers of the RPA provider, etc.), and the RPA robot can continue executing the rest of the workflow.

[0020] Some embodiments can be employed for both manned and unmanaged robots. In the context of manned automation, humans can label data and work with the RPA robot. Consider a manned RPA robot running in the background and categorizing contract terms for lawyers. When retraining is triggered, the lawyer can be prompted to select the type of terms, and this categorization can become part of the labeled data for retraining. Thus, in some embodiments, labeling of the ML model invoked by the manned RPA robot can be synchronous.

[0021] In the context of unattended automation, an RPA bot can send a request to an ML model, which returns a prediction and a confidence level for that prediction. The unattended RPA bot can then evaluate whether a confidence threshold and / or some other criterion / standard is met. If not, the unattended RPA bot can add the labeling task to a task queue for later execution by the user. Therefore, in some embodiments, labeling for the ML model invoked by the unattended RPA bot can be asynchronous.

[0022] The server can receive and store training data from various users. Over time, a batch of labeled training data is collected. The collected training data can be used to train or retrain the ML model when a predetermined time period has elapsed, when a predetermined amount of training data is received, when a predetermined number of training data samples are received, when the data falls outside a certain distribution (e.g., normal, binomial, or any other statistical distribution), when custom criteria or standards are used, or any combination thereof. Training can occur using only the collected training data, or using the collected training data along with previous training data used to train the current version of the ML model. Potentially, a new version of the ML model can be deployed once trained or retrained without human intervention.

[0023] In some embodiments, a new version of the ML model can simply replace the previous version, and alternatively, a call to the ML model from the RPA bot workflow can invoke the new version of the ML model. In some embodiments, the new version of the ML model can be rolled out to the computing system of the RPA bot's user. In some embodiments, the RPA bot workflow can be modified to invoke the new version of the ML model. In some embodiments, the RPA bot can first try the new version of the ML model, and if a certain confidence level is not achieved, try the previous version of the ML model to see if the previous version achieved a higher confidence level. In some embodiments, the RPA bot can invoke two versions of the ML model and select the output from the model with the highest confidence level.

[0024] Some embodiments provide a framework for supervising ML models, which has standard mechanisms for enabling humans to correct ML model predictions to improve accuracy. In some embodiments, the user may not be aware that labeled data is being captured for training / retraining. In some embodiments, this standard framework can be used to collect training data and improve the accuracy of ML models for various business processes without requiring programming knowledge.

[0025] In some embodiments, the circumstances and timing of sample collection are dependent on the ML model. If acceptable confidence thresholds are used for the ML model, then business or technical decisions can be made regarding these confidence thresholds. If false negatives are very unfavorable (such as in medical diagnostics, vehicle control models, etc.), then the thresholds can be high (e.g., 99.9%, 99.9999%, etc.). If transaction speed is important, then a higher error rate may be acceptable (e.g., 75%, 85%, 90%, 95%, etc.). If false positives are more acceptable than false negatives (and vice versa), then this also influences which conditions will make training or retraining the ML model more desirable.

[0026] Figure 1 This diagram illustrates the architecture of an RPA system 100 according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that allows developers to design and implement workflows. The designer 110 can provide solutions for application integration and automation of third-party applications, management of information technology (IT) tasks, and business IT processes. The designer 110 can support the development of automation projects, which are graphical representations of business processes. In short, the designer 110 supports the development and deployment of workflows and robots.

[0027] Automation projects automate rule-based processes by giving developers control over the execution order and relationships between a set of custom steps (defined herein as "activities") developed within a workflow. A commercial example of an embodiment of Designer 110 is UiPath Studio. TM Each activity can include actions such as clicking a button, reading a file, or writing to a log panel. In some embodiments, workflows can be nested or embedded.

[0028] Some types of workflows may include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly well-suited for linear processes, enabling flow from one activity to another without disrupting the workflow. Flowcharts may be particularly well-suited for more complex business logic, allowing for the integration of decisions made in more distinct ways through multiple branching logic operators, as well as the connection of activities. FSMs may be particularly well-suited for larger workflows. FSMs can utilize a limited number of states in their execution, triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly well-suited for determining workflow behavior when execution errors are encountered and are especially suitable for debugging processes.

[0029] Once the workflow is developed in designer 110, the execution of the business process is orchestrated by conductor 120, which orchestrates one or more robots 130 to execute the workflow developed in designer 110. A commercial example of a conductor 120 is the UiPath Orchestrator. TM Commander 120 supports the creation, monitoring, and deployment management of resources in the environment. Utilizing third-party solutions and applications, Commander 120 can act as an integration point.

[0030] Commander 120 can manage a fleet of robots 130, thereby connecting and executing the robots 130 from a central point. The types of robots 130 that can be managed include, but are not limited to: attended robots 132, unattended robots 134, development robots (similar to unattended robots 134, but used for development and testing purposes), and non-production robots (similar to attended robots 132, but used for development and testing purposes). Attended robots 132 are triggered by user events and operate on the same computing system as humans. Attended robots 132 can be used with commander 120 for centralized process deployment and recording media. Attended robots 132 can assist human users in completing various tasks and can be triggered by user events. In some embodiments, processes cannot be started from commander 120 on this type of robot and / or these processes cannot run under a locked screen. In some embodiments, attended robots 132 can only be started from a robot tray or upon command prompt. In some embodiments, attended robots 132 should operate under human supervision.

[0031] Unattended robot 134 operates unattended in a virtual environment and can automate many processes. Unattended robot 134 can be responsible for remote execution, monitoring, scheduling, and supporting work queues. In some embodiments, debugging for all robot types can be run in designer 110. Both manned and unattended robots can automate various systems and applications, including but not limited to mainframes, web applications, VMs, and enterprise applications (e.g., those powered by...). Sales Enterprise applications (such as those used in production) and computing system applications (such as desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0032] Commander 120 may have various capabilities, including but not limited to provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between robot 130 and commander 120 (e.g., a web application). Provisioning may include ensuring that package versions are correctly delivered to the assigned robot 130 for execution. Configuration may include maintaining and delivering robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs to a database (e.g., an SQL database) and / or another storage mechanism (e.g., Elastic). It provides the ability to store and quickly query large datasets. The Commander 120 can provide interconnectivity by acting as a central point for communication between third-party solutions and / or applications.

[0033] Robot 130 is an execution agent that runs workflows built into Designer 110. A commercial example of some embodiments of (multiple) robots 130 is UiPath Robots. TM In some embodiments, robot 130 has Microsoft installed by default. Services managed by the Service Control Manager (SCM). As a result, this type of robot 130 can open interactive windows under the local system account. Conversation, and has Service permissions.

[0034] In some embodiments, robot 130 can be installed in user mode. For such a robot 130, this means it has the same permissions as a user who has already installed the given robot 130. This feature can also be used with high-density (HD) robots, which ensure that each machine is fully utilized to its maximum potential. In some embodiments, any type of robot 130 can be configured in an HD environment.

[0035] In some embodiments, robot 130 is divided into several components, each dedicated to a specific automation task. In some embodiments, robot components include, but are not limited to, SCM-managed robot services, user-mode robot services, actuators, agents, and command lines. SCM-managed robot service management and monitoring. The session acts as a delegate between the controller 120 and the execution host (i.e., the computing system on which the robot 130 executes). These services are trusted by the robot 130 and manage the robot 130's credentials. The console application is launched by the SCM on the local system.

[0036] In some embodiments, user-mode robot service management and monitoring The session acts as a delegate between the command center 120 and the execution host. The user-mode robot service can be trusted by the robot 130 and manage the robot 130's credentials. Without an SCM-managed robot service installed, The application can be launched automatically.

[0037] The actuator can be A given job runs within a session (i.e., the executor can execute a workflow). The executor can know the dots per inch (DPI) setting for each monitor. An agent can be... Presentation Foundation (WPF) applications display available jobs in a system tray window. Agents can be clients of the service. Agents can request to start or stop jobs and change settings. The command line is a client of the service. The command line is a console application that can request to start a job and wait for its output.

[0038] As explained above, separating the components of robot 130 generally helps developers, support users, and the computing system to more easily run, identify, and track what each component is performing. Special behaviors can be configured per component in this way, such as setting different firewall rules for executors and services. In some embodiments, the executor can always know the DPI setting of each monitor. As a result, the workflow can be executed at any DPI, regardless of the configuration of the computing system that created the workflow. In some embodiments, projects from designer 110 can also be independent of browser zoom levels. In some embodiments, DPI can be disabled for applications that are unaware of or intentionally marked as unaware of the DPI.

[0039] Figure 2 This is an architectural diagram illustrating an RPA system 200 deployed according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be... Figure 1 The RPA system 100 may be part of the RPA system. It should be noted that, without departing from the scope of the invention, the client side, server side, or both may include any desired number of computing systems. On the client side, the robot application 210 includes an actuator 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not run on the computing system 210. The actuator 212 is running the process. Figure 2 As shown, several business items can run simultaneously. In this embodiment, agent 214 (e.g., The service is a single point of contact for all executors 212. In this embodiment, all messages are logged to the controller 230, which further processes these messages via a database server 240, an indexer server 250, or both. (As described above relative to...) Figure 1 The actuator 212 discussed may be a robot component.

[0040] In some embodiments, a robot represents an association between a machine name and a username. A robot can manage multiple actuators simultaneously. In computing systems that support multiple concurrently running interactive sessions (e.g., On a server (2012), multiple bots can run simultaneously, each using a unique username. During the conversation. This will be referred to as the HD robot above.

[0041] Agent 214 is also responsible for sending the robot's status (e.g., periodically sending "heartbeat" messages indicating that the robot is still operating) and downloading the required version of the package to be executed. In some embodiments, communication between agent 214 and controller 230 is always initiated by agent 214. In notification scenarios, agent 214 may open a WebSocket channel, which controller 230 later uses to send commands to the robot (e.g., start, stop, etc.).

[0042] On the server side, there are a presentation layer (web application 232, Open Data Protocol (OData) Representative State Transfer (REST) ​​Application Programming Interface (API) endpoint 234, and notification and monitoring 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240 and indexer server 250). The controller 230 includes the web application 232, the ODataREST API endpoint 234, notification and monitoring 236, and the API implementation / business logic 238. In some embodiments, most actions performed by the user in the interface of the controller 220 (e.g., via browser 220) are performed by calling various APIs. Such actions may include, but are not limited to, starting a job on a robot, adding / removing data from a queue, scheduling a job to run unattended, etc., without departing from the scope of the invention. The web application 232 is the visual layer of the server platform. In this embodiment, the web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the invention. In this embodiment, the user interacts with a webpage from web application 232 via browser 220 to perform various actions to control the controller 230. For example, the user can create robot groups, assign packages to robots, analyze logs of each robot and / or each process, start and stop robots, etc.

[0043] In addition to web application 232, the controller 230 also includes a service layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the invention. The REST API is consumed by both web application 232 and agent 214. In this embodiment, agent 214 is a supervisor of one or more bots on a client computer.

[0044] In this embodiment, the REST API covers configuration, logging, monitoring, and queuing functionality. In some embodiments, the configuration endpoint can be used to define and configure application users, permissions, bots, assets, publications, and environments. The logging REST endpoint can be used to log various information, such as errors, explicit messages sent by bots, and other environment-specific information. If a start job command is used in the controller 230, the bot can use the deployment REST endpoint to query the package version that should be executed. The queuing REST endpoint can be responsible for queue and queue item management, such as adding data to the queue, retrieving transactions from the queue, and setting the status of transactions.

[0045] The monitoring REST endpoint can monitor web application 232 and agent 214. The notification and monitoring API 236 can be a REST endpoint used to register agent 214, deliver configuration settings to agent 214, and send / receive notifications from the server and agent 214. In some embodiments, the notification and monitoring API 236 can also use WebSocket communication.

[0046] In this embodiment, the persistence layer includes a pair of servers—a database server 240 (e.g., an SQL server) and an indexer server 250. In this embodiment, the database server 240 stores configurations for robots, robot groups, associated processes, users, roles, schedules, etc. In some embodiments, this information is managed via a web application 232. The database server 240 can manage queues and queue items. In some embodiments, the database server 240 can store messages logged by robots (in addition to or in place of the indexer server 250).

[0047] In some embodiments, the optional indexer server 250 stores information logged by the robot logs and indexes that information. In some embodiments, the indexer server 250 can be disabled through configuration settings. In some embodiments, the indexer server 250 uses Elasticsearch. It is an open-source full-text search engine project. Messages logged by the robot (such as activities using log messages or writing lines) can be sent to indexer server 250 via (multiple) logging REST endpoints, where these messages are indexed for future use.

[0048] Figure 3This is an architectural diagram illustrating the relationship 300 between designer 310, activities 320, 330, and driver 340 according to an embodiment of the present invention. As described above, developers use designer 310 to develop workflows executed by a robot. Workflows may include user-defined activities 320 and UI automation activities 330. Some embodiments are capable of identifying non-textual visual components in an image, referred to herein as computer vision (CV). Some CV activities associated with such components may include, but are not limited to, clicking, typing, text retrieval, hovering, element presence, refresh range, highlighting, etc. In some embodiments, clicking identifies an element using, for example, CV, optical character recognition (OCR), fuzzy text matching, and multi-anchor, and the element is clicked. Typing can use the above to identify an element and the type within that element. Retrieving text can identify the location of specific text and scan that location using OCR. Hovering can identify an element and hover over it. Element presence can use the techniques described above to check if an element exists on the screen. In some embodiments, hundreds or even thousands of activities that can be implemented in designer 310 may exist. However, any number and / or type of activity may be available without departing from the scope of the invention.

[0049] UI automation activities 330 are a subset of specialized low-level activities (such as CV activities) written using low-level code and that facilitate interaction with the screen. UI automation activities 330 support these interactions via drivers 340 that allow the robot to interact with the desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc.

[0050] Driver 340 can perform low-level interactions with the OS, thereby locating hooks, monitoring keys, etc. The driver can facilitate... Integration of features such as "click" activity, which plays the same role in these different applications via driver 340.

[0051] Figure 4 This is an architectural diagram illustrating an RPA system 400 according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be or include Figure 1 and / or Figure 2 The RPA system 100 and / or 200. The RPA system 400 includes multiple client computing systems 410 that operate the robot. The computing systems 410 are capable of communicating with the command computing system 420 via a web application running thereon. The command computing system 420 is also capable of communicating with a database server 430 and an optional indexer server 440.

[0052] Compared to Figure 1 and Figure 3 It should be noted that while web applications are used in these embodiments, any suitable client / server software can be used without departing from the scope of the invention. For example, the controller can run a server-side application that communicates with non-web-based client software applications on the client computing system.

[0053] Figure 5 This is an architectural diagram illustrating a computing system configured to implement part of an ML model retraining pipeline for RPA according to an embodiment of the present invention. In some embodiments, computing system 500 may be one or more of the computing systems depicted and / or described herein. Computing system 500 includes a bus 505 or other communication mechanism for transmitting information, and processor(s) 510 coupled to bus 505 to process information. Processors(s) 510 may be any type of general-purpose or special-purpose processor, including a central processing unit (CPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. Processors(s) 510 may also have multiple processing cores, and at least some cores may be configured to perform specific functions. In some embodiments, multi-parallel processing may be used. In some embodiments, at least one of processors(s) 510 may be a neuromorphic circuit including processing elements that mimic biological nerves. In some embodiments, neuromorphic circuits may not require typical components of a von Neumann computing architecture.

[0054] The computing system 500 also includes a memory 515 for storing information and instructions to be executed by the processor(s) 510. The memory 515 may consist of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage devices (such as disks or optical discs), or any other type of non-transitory computer-readable medium or any combination thereof. The non-transitory computer-readable medium may be any available medium accessible by the processor(s) 510 and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both.

[0055] Additionally, the computing system 500 includes a communication device 520 (such as a transceiver) to provide access to a communication network via a wireless and / or wired connection. In some embodiments, without departing from the scope of the invention, the communication device 520 may be configured to use Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiplexing (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), CDMA2000, Wideband CDMA (W-CDMA), High-Speed ​​Downlink Packet Access (HSD). High-speed uplink packet access (HSUPA), high-speed packet access (HSPA), Long Term Evolution (LTE), LTE Advanced (LTE-A), 802.11x, Wi-Fi, Zigbee, Ultra Wideband (UWB), 802.16x, 802.15, Home Node B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near Field Communication (NFC), 5G, New Radio (NR), any combination thereof and / or any other existing or future communication standards and / or protocols. In some embodiments, without departing from the scope of the invention, the communication device 520 may include one or more antennas, which may be singular, arrayed, phased, switched, beamformed, beamguided, combinations thereof and / or any other antenna configuration.

[0056] Multiple processors 510 are further coupled to a display 525 via a bus 505, such as a plasma display, liquid crystal display (LCD), light-emitting diode (LED) display, field emission display (FED), organic light-emitting diode (OLED) display, flexible OLED display, flexible substrate display, projection display, 4K display, high-definition display, etc. The display 525 may be a touch (haptic) display, an in-plane switching (IPS) display, or any other suitable display for displaying information to a user. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, etc., using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse recognition, suppressed total internal reflection, etc., employing resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse recognition, suppressed total internal reflection, etc., without departing from the scope of this invention. Any suitable display device and haptic I / O may be used without departing from the scope of this invention.

[0057] Keyboard 530 and cursor control devices 535 (such as a computer mouse, touchpad, etc.) are further coupled to bus 505 to enable a user to interface with computing system 500. However, in some embodiments, a physical keyboard and mouse may be absent, and the user may interact with the device solely through display 525 and / or touchpad (not shown). Any type and combination of input devices can be considered as a design choice. In some embodiments, there are no physical input devices and / or displays. For example, a user may interact remotely with computing system 500 via another computing system with which it communicates, or computing system 500 may operate autonomously.

[0058] Memory 515 stores software modules that provide functionality when executed by processor(s) 510. These modules include an operating system 540 of the computing system 500. The modules also include an ML model retraining pipeline module 545 configured to perform all or part of the processes described herein or derivatives thereof. The computing system 500 may include one or more additional functional modules 550 that include additional functionality.

[0059] Those skilled in the art will understand that, without departing from the scope of this invention, the "system" can be implemented as a server, embedded computing system, personal computer, console, personal digital assistant (PDA), mobile phone, tablet computing device, quantum computing system, or any other suitable computing device or combination of devices. Presenting the functions described above as being performed by the "system" is not intended to limit the scope of the invention in any way, but rather to provide an example of many embodiments of the invention. In fact, the methods, systems, and apparatuses disclosed herein can be implemented in a localized and distributed manner consistent with computing technologies including cloud computing systems.

[0060] It should be noted that some system features described in this specification have been presented as modules to more specifically emphasize their implementation independence. For example, modules can be implemented as hardware circuits that include custom-designed very large-scale integration (VLSI) circuits or gate arrays, off-the-shelf semiconductors (such as logic chips, transistors, or other discrete components). Modules can also be implemented in programmable hardware devices, such as field-programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, etc.

[0061] Modules can also be implemented, at least partially, in software for execution by various types of processors. For example, the identified units of executable code may include one or more physical or logical blocks of computer instructions, which may be organized, for example, into objects, procedures, or functions. However, the executable file of the identified module does not need to be physically located in one place, but may include entirely different instructions stored in different locations that, when logically combined, comprise the module and achieve its intended purpose. Furthermore, modules can be stored on a computer-readable medium, which, without departing from the scope of the invention, may be, for example, a hard disk drive, flash memory device, RAM, magnetic tape, and / or any other such non-transitory computer-readable medium for storing data.

[0062] In practice, a module of executable code can be a single instruction or many instructions, and can even be distributed across several different code segments, different programs, and multiple memory devices. Similarly, operational data can be identified and illustrated herein as being within a module, and this operational data can be implemented in any suitable form and organized within any suitable type of data structure. Operational data can be collected as a single dataset, or it can be distributed across different locations on different storage devices, and can exist at least in part solely as electronic signals on a system or network.

[0063] Figure 6 This illustration shows an architecture diagram of a system 600 configured to implement an ML model retraining pipeline for RPA according to an embodiment of the present invention. System 600 includes user computing systems such as desktop computers 602, 603, tablets 604, and smartphones 606. However, any desired computing system can be used without departing from the scope of the invention, including but not limited to: smartwatches, laptops, Internet of Things (IoT) devices, vehicle computing systems, etc. Furthermore, in Figure 6 The four user computing systems shown are illustrated, but any suitable number of computing systems can be used without departing from the scope of the invention. For example, in some embodiments, dozens, hundreds, thousands, or millions of computing systems may be used.

[0064] Each computing system 602, 603, 604, 606 runs a digital process 610 that invokes and / or executes an ML model. Without departing from the scope of the invention, the digital process 610 may be a robot generated via an RPA designer application, may be part of an operating system, may be a downloadable application for a personal computer (PC) or smartphone, or may be any other software and / or hardware. In fact, in some embodiments, the logic of one or more digital processors in the digital processor 610 is partially or completely implemented via physical hardware. The digital processor 610 may collect and send training data for training / retraining the ML model in response to triggers. Without departing from the scope of the invention, triggers may include, but are not limited to: ML model performance falling below a confidence threshold, ML model results deviating from a statistical distribution (e.g., ML model predictions no longer falling within a normal distribution relative to a set of users), ML model results falling outside the previously observed range, human users (e.g., RPA developers) noticing undesirable performance characteristics and initiating the collection of labeled data for retraining, and / or any other suitable trigger.

[0065] Digital process 610 sends training data collected from users of its corresponding computing system to server 630 via network 620 (e.g., local area network (LAN), mobile communication network, satellite communication network, Internet, or any combination thereof). Server 630 stores the training data in database 640. The training data may include, but is not limited to: images and user corrections (e.g., areas of components on the screen, textual descriptions of errors from users, markings of areas, etc.).

[0066] When instructed by a human user (e.g., an RPA engineer or data scientist), when a predetermined amount of training data has been collected, when a predetermined amount of time has elapsed since the last training, or when a training request is received, the server 630 runs the training data through multiple AI layers 632. The AI ​​layers 632 process the data and generate new versions of the ML model. The AI ​​layers 632 can perform statistical modeling (e.g., Hidden Markov Models (HMMs)) and utilize deep learning techniques (e.g., Long Short-Term Memory (LSTM) deep learning, encoding of previous hidden states, etc.).

[0067] In some embodiments, each AI layer 632 is an algorithm (or model) running on training data, and the ML model itself can be a deep learning neural network (DLNN) of trained artificial “neurons” trained on the training data. Layers can run sequentially, in parallel, or in combination thereof. AI layer 632 can include, but is not limited to: sequence extraction layers, clustering detection layers, visual component detection layers, text recognition layers (e.g., OCR), audio-to-text conversion layers, multiple instances of one or more types of layers, or any combination thereof. Any desired number and (multiple) types of layers can be used without departing from the context of the invention. Using multiple layers allows the system to have a global depiction of what is happening on the screen or in a process. For example, one AI layer can perform OCR, another AI layer can detect buttons, etc. Patterns can be determined individually by AI layers or can be determined jointly by multiple AI layers.

[0068] In some embodiments, data scientists can initiate and / or monitor the training of the ML model. For example, data scientists can supplement the training dataset with additional data, or retrain the ML model with different or supplementary training data if the newly trained ML model fails to reach the desired confidence level. In some embodiments, data scientists can generate update notifications when the ML model is ready and approved. In some embodiments, server 630 can generate notifications.

[0069] After the new version of the ML model has been properly trained, server 630 can save the new version of the ML model in database 640, potentially replacing the previous version of the ML model. The new version of the ML model is executed when it is invoked from user computing systems 602, 603, 604, and 606. In some embodiments, the new version of the ML model can be rolled out to computing systems 602, 603, 604, and 606. In some embodiments, the workflow of the RPA robot invoking the ML model can be modified to invoke the new version of the ML model. In some embodiments, for a given scenario, the RPA robot can first try the new version of the ML model, and if a certain confidence threshold is not met, try the previous version of the ML model to see if the previous version achieved a higher confidence level. In some embodiments, the RPA robot can invoke two versions of the ML model and select the output from the model with the highest confidence level.

[0070] Figure 7This is a flowchart illustrating process 700 of a pipeline for retraining an ML model for RPA according to an embodiment of the present invention. At 710, the process begins by invoking an ML model from an RPA bot workflow executed on a user's computing system. At 720, the RPA bot receives outputs from the ML model (e.g., predictions and confidence thresholds, variables, and statistical distributions, etc.) and at 730 determines whether one or more triggering conditions are met. Without departing from the scope of the invention, triggering conditions may include, but are not limited to: ML model performance falling below a confidence threshold, ML model results deviating from a statistical distribution (e.g., ML model predictions no longer falling within a normal distribution relative to a set of users), ML model results falling outside the previously observed range, human users (e.g., RPA developers) noticing undesirable performance characteristics and initiating the collection of labeled data for retraining, and / or any other suitable trigger.

[0071] If multiple trigger conditions are met at point 730, then labeled training data is obtained at point 740 and submitted to the server. For example, for an attended RPA bot, an interface or options can be presented to the human user of the computing system on which the RPA bot is executing, for correction; or information about where an identifier below a confidence threshold occurred can be provided to the human user of another computing system, and an interface or options can be presented to that human user for correction. For an unattended RPA bot, a screenshot can be automatically captured, and coordinates of a given component on the screen where the ML model is applied can be provided. This information can then be added to a queue for subsequent review and labeling.

[0072] When one or more training criteria are met at 750, the ML model is trained at 760 using the collected training data or using the collected training data and previous training data from the current version used to train the ML model. One or more training criteria may include, but are not limited to: when a predetermined time period has elapsed, when a predetermined amount of training data is received, when a predetermined number of training data samples are received, when the data falls outside a certain distribution of the data (e.g., a normal distribution, a binomial distribution, or any other statistical distribution), custom criteria or standards, any combination thereof, etc. In some embodiments, in the absence of a link to... Figure 7 The training criteria are checked in a separate process for the other steps shown.

[0073] After training, a new version of the ML model is deployed at 770. In some embodiments, the new version of the ML model can simply replace the previous version, and calls to the ML model from the RPA bot workflow can invoke the new version of the ML model without further modification. However, in some embodiments, further modifications are made at 780 to allow the RPA bot to interact with the new version of the ML model. For example, in some embodiments, the new version of the ML model can be rolled out to the computing system of the RPA bot's user. In some embodiments, the RPA bot workflow can be modified to invoke the new version of the ML model.

[0074] In some embodiments, at 790, the RPA bot may invoke multiple versions of the ML model. For example, in some embodiments, the RPA bot may first try a new version of the ML model, and if a certain confidence level is not achieved, try one or more previous versions of the ML model to see if the previous versions achieve a higher confidence level or better results (e.g., better conformity to a statistical distribution, values ​​falling within a range, faster execution, etc.). In some embodiments, the RPA bot may invoke multiple versions of the ML model (e.g., a new version of the ML model and one or more previous versions of the ML model) and select the output from the model with the highest confidence level or better results.

[0075] According to an embodiment of the present invention, Figure 7 The process steps executed in the process can be performed by a computer program that is encoded with instructions that cause (multiple) processors to execute them. Figure 7 The computer program may be implemented on a non-transient computer-readable medium, including but not limited to hard disk drives, flash memory devices, RAM, magnetic tape, and / or any other such medium or combination of media for storing data. The computer program may include processors (e.g., multiple processors) for controlling a computing system. Figure 5 The computing system 500 has (multiple) processors 510 to achieve Figure 7 The coded instructions, in whole or in part, of the process steps described herein, may also be stored on a computer-readable medium.

[0076] Computer programs can be implemented in hardware, software, or a hybrid implementation. A computer program can consist of modules that can operatively communicate with each other and are designed to exchange information or instructions for display. A computer program can be configured to operate on a general-purpose computer, an ASIC, or any other suitable device.

[0077] It will be readily understood that, as generally described and illustrated in the figures herein, the components of various embodiments of the invention can be arranged and designed in a variety of different configurations. Therefore, the detailed description of embodiments of the invention as illustrated in the figures is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention.

[0078] The features, structures, or characteristics of the invention described throughout this specification can be combined in any suitable manner in one or more embodiments. For example, throughout this specification, references to "certain embodiments," "some embodiments," or similar language mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Therefore, throughout this specification, the phrases "in some embodiments," "in some embodiments," "in other embodiments," or similar language do not necessarily all refer to the same set of embodiments, and the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0079] It should be noted that throughout this specification, references to features, advantages, or similar language do not imply that all features and advantages achievable with the invention should be, or all exist, in any single embodiment of the invention. Rather, language relating to features and advantages is understood to mean that a particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Therefore, throughout this specification, discussions of features and advantages, as well as similar language, may, but are not necessarily, of the same embodiments.

[0080] Furthermore, the features, structures, and characteristics described in this invention can be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the invention can be practiced without having one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages that may not be present in all embodiments of the invention may be identified in some embodiments.

[0081] It will be readily understood by those skilled in the art that the invention as described above can be practiced using steps in different sequences and / or hardware elements with configurations different from those disclosed. Therefore, although the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative constructions will be apparent while still remaining within the spirit and scope of the invention. Therefore, reference should be made to the appended claims to determine the limits and boundaries of the invention.

Claims

1. A computer-based method for implementing a machine learning (ML) model retraining pipeline for robotic process automation (RPA), comprising: The ML model is invoked by the RPA robot when performing an RPA workflow that uses a machine learning ML model; The RPA robot receives the execution results from the ML model; The RPA robot completes the execution of the activities in the RPA workflow using the results of the execution from the ML model; The RPA robot determines whether one or more triggering conditions are met for labeling data for the ML model; as well as In response to the one or more triggering conditions being met: The RPA robot prompts the user to provide labeled data for training or retraining the ML model, and sends the labeled data to the server for training or retraining the ML model, or The RPA robot adds information related to the results from the ML model to a queue for subsequent labeling, wherein... The RPA workflow includes multiple activities, which may include one or more activities, and The one or more activities are a set of steps developed within the workflow.

2. The computer-implemented method according to claim 1, wherein the one or more triggering conditions include: ML model performance falling below the confidence threshold, ML model results deviating from the statistical distribution, ML model results falling outside the previously observed range, or any combination of the above.

3. The computer-implemented method of claim 1, wherein the one or more triggering conditions include receiving an initiation command for collecting the tagged data.

4. The computer-implemented method according to claim 1 further includes: The tagged data is received by the server; as well as The server stores the tagged data in a database.

5. The computer-implemented method according to claim 1, further comprising: In response to one or more training criteria being met, the server uses the labeled data to train the ML model to produce a new version of the ML model; as well as The new version of the ML model is deployed by the server.

6. The computer-implemented method according to claim 5, wherein... The training of the ML model additionally includes: The ML model is trained using previously trained data that was used to train a version of the ML model that was trained prior to the new version of the ML model.

7. The computer-implemented method of claim 5, wherein the one or more training criteria include: The predetermined time period, the predetermined amount of training data received, the predetermined number of training data samples received, the data falling outside the data distribution, or any combination of the above.

8. The computer-implemented method of claim 5, wherein the new version of the ML model replaces the previous version of the ML model.

9. The computer-implemented method according to claim 5, further comprising: Modify the workflow of the RPA robot to invoke the new version of the ML model.

10. The computer-implemented method according to claim 5, further comprising: The workflow of the RPA robot is modified to invoke multiple versions of the ML model, including the new version of the ML model.

11. The computer-implemented method according to claim 10, further comprising: The new version of the ML model is invoked by the RPA robot; as well as In response to one or more triggering conditions not being met, the RPA robot invokes a previous version of the ML model and selects the best result from the current version of the ML model and the previous version of the ML model.

12. The computer-implemented method according to claim 10, further comprising: The RPA robot invokes the new version of the ML model and at least one previous version of the ML model; as well as The RPA robot selects the best result from the current version of the ML model and the at least one previous version of the ML model.

13. The computer-implemented method of claim 1, wherein the collection of labeled data and the training of new versions of the ML models are unified for multiple ML models invoked by multiple RPA robots.

14. A computer implementation method for implementing a machine learning ML model retraining pipeline for robotic process automation (RPA), comprising: An RPA robot determines whether one or more trigger conditions are met for tagging data against an ML model. The RPA robot is configured to invoke the ML model in response to an activity performed by the RPA robot that uses an RPA workflow employing the ML model. In response to the one or more triggering conditions being met: The RPA robot prompts the user to provide labeled data for training or retraining the ML model, and sends the labeled data to the server for training or retraining the ML model, or The RPA robot adds information related to the results from the ML model to a queue for subsequent labeling, wherein The RPA workflow includes multiple activities, which may include one or more activities, and The one or more activities are a set of steps developed within the workflow.

15. The computer-implemented method of claim 14, wherein the one or more triggering conditions include: ML model performance falling below the confidence threshold, ML model results deviating from the statistical distribution, ML model results falling outside the previously observed range, or any combination of the above.

16. The computer-implemented method according to claim 14, further comprising: In response to one or more training criteria being met, the server uses the labeled data to train the ML model to produce a new version of the ML model; as well as The new version of the ML model is deployed by the server.

17. The computer-implemented method of claim 16, wherein the one or more training criteria comprise: The predetermined time period, the predetermined amount of training data received, the predetermined number of training data samples received, the data falling outside the data distribution, or any combination of the above.

18. The computer-implemented method according to claim 16, further comprising: The workflow of the RPA robot is modified to invoke multiple versions of the ML model, including the new version of the ML model.

19. A computer implementation method for implementing a machine learning ML model retraining pipeline for robotic process automation (RPA), comprising: The ML model is invoked by the RPA robot when performing an RPA workflow that uses a machine learning ML model; The RPA robot receives the execution results from the ML model; The RPA robot completes the execution of the activities in the RPA workflow using the results of the execution from the ML model; The RPA robot determines whether one or more triggering conditions are met for labeling data for the ML model; In response to the one or more triggering conditions being met: The RPA robot prompts the user to provide labeled data for training or retraining the ML model, and sends the labeled data to the server for training or retraining the ML model, or The RPA robot adds information related to the results from the ML model to a queue for subsequent labeling; In response to one or more training criteria being met, the server uses the labeled data to train the ML model to produce a new version of the ML model; as well as The new version of the ML model is deployed by the server, wherein The RPA workflow includes multiple activities, which may include one or more activities, and The one or more activities are a set of steps developed within the workflow.

20. The computer-implemented method according to claim 19, wherein... The one or more triggering conditions include: ML model performance falling below the confidence threshold, ML model results deviating from the statistical distribution, ML model results falling outside the previously observed range, or any combination of the above, and The one or more training criteria include: a predetermined time period, a predetermined amount of training data received, a predetermined number of training data samples received, the data falling outside the data distribution, or any combination of the above.

Images