Semantic matching between source screens or source data and target screens using semantic artificial intelligence

Abstract

Semantic matching between source screens or source data and target screens using semantic artificial intelligence (AI) for robotic process automation (RPA) workflows is disclosed. Source data or source screens and target screens are selected on a matching interface, and semantic matching is performed between the source data / screens and the target screens using artificial intelligence / machine learning (AI / ML) models, highlighting matching and non-matching graphical elements, allowing a developer to see which graphical elements match and which do not. The matching interface can also provide a confidence score for individual matches, provide an overall mapping score, and allow a developer to hide / show matching / non-matching graphical elements. RPA workflow activities can be automatically created based on the semantic mapping, which can be performed to execute the automation.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Non-Provisional Patent Application No. 17 / 494,744, filed October 5, 2021. The subject matter of this prior application is incorporated herein by reference in its entirety.

[0002] The present invention relates generally to semantic matching, and more particularly to semantic matching between source screens or source data and target screens using semantic artificial intelligence (AI) for robotic process automation (RPA) workflows. [Background technology]

[0003] Currently, developers must manually create robotic process automation (RPA) workflows in an RPA designer application using activities. While creating an RPA workflow, developers must indicate target graphical elements on the screen, and the RPA designer application then automatically generates a selector that corresponds to the target element with a set of anchors. While activity recommendation and suggestion functionality currently exists in UiPath Studio™, for example, it does not support fully automated RPA workflow creation. Manually indicating all of the target graphical elements while creating an RPA workflow is time-consuming. Therefore, an improved approach for creating RPA workflows would be beneficial. Summary of the Invention

[0004] Certain embodiments of the present invention may provide solutions to problems and needs in the art that have not yet been fully identified, recognized, or solved by current RPA workflow generation technology. For example, some embodiments of the present invention relate to semantic matching between source screens or source data and target screens using semantic AI for RPA workflows.

[0005] In one embodiment, a non-transitory computer-readable medium stores a computer program. The computer program is configured to cause at least one processor to receive a selection of a source screen or source data and to receive a selection of a target screen. The computer program is also configured to cause the at least one processor to invoke one or more AI / ML models trained to perform semantic matching between labels in the source screen and labels in the target screen, between data elements in the source data and labels in the target screen, or both. The computer program is further configured to cause the at least one processor to receive, from the one or more AI / ML models, representations of graphical elements associated with semantically-matching labels in the target screen and respective confidence scores, and to display the graphical elements associated with the semantically-matching labels on the target screen in a matching interface.

[0006] In another embodiment, a computer-implemented method for performing semantic AI for RPA includes invoking, by an RPA designer application, one or more AI / ML models trained to perform semantic matching between labels in a source screen and labels in a target screen, between data elements in the source data and labels in the target screen, or both. The computer-implemented method also includes receiving, by the RPA designer application, an indication from the one or more AI / ML models of graphical elements associated with semantically matching labels in the target screen and their respective confidence scores, and displaying, by the RPA designer application, the graphical elements associated with the semantically matching labels on the target screen in a matching interface. The computer-implemented method further includes automatically generating, by the RPA designer application, one or more activities in an RPA workflow that copy data from fields of the source screen or data elements of the source data to fields in the target screen having labels identified by the one or more AI / ML models as semantically matching labels from the source screen or data elements from the source data.

[0007] In yet another embodiment, a computing system includes a memory storing computer program instructions for performing semantic AI for RPA and at least one processor configured to execute the computer program instructions. The computer program instructions are configured to cause the at least one processor to receive, from one or more AI / ML models trained to perform semantic matching between labels in a source screen and labels in a target screen, between data elements in the source data and labels in the target screen, an indication of graphical elements associated with semantically matching labels in the target screen and their respective confidence scores. The computer program instructions are also configured to cause the at least one processor to display the graphical elements associated with the semantically matching labels on the target screen in a matching interface. The computer program instructions are further configured to cause the at least one processor to receive modifications to graphical elements in the target screen identified by the one or more AI / ML models as having labels that match the associated semantics, receive an indication of new elements in the target screen that were not semantically matched by the one or more AI / ML models to the labels in the source screen, or both. Further, the computer program instructions are configured to cause the at least one processor to collect information regarding the corrected and / or newly labeled graphical elements in the target screen and associated labels, and to directly or indirectly store the collected information for retraining of one or more AI / ML models. [Brief explanation of the drawings]

[0008] So that the advantages of particular embodiments of the present invention may be readily understood, a more particular description of the invention, briefly described above, will be rendered by reference to specific embodiments that are illustrated in the accompanying drawings. It being understood that these drawings depict only typical embodiments of the invention and therefore should not be considered limiting of its scope, but the invention will be illustrated and explained with added specificity and detail through the use of the accompanying drawings.

[0009] [Figure 1] FIG. 1 is an architectural diagram illustrating an RPA system, according to one embodiment of the present invention.

[0010] [Figure 2] FIG. 1 is an architectural diagram illustrating a deployed RPA system according to one embodiment of the present invention.

[0011] [Figure 3] FIG. 2 is an architecture diagram illustrating the relationships between designers, activities, and drivers according to one embodiment of the present invention.

[0012] [Figure 4] FIG. 1 is an architectural diagram illustrating an RPA system, according to one embodiment of the present invention.

[0013] [Figure 5] FIG. 1 is an architecture diagram illustrating a computing system configured to perform semantic matching between source screens or source data and target screens using semantic AI for RPA workflows, according to one embodiment of the present invention.

[0014] [Figure 6] FIG. 1 is an architecture diagram illustrating a system configured to train an artificial intelligence / machine learning (AI / ML) model and perform semantic matching between source screens or source data and target screens using semantic AI for RPA workflows, according to one embodiment of the present invention.

[0015] [Figure 7A] FIG. 2 illustrates an example of a neural network trained to recognize graphical elements in an image, according to one embodiment of the present invention.

[0016] [Figure 7B] FIG. 2 illustrates an example of a neuron, according to one embodiment of the present invention.

[0017] [Figure 8] FIG. 1 is a flowchart illustrating a process for training an AI / ML model to perform semantic matching between a source screen or source data and a target screen using semantic AI for RPA workflows, according to an embodiment of the present invention.

[0018] [Figure 9A] FIG. 1 illustrates a matching interface for an RPA designer application according to one embodiment of the present invention. [Figure 9B] FIG. 1 illustrates a matching interface for an RPA designer application according to one embodiment of the present invention. [Figure 9C] FIG. 1 illustrates a matching interface for an RPA designer application according to one embodiment of the present invention. [Figure 9D] FIG. 1 illustrates a matching interface for an RPA designer application according to one embodiment of the present invention. [Figure 9E] FIG. 1 illustrates a matching interface for an RPA designer application according to one embodiment of the present invention. [Figure 9F] FIG. 1 illustrates a matching interface for an RPA designer application according to one embodiment of the present invention. [Figure 9G] FIG. 1 illustrates a matching interface for an RPA designer application according to one embodiment of the present invention.

[0019] [Figure 10] FIG. 1 illustrates an RPA designer application with an automatically generated RPA workflow according to one embodiment of the present invention.

[0020] [Figure 11A] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11B] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11C] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11D] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11E] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11F] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11G] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11H] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11I] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11J] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11K]1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11L] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11M] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention. [Figure 11N] 1A-1C illustrate screen shots of an exemplary semantic copy-and-paste interface, according to one embodiment of the present invention.

[0021] [Figure 12] FIG. 1 is an architecture diagram illustrating the architecture of an AI / ML model for performing semantic AI, according to one embodiment of the present invention.

[0022] [Figure 13] 1 is a flowchart illustrating a process for performing semantic matching between a source screen or source data and a target screen using semantic AI for RPA workflows according to one embodiment of the present invention.

[0023] [Figure 14] 1 is a flowchart illustrating a process for performing semantic matching between a source screen or source data and a target screen using semantic AI for use with an attended automation interface according to one embodiment of the present invention.

[0024] Unless otherwise indicated, like reference numerals refer to corresponding features consistently throughout the accompanying drawings. DETAILED DESCRIPTION OF THE INVENTION

[0025] Some embodiments relate to semantic matching between source screens or source data and target screens using semantic AI for RPA workflows. For example, in some embodiments, source data or source screens and target screens are selected on a matching interface, and semantic matching is performed between the source data / screens and the target screens using an artificial intelligence / machine learning (AI / ML) model. Matching and non-matching graphical elements are highlighted using different colors (e.g., green and red, respectively), allowing a developer to see which graphical elements match and which do not. The matching interface also has additional features, in some embodiments, such as providing a confidence score for each match, providing an overall mapping score, and allowing a developer to hide / show matching / non-matching graphical elements. Furthermore, one or more RPA workflow activities are automatically created based on the semantic mapping, which in some embodiments can be performed to execute semantic AI functions as part of automation executed by the RPA robot.

[0026] In some embodiments, the list of data fields can be obtained from an Excel® spreadsheet, a relational database, a flat file source, etc. The semantic matching AI / ML model can iterate through the entries in the data and enter them into a target screen. The semantic matching AI / ML model can be trained to do this regardless of the type of data source. Due to the semantic matching capabilities of this AI / ML model, a one-to-one match may not be necessary. For example, a natural language model may seek to match identical or similar names / phrases in the target screen with those in the source data (or start with the source data and look for similar names / phrases in the target screen). In certain embodiments, a broad set of training data is used to make the semantic matching AI / ML model more accurate, as there may be many similar words or phrases for a particular term, and also many different subsets depending on the context. In some embodiments, context may also be used. For example, a semantic matching AI / ML model can learn that a given target is related to banking details vs. invoices, vs. purchase orders vs. contact information, etc.

[0027] In some embodiments, the semantic matching AI / ML model may be deployed to assist the RPA developer during design time. However, in some embodiments, the semantic matching AI / ML model may be used at runtime to provide more robust functionality and self-healing. This is not common in some embodiments because UI descriptors tend to be significantly faster than semantic matching AI / ML models, and may be employed when the UI descriptor cannot identify the target graphical element at runtime. Therefore, UI descriptors should be used first for the same or similar target screens.

[0028] If a given target element cannot be identified by a given user interface (UI) descriptor at runtime, such as when a UI changes due to a new version of the target application, a semantic matching AI / ML model can be used to attempt to identify the target graphical element. This information can then be added as a synonym for the word or phrase of interest, and the UI descriptor for that graphical element can be updated so that the UI descriptor works going forward. If the user interface changes again, and the changed graphical element and / or anchor are sufficiently similar, the RPA robot can identify the target graphical element in the new version of the application. See, for example, U.S. Patent Application Publication No. 16 / 922,289.

[0029] A UI descriptor is a set of instructions for locating UI elements. In some embodiments, the UI descriptor is an encapsulated data / structure format that includes a UI element selector, an anchor selector, a computer vision (CV) descriptor, an integrated target descriptor, a screen image capture (context), an element image capture, other metadata (e.g., application and application version), a combination thereof, etc. The encapsulated data / structure format may be extensible with future updates to the platform and is not limited to the above definition. Any suitable UI descriptor for identifying UI elements on a screen may be used without departing from the scope of the present invention.

[0030] In some embodiments, what the semantic matching AI / ML model is detecting may be combined with a unified target descriptor for runtime detection. For such embodiments, in addition to the source and target words and phrases, once the mapping is confirmed, unified target information may be collected. At runtime, the unified target descriptor may be attempted first, and if unsuccessful, the semantic matching AI / ML model may be used.

[0031] Unified target descriptors tend to be more stable and accurate than AI / ML models. Unified target descriptors chain multiple types of UI descriptors together. Unified target information includes UI descriptor information that facilitates identification of graphical elements in the UI descriptors used.

[0032] The unified target descriptor can function like a finite state machine (FSM): in a first context, a first UI descriptor mechanism is applied, in a second context, a second UI descriptor mechanism is applied, and so on. In other words, the UI descriptor, in some embodiments, can function with a unified target that encompasses some or all UI detection mechanisms through which image detection and definition are performed. The unified target can integrate multiple technologies for identifying and automating UI elements into a single, cohesive approach. In some embodiments, the unified target can prioritize specific UI descriptor types, such as prioritizing selector-based and driver-based UI detection mechanisms and using CV as a fallback to find target UI elements if the first two mechanisms are unsuccessful.

[0033] In some embodiments, natural language processing (NLP) AI / ML models can be used in addition to or instead of semantic matching AI / ML models. In particular embodiments, these AI / ML models can be used together. For example, a match can be suggested to the user if one of the models meets or exceeds a certain threshold, if the average of both models meets or exceeds a threshold, etc.

[0034] In some embodiments, a feedback loop feature is provided. For example, when a user adds a match or modifies a match suggested by a semantic matching AI / ML model, information about the match can be saved. This information can include, but is not limited to, screenshots of the target application, labels for the target application and corresponding labels for the source screen or source data, labels for incorrect matches, etc. Context can also be captured, such as modifications made to a web page, SAP®, etc. This data can be used along with other labeled data collected in this way to retrain the semantic matching AI / ML model.

[0035] FIG. 1 is an architectural diagram illustrating an RPA system 100 according to one embodiment of the present invention. The RPA system 100 includes a designer 110 that enables developers to design and implement workflows. The designer 110 can provide application integration and solutions for automating third-party applications, management information technology (IT) tasks, and business IT processes. The designer 110 can facilitate the development of automation projects, which are graphical representations of business processes. Simply put, the designer 110 facilitates the development and deployment of workflows and robots.

[0036] Automation projects enable rule-based process automation by giving developers control over the order of execution and relationships between custom sets of steps developed in a workflow, defined herein as "activities." One commercial example of an embodiment of the designer 110 is UiPath Studio™. Each activity can include actions such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0037] Some types of workflows can include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly well-suited for linear processes, allowing for flow from one activity to another without disrupting the workflow. Flowcharts are particularly well-suited for more complex business logic, allowing for the integration of decisions and the connection of activities in more diverse ways through multiple branching logical operators. FSMs may be particularly well-suited for large workflows. FSMs can use a finite number of states in their execution, triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly well-suited for determining workflow behavior when an execution error is encountered and debugging the process.

[0038] Once a workflow is developed in Designer 110, the execution of the business process is orchestrated by Conductor 120, which orchestrates one or more Robots 130 that execute the workflow developed in Designer 110. One commercial example of an embodiment of Conductor 120 is UiPath Orchestrator™. Conductor 120 facilitates the creation, monitoring, and management of deployment of resources in an environment. Conductor 120 can serve as an integration point with third-party solutions and applications.

[0039] The conductor 120 can manage all robots 130, connecting and executing them from a centralized point. Types of robots 130 that can be managed include, but are not limited to, attended robots 132, unattended robots 134, development robots (similar to unattended robots 134 but used for development and testing purposes), and non-production robots (similar to attended robots 132 but used for development and testing purposes). Attended robots 132 are triggered by user events and operate alongside humans on the same computing system. Attended robots 132 can be used with the conductor 120 for centralized process deployment and recording. Attended robots 132 can help human users accomplish various tasks and can be triggered by user events. In some embodiments, processes cannot be started from the conductor 120 for this type of robot and / or cannot run under a locked screen. In certain embodiments, attended robots 132 can only be launched from the robot tray or command prompt. In some embodiments, the attended robot 132 should operate under human supervision.

[0040] Unattended robots 134 operate unattended in virtual environments and can automate many processes. Unattended robots 134 can be responsible for providing remote execution, monitoring, scheduling, and work queue support. In some embodiments, debugging of all robot types can be performed in designer 110. Both attended and unattended robots can automate a variety of systems and applications, including, but not limited to, mainframes, web applications, VMs, enterprise applications (e.g., those manufactured by SAP®, Salesforce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0041] The conductor 120 can have a variety of functions, including, but not limited to, provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning can include creating and maintaining connections between robots 130 and the conductor 120 (e.g., web applications). Deployment can include ensuring the correct delivery of package versions to robots 130 assigned to perform. Configuration can include maintaining and delivering robot environment and process configurations. Queuing can include providing management of queues and queue items. Monitoring can include tracking robot identification data and maintaining user permissions. Logging can include storing and indexing logs in a database (e.g., an SQL database) and / or another storage mechanism (e.g., ElasticSearch®, which stores large data sets and provides the ability to quickly query them). The conductor 120 can provide interconnectivity by acting as a centralized point of communication for third-party solutions and / or applications.

[0042] Robots 130 are execution agents that execute workflows built in designer 110. One commercial example of some embodiments of robots 130 is UiPath Robots™. In some embodiments, robots 130 install the Microsoft Windows Service Control Manager (SCM) management service by default. As a result, such robots 130 can open interactive Windows sessions under the local system account and have the rights of a Windows service.

[0043] In some embodiments, a robot 130 can be installed in user mode. For such a robot 130, this means that a given robot 130 has the same rights as the user to which it is installed. This feature may also be available for high-density (HD) robots, ensuring full utilization of each machine to its maximum potential. In some embodiments, any type of robot 130 can be configured in an HD environment.

[0044] In some embodiments, the robot 130 is divided into several components, each dedicated to a specific automation task. In some embodiments, the robot components include, but are not limited to, an SCM-managed robot service, a user-mode robot service, an executor, an agent, and a command line. The SCM-managed robot service manages and monitors Windows sessions and acts as a proxy between the conductor 120 and the execution host (i.e., the computing system on which the robot 130 executes). These services are trusted with and manage the credentials of the robot 130. A console application is launched by the SCM under Local System.

[0045] The user-mode robot service in some embodiments manages and monitors Windows sessions and acts as a proxy between the conductor 120 and the execution host. The user-mode robot service can trust and manage credentials for the robot 130. If the SCM management robot service is not installed, the Windows application can be launched automatically.

[0046] An Executor can execute a given job under a Windows session (i.e., an Executor can execute a workflow). An Executor can be aware of per-monitor dots per inch (DPI) settings. An Agent can be a Windows Presentation Foundation (WPF) application that displays available jobs in a system tray window. An Agent can be a client of a Service. An Agent can request to start or stop jobs and change settings. A Command Line is a client of a Service. A Command Line is a console application that can request the start of a job and wait for its output.

[0047] Dividing the robot 130 components as described above helps developers, support users, and computing systems more easily implement, identify, and track what each component is doing. In this way, special behaviors can be configured per component, such as setting different firewall rules for executors and services. Executors, in some embodiments, can always be aware of DPI settings per monitor. As a result, workflows can be executed at any DPI, regardless of the configuration of the computing system on which they were created. In some embodiments, projects from the designer 110 may be independent of the browser zoom level. For applications that are not DPI-aware or are intentionally marked as not-aware, DPI can be disabled in some embodiments.

[0048] FIG. 2 is an architecture diagram illustrating a deployed RPA system 200 according to one embodiment of the present invention. In some embodiments, the RPA system 200 may be or be part of the RPA system 100 of FIG. 1. Note that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 210 includes an executor 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not be running on the computing system 210. The executor 212 is a running process. As shown in FIG. 2, several business projects may run simultaneously. The agent 214 (e.g., a Windows service) is a single connection point for all executors 212 in this embodiment. All messages in this embodiment are logged to the conductor 230, which further processes them via the database server 240, the indexer server 250, or both. As discussed above with respect to FIG. 1, the executor 212 may be a robotic component.

[0049] In some embodiments, a robot represents an association between a machine name and a username. A robot can manage multiple executors simultaneously. In computing systems that support multiple interactive sessions running simultaneously (e.g., Windows Server 2012), multiple robots may run simultaneously, each running in a separate Windows session using a unique username. This is referred to above as an HD robot.

[0050] The agent 214 is also responsible for transmitting the robot's status (e.g., periodically sending "heartbeat" messages indicating that the robot is still functioning) and downloading necessary versions of packages to be fulfilled. Communication between the agent 214 and the conductor 230 is always initiated by the agent 214 in some embodiments. In notification scenarios, the agent 214 can open a WebSocket channel that is later used by the conductor 230 to send commands (e.g., start, stop, etc.) to the robot.

[0051] The server side includes a presentation layer (web application 232, Open Data Protocol (OData) Representational State Transfer (REST) ​​Application Programming Interface (API) endpoint 234, and Notifications and Monitoring 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240, indexer server 250). Conductor 230 includes web application 232, OData REST API endpoint 234, Notifications and Monitoring 236, and API implementation / business logic 238. In some embodiments, most actions a user performs within the conductor 230 interface (e.g., via browser 220) are performed by calling various APIs. Such actions may include, but are not limited to, starting a job on a robot, adding / removing data from a queue, scheduling a job for unattended execution, etc., without departing from the scope of the present invention. Web application 232 is the visual layer of the server platform. In this embodiment, web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the present invention. A user interacts with web pages from web application 232, in this embodiment via browser 220, to perform various actions to control conductor 230. For example, a user can create robot groups, assign packages to robots, analyze logs per robot and / or per process, start and stop robots, etc.

[0052] In addition to the web application 232, the conductor 230 also includes a services layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the present invention. The REST API is consumed by both the web application 232 and the agent 214, which in this embodiment is the administrator of one or more robots on a client computer.

[0053] The REST API in this embodiment covers configuration, logging, monitoring, and queuing functions. The configuration endpoint, in some embodiments, may be used to define and configure application users, permissions, robots, assets, releases, and environments. For example, a logging REST endpoint may be used to log various information such as errors, explicit messages sent by robots, and other environment-specific information. A deployment REST endpoint may be used by robots to query the package version that should be committed when a start job command is used within conductor 230. The queuing REST endpoint may be responsible for queue and queue item management, such as adding data to a queue, retrieving transactions from a queue, and setting the state of transactions.

[0054] Monitoring REST endpoints can monitor web applications 232 and agents 214. Notification monitoring API 236 may be a REST endpoint used to register agents 214, deliver configuration settings to agents 214, and send / receive notifications from the server and agents 214. Notification monitoring API 236 may also use WebSocket communication in some embodiments.

[0055] The persistence layer includes a pair of servers in this embodiment: database server 240 (e.g., SQL Server) and indexer server 250. Database server 240 in this embodiment stores configurations for robots, robot groups, associated processes, users, roles, schedules, etc. This information is managed in some embodiments via web application 232. Database server 240 can manage queues and queue items. In some embodiments, database server 240 can store messages logged by robots (in addition to or instead of indexer server 250).

[0056] Indexer server 250, which is optional in some embodiments, stores and indexes information logged by the robots. In certain embodiments, indexer server 250 can be disabled through a configuration setting. In some embodiments, indexer server 250 uses ElasticSearch®, a full-text search engine from an open source project. Messages logged by robots (e.g., using activities such as log messages or line writes) may be sent via a logging REST endpoint to indexer server 250, where they are indexed for future use.

[0057] FIG. 3 is an architecture diagram illustrating the relationship 300 between a designer 310, activities 320 and 330, a driver 340, and an AI / ML model 350, according to one embodiment of the present invention. Accordingly, a developer uses the designer 310 to develop a workflow to be performed by the robot. The workflow can include a user-defined activity 320 and a UI automation activity 330. The user-defined activity 320 and / or the UI automation activity 330, in some embodiments, can invoke one or more AI / ML models 350, which can be located locally and / or remotely to the computing system on which the robot is operating. Some embodiments can identify non-textual visual components within an image, referred to herein as computer vision (CV). Some CV activities related to such components may include, but are not limited to, click, type, get text, hover, element presence, refresh range, highlight, etc. In some embodiments, click identifies an element and clicks on it, for example, using CV, optical character recognition (OCR), fuzzy character matching, and multi-anchors. Type can identify an element using the above and types within elements. Text can be acquired and the location of specific text can be identified using OCR and then scanned. Hover can identify an element and hover over it. Element Presence can check whether an element is present on the screen using the techniques described above. In some embodiments, there may be hundreds or thousands of activities implemented in designer 310. However, any number and / or type of activities may be utilized without departing from the scope of the present invention.

[0058] UI automation activities 330 are a subset of specialized, low-level activities written in lower-level code (e.g., CV activities) that facilitate interactions with the screen. UI automation activities 330 facilitate these interactions via drivers 340 and / or AI / ML models 350 that enable the robot to interact with desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc. One or more of the AI / ML models 350 may be used by UI automation activities 330 to determine the execution of interactions with the computing system. In some embodiments, AI / ML models 350 may augment drivers 340 or replace them entirely. Indeed, in certain embodiments, drivers 340 are not included.

[0059] Drivers 340 can interact with the OS at a low level by looking for hooks, monitoring keys, etc. They can facilitate integration with Chrome®, IE®, Citrix®, SAP®, etc. For example, a "click" activity performs the same role in these different applications via drivers 340.

[0060] FIG. 4 is an architecture diagram illustrating an RPA system 400, according to one embodiment of the present invention. In some embodiments, RPA system 400 can be or include RPA systems 100 and / or 200 of FIGS. 1 and / or 2. RPA system 400 includes multiple client computing systems 410 that execute robots. Computing systems 410 can communicate with a conductor computing system 420 via web applications running thereon. Conductor computing system 420 can, in turn, communicate with a database server 430 and an optional indexer server 440.

[0061] 1 and 3, it should be noted that although web applications are used in these embodiments, any suitable client / server software may be used without departing from the scope of the present invention. For example, the conductor may run a server-side application that communicates with a non-web-based client software application on a client computing system.

[0062] FIG. 5 is an architecture diagram illustrating a computing system 500 configured to perform semantic matching between source screens / source data and target screens using semantic AI for RPA workflows, according to one embodiment of the present invention. In some embodiments, computing system 500 may be one or more of the computing systems illustrated and / or described herein. Computing system 500 includes a bus 505 or other communication mechanism for communicating information and a processor 510 coupled to bus 505 for processing information. Processor 510 may be any type of general-purpose or special-purpose processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. Processor 510 may also have multiple processing cores, at least some of which may be configured to perform specific functions. In some embodiments, multiple parallel processing may be used. In certain embodiments, at least one of processors 510 may be a neuromorphic circuit including processing elements that mimic biological neurons. In some embodiments, neuromorphic circuits may not require the typical components of a von Neumann computing architecture.

[0063] The computing system 500 further includes a memory 515 for storing information and instructions executed by the processor 510. The memory 515 may be comprised of any combination of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage such as a magnetic or optical disk, or any other type of non-transitory computer-readable medium, or a combination thereof. The non-transitory computer-readable medium may be any available medium that can be accessed by the processor 510 and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both.

[0064] Additionally, the computing system 500 includes a communications device 520, such as a transceiver, for providing access to a communications network via wireless and / or wired connections. In some embodiments, the communications device 520 may be configured to support a variety of communications technologies, including Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiplexing (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), cdma2000, Wideband CDMA (W-CDMA), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), High Speed ​​Packet Access (HSPA), and the like. The communication device 520 may be configured to use any of the following communication standards and / or protocols: High-Speed ​​Access (HSPA), Long Term Evolution (LTE), LTE-Advanced (LTE-A), 802.11x, Wi-Fi, Zigbee, Ultra Wideband (UWB), 802.16x, 802.15, Home Node B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near Field Communications (NFC), Fifth Generation (5G), New Radio (NR), any combination thereof, and / or any other currently existing or future implemented communication standards and / or protocols without departing from the scope of the present invention. In some embodiments, the communication device 520 may include one or more antennas that are single, array, phased, switched, beamforming, beamsteering, combinations thereof, and / or any other antenna configuration without departing from the scope of the present invention.

[0065] The processor 510 is further coupled via bus 505 to a display 525, such as a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, a Retina® display, an in-plane switching (IPS) display, or any other suitable display for displaying information to a user. The display 525 may be configured as a touch (tactile) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, etc., using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse recognition, frustrated total internal reflection, etc. Any suitable display device and tactile I / O may be used without departing from the scope of the invention.

[0066] A keyboard 530 and cursor control device 535, such as a computer mouse, touchpad, etc., are further coupled to bus 505 to allow a user to interface with computing system 500. However, in certain embodiments, a physical keyboard and mouse may not be present, and a user may interact with the device solely through display 525 and / or a touchpad (not shown). Any type and combination of input devices may be used as a matter of design choice. In certain embodiments, no physical input devices and / or displays are present. For example, a user may interact with computing system 500 remotely via another computing system that communicates with it, or computing system 500 may operate autonomously.

[0067] The memory 515 stores software modules that provide functionality when executed by the processor 510. The modules include an operating system 540 for the computing system 500. The modules further include a semantic matching module 545 configured to perform all or a portion of the processes described herein or derivatives thereof. The computing system 500 may include one or more additional functional modules 550 that include additional functionality.

[0068] Those skilled in the art will understand that a "system" may be embodied as a server, embedded computing system, personal computer, console, personal digital assistant (PDA), mobile phone, tablet computing device, quantum computing system, or any other suitable computing device or combination of devices without departing from the scope of the present invention. Presenting the above functions as being performed by a "system" is not intended to limit the scope of the present invention in any way, but rather to provide one example of many embodiments of the present invention. Indeed, the methods, systems, and apparatuses disclosed herein may be implemented in localized and distributed forms consistent with computing technologies, including cloud computing systems. The computing system may be part of or accessible by a local area network (LAN), a mobile communications network, a satellite communications network, the Internet, a public or private cloud, a hybrid cloud, a server farm, any combination thereof, or the like. Any localized or distributed architecture may be used without departing from the scope of the present invention.

[0069] Note that some of the system features described herein are presented as modules to more specifically emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom very large scale integrated (VLSI) circuits or off-the-shelf semiconductors such as gate arrays, logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, etc.

[0070] Modules may also be implemented, at least in part, in software for execution by various types of processors. An identified unit of executable code may comprise one or more physical or logical blocks of computer instructions, which may be organized, for example, as an object, procedure, or function. Nevertheless, the executable files of identified modules need not be physically located together, but may comprise different instructions stored in different locations that, when logically combined together, comprise a module and achieve the module's stated purpose. Furthermore, modules may be stored on a computer-readable medium, which may be, for example, a hard disk drive, a flash device, RAM, tape, and / or any other such non-transitory computer-readable medium used to store data without departing from the scope of the present invention.

[0071] Indeed, a module of executable code may be a single instruction, or many instructions, and may be distributed across several different code segments, different programs, and several memory devices. Similarly, operational data may be identified and illustrated herein in modules and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set or distributed in different locations, including different storage devices, and may exist, at least in part, solely as electronic signals on a system or network.

[0072] 6 is an architectural diagram illustrating a system 600 configured to use semantic AI for RPA workflows to train AI / ML models and perform semantic matching between source screens / source data and target screens, according to one embodiment of the present invention. System 600 includes user computing systems such as a desktop computer 602, a tablet 604, and a smartphone 606. However, any desired computing system, including but not limited to a smartwatch, a laptop computer, an Internet of Things (IoT) device, a vehicle computing system, etc., can be used without departing from the scope of the present invention.

[0073] Each computing system 602, 604, 606 has an RPA designer application installed. The RPA designer application 610 provides a mapping function that allows each user to select a source screen or source data and a target screen. The RPA designer application 610 is also configured to call an AI / ML model 632 on a server 630 via a network 620 (e.g., a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, any combination thereof, etc.). The server 630 stores data in a database 640 and retrieves data from the database.

[0074] The AI / ML models 632 provide semantic AI functions, CV, OCR, NLP, etc. For example, one AI / ML model may provide CV functions, another may perform OCR, and yet another may use this data to perform semantic matching, etc.

[0075] 7A shows an example of a neural network 700 trained to recognize graphical elements in an image, according to one embodiment of the present invention. Here, neural network 700 receives pixels of a 1920x1080 screen screenshot image for input "neurons" 1 through I of its input layer. In this case, I is 2,073,600, the total number of pixels in the screenshot image.

[0076] Neural network 700 also includes several hidden layers. Both DLNNs and SLNNs typically have multiple layers, although SLNNs may have only one or two layers and typically have fewer layers than DLNNs. Typically, a neural network architecture, as in the case of neural network 700, includes an input layer, several hidden layers, and an output layer.

[0077] DLNNs often have many layers (e.g., 10, 50, 200, etc.), with subsequent layers typically reusing features from previous layers to compute more complex and general functions. SLNNs, on the other hand, tend to have only a few layers and train relatively quickly because expert features are created in advance from raw data samples. However, feature extraction is laborious. DLNNs, on the other hand, typically do not require expert features, but they take longer to train and tend to have more layers.

[0078] For both approaches, layers are trained simultaneously on the training set, usually with a check for overfitting on a separate cross-validation set. Both techniques can produce excellent results, and there is considerable enthusiasm for both approaches. The optimal size, shape, and quantity of individual layers varies depending on the problem being addressed by each neural network.

[0079] Returning to Figure 7A, the pixels provided as the input layer are fed as inputs to J neurons in hidden layer 1. While in this example all pixels are fed to each neuron, a variety of architectures are possible that can be used individually or in combination, including, but not limited to, feedforward networks, radial basis networks, deep feedforward networks, deep convolutional inverse graphics networks, convolutional neural networks, recurrent neural networks, artificial neural networks, long / short-term memory networks, gated recurrent unit networks, generative adversarial networks, liquid state machines, autoencoders, variational autoencoders, denoising autoencoders, sparse autoencoders, extreme learning machines, echo state networks, Markov chains, Hopfield networks, Boltzmann machines, restricted Boltzmann machines, deep residual networks, Koenen networks, deep probabilistic networks, deep convolutional networks, support vector machines, neural Turing machines, or any other suitable type or combination of neural networks without departing from the scope of the present invention.

[0080] Hidden layer 2 receives input from hidden layer 1, hidden layer 3 receives input from hidden layer 2, and so on for all hidden layers until the last hidden layer provides its output as input for the output layer. Note that the numbers of neurons I, J, K, and L are not necessarily equal, and thus any desired number of layers may be used in a given layer of neural network 700 without departing from the scope of the present invention. Indeed, in certain embodiments, the neurons in a given layer may not all be of the same type.

[0081] The neural network 700 is trained to assign confidence scores to graphical elements believed to have been found in the image. To reduce matches with unacceptably low likelihood, some embodiments may provide only results with confidence scores that meet or exceed a confidence threshold. For example, if the confidence threshold is 80%, outputs with confidence scores above this amount may be used, and the rest may be ignored. In this case, the output layer may indicate that two text fields, a text label, and a submit button were found. The neural network 700 may provide the location, dimensions, image, and / or confidence score of these elements, which may then be used by the RPA robot or another process that uses this output for a given purpose, without departing from the scope of the present invention.

[0082] Note that neural networks are probabilistic constructs that typically have a confidence score. This can be a score learned by the AI / ML model based on how often similar inputs were correctly identified during training. For example, text fields often have a rectangular shape and a white background. A neural network can learn to identify graphical elements with these characteristics with high confidence. Some common types of confidence scores include a decimal number between 0 and 1 (which can be interpreted as a percentage of confidence), a number between negative infinity and positive infinity, or a set of formulas (e.g., "low," "medium," "high"). Various post-processing calibration techniques can also be used to obtain more accurate confidence scores, such as temperature scaling, batch normalization, weight decay, and negative log-likelihood (NLL).

[0083] A "neuron" in a neural network is a mathematical function typically based on the function of a biological neuron. Neurons receive weighted inputs and have summation and activation functions that govern whether they pass on an output to the next layer. This activation function can be a nonlinear thresholded activity function where nothing happens if the value is below a threshold, but the function responds linearly above the threshold (i.e., rectified linear unit (ReLU) nonlinearity). Summation and ReLU functions are used in deep learning because real neurons can have nearly similar activity functions. Through linear transformations, information can be subtracted, added, etc. Essentially, neurons act as gating functions that pass on outputs to the next layer as governed by their underlying mathematical function. In some embodiments, different functions can be used for at least some neurons.

[0084] JPEG0007800857000001.jpg69170

[0085] JPEG0007800857000002.jpg43170

[0086] JPEG0007800857000003.jpg27157

[0087] In this case, neuron 710 is a single-layer perceptron, however, any suitable neuron type or combination of neuron types may be used without departing from the scope of the present invention.

[0088] A goal or "reward function" is often used, such as in this case successfully identifying a graphical element in an image. The reward function guides the exploration of the state space, exploring intermediate transitions and steps that have both short-term and long-term rewards in an attempt to achieve a goal (e.g., successfully identifying a graphical element, successfully identifying the next set of activities in an RPA workflow, etc.).

[0089] During training, various labeled data (in this case, images) are fed through the neural network 700. Successful identification strengthens the input weights to the neurons, while failed identification weakens the neurons. Cost functions such as mean squared error (MSE) or gradient descent can be used to handle slightly erroneous predictions much less often than very erroneous predictions. If the performance of the AI / ML model is not improving after a certain number of training iterations, a data scientist can modify the reward function, provide an indication of where the misidentified graphical elements are, provide corrections for misidentified graphical elements, etc.

[0090] Backpropagation is a technique for optimizing synaptic weights in feedforward neural networks. Backpropagation can be used to "pop the hood" in the hidden layers of a neural network to see how much loss every node contributes, and then update the weights to minimize the loss by giving lower weights to nodes with higher error rates, and vice versa. In other words, backpropagation allows data scientists to iteratively adjust the weights to minimize the difference between the actual output and the desired output.

[0091] The backpropagation algorithm is mathematically established in optimization theory. In supervised learning, training data with known outputs is passed through a neural network, and an error is calculated from the known target outputs using a cost function, which gives the backpropagation error. An error is calculated at the output, and this error is converted into a modification of the network weights that minimizes the error.

[0092] JPEG0007800857000004.jpg58170

[0093] JPEG0007800857000005.jpg22170

[0094] JPEG0007800857000006.jpg99170

[0095] The weight update can be calculated by the following formula:

number

[0096] JPEG0007800857000008.jpg67170

[0097] The AI / ML model is trained over multiple epochs until it reaches a good level of accuracy (e.g., 97% or higher using an F2 or F4 threshold for detection and approximately 2,000 epochs). This accuracy level can, in some embodiments, be determined using an F1 score, an F2 score, an F4 score, or any other suitable technique without departing from the scope of the present invention. Once trained on the training data, the AI / ML model is tested on a set of evaluation data that the AI / ML model has not previously encountered. This helps ensure that the AI / ML model does not "overfit," such that it identifies graphical elements in the training data well but does not generalize well to other images.

[0098] In some embodiments, it may be unknown what accuracy level can be achieved. Thus, if the accuracy of the AI / ML model begins to degrade when analyzing the evaluation data (i.e., the model performs well on the training data but poorly on the evaluation data), the AI / ML model may undergo more epochs of training on the training data (and / or new training data). In some embodiments, the AI / ML model is deployed only when accuracy reaches a certain level or when the accuracy of the trained AI / ML model is superior to the existing deployed AI / ML model.

[0099] In particular embodiments, a collection of trained AI / ML models can be used to accomplish tasks such as using an AI / ML model for each type of graphical element of interest, using an AI / ML model to perform OCR, deploying yet another AI / ML model to recognize proximity relationships between graphical elements, and using yet another AI / ML model to generate RPA workflows based on output from other AI / ML models. This can, for example, collectively enable the AI / ML models to enable semantic automation.

[0100] Some embodiments may use Transformer networks such as SentenceTransformers™, a state-of-the-art Python™ framework for sentence, text, and image embedding. Such Transformer networks learn associations between words and phrases with both high and low scores. This trains AI / ML models to determine which are close to the input and which are not, respectively. Transformer networks can also use field lengths and field types rather than simply using word / phrase pairs.

[0101] 8 is a flowchart illustrating a process 800 for training an AI / ML model to perform semantic matching between a source screen or source data and a target screen using semantic AI for RPA workflows, according to one embodiment of the present invention. The process begins at 810 by providing labeled screens (e.g., where graphical elements and text are identified), words and phrases, a "thesaurus" of semantic associations between words and phrases so that similar words and phrases for a given word or phrase can be identified, etc. The AI / ML model is then trained for multiple epochs at 820, and the results are examined at 830.

[0102] If the AI / ML model does not meet the desired confidence threshold at 840, training data is supplemented and / or the reward function is modified at 850 to help the AI / ML model better achieve its objective, and the process returns to step 820. If the AI / ML model meets the confidence threshold at 840, the AI / ML model is tested against evaluation data at 860 to ensure that the AI / ML model generalizes well and that the AI / ML model does not overfit to the training data. The evaluation data may include source screens, source data, and target screens that the AI / ML model has not previously processed. If the confidence threshold is met for the evaluation data at 870, the AI / ML model is deployed at 880. Otherwise, the process returns to step 850 and the AI / ML model is further trained.

[0103] Some embodiments bring semantic automation to an automation platform for creating fully automated workflows with little or minimal interaction input from developers. Using semantic mapping, UI fields from a data source / source screen are semantically mapped to UI fields on a target screen using one or more AI / ML models, and fully automated workflows can be created from this semantic mapping without developer intervention. In the current prototype, mapping can be achieved for up to 80% of the UI fields, and with developer assistance, the remaining approximately 20% can be mapped. The AI / ML models can be retrained to learn to match UI fields more accurately over time, with the hope that mapping will approach 100% accuracy in the future.

[0104] In some embodiments, the RPA designer application includes a semantic matching feature that enables an RPA developer to perform matching between two screens or between data (e.g., customer data) and a screen. This may be implemented, for example, as a “Semantic AI” button on a ribbon, a drop-down menu, or another appropriate user interface element. Selecting the Semantic AI feature may cause the RPA designer application to display a matching interface, such as matching interface 900 in FIGS. 9A-9G. While this is a generic example, there are many use cases for the Semantic AI provided by embodiments of the present invention, such as mapping invoices to SAP®, automatically entering data from an Excel spreadsheet into a CRM application, or mapping XAML from one RPA workflow to another. Also, while this example consists of text fields, other graphical elements, such as buttons, text areas, etc., may be mapped without departing from the scope of the present invention.

[0105] The matching interface 900 includes a mapping options pane 910 and a mapping pane 920. When a developer selects the map screen option 912, a source selection button 922 and a target selection button 924 appear in the mapping pane 920. The user can return to the previous designer application screen by clicking a back button 930. When the user clicks one of these buttons, the user can select a source and target using on-screen display features similar or identical to those in UiPath Studio™, in some embodiments. See, e.g., U.S. Patent Application No. 17 / 100,146. This causes the mapping pane 920 to display a selected source screen 940 and a selected target screen 950. Note that the source screen 940 and / or target screen 950 may be application windows, portions of a displayed application, etc.

[0106] When a user clicks the map button 932, the designer application invokes one or more AI / ML models that perform OCR and CV on the source screen 940 and target screen 950, performs semantic AI analysis that attempts to match fields in the source screen 940 with fields in the target screen 950, and displays matches with confidence scores that meet or exceed a confidence threshold. See FIG. 9B. An overall confidence score 960 is also displayed. In some embodiments, the mapping may be stored in an object repository for future use on the same or similar screens. See, e.g., U.S. Patent Application Serial No. 16 / 922,289.

[0107] In this example, the designer application, through the AI / ML model, was able to correctly match most of the fields in the source screen 940 and the target screen 950. The currency field in the source screen 940 was left blank, so no match was attempted for this graphical element. However, no matches were found for the company and invoice # fields, which is indicated to the developer by highlighting these elements in a different color and with a confidence score of 0.

[0108] Developers can manually match fields not matched by the AI / ML model, and in some embodiments, user matching information is automatically stored as labeled training data for retraining the AI / ML model. For example, the source and target screens may be saved along with bounding box information (e.g., coordinates) and the coordinates and text of labels associated with matched fields in the source screen 940 and target screen 950. This can be seen, for example, in FIG. 9C , where a user indicated that the invoice # field in the source screen 940 matches the invoice number field in the target screen 950. This increases the confidence score for that element and the overall confidence score 960 accordingly.

[0109] In some embodiments, a developer may be prompted or otherwise allowed to provide synonyms for the target field name. For example, in the example of FIG. 9C , the user is prompted to enter a synonym for the invoice number field in synonym text field 952 because this element was not correctly identified. For example, the developer may add “invoice number,” “account number,” “invoice ID,” “account ID,” etc. In particular embodiments, a developer may enter synonyms even if a given graphical element is correctly identified. This feature also allows a developer to add their own terminology and context. For example, if the target application is a tool that the AI / ML model has not seen before, the developer can add terminology to make the AI / ML model more accurate for that tool. This information can then be used to train the AI / ML model to be more accurate not only for that tool, but also for similar words and phrases found in other target screens globally.

[0110] In certain embodiments, synonyms may be suggested to the developer. These can be accepted or rejected to make the AI / ML model more accurate for that context. This allows the AI / ML model to learn both positive and negative examples. It also allows the AI / ML model to learn different subsets of synonyms or alternative sets of synonyms that are applicable to a given context.

[0111] In some embodiments, source data other than images can be used. For example, when a developer selects map data option 914 and clicks source selection button 926 in Figure 9D, data source options 927 appear. See Figure 9E. The developer can select a desired source data format, such as Excel, JavaScript Object Notation (JSON), XAML for an RPA workflow, or a comma-separated variable (CSV) file.

[0112] The data source 970 and target image 950 are then displayed. When the developer clicks the map button 932, the AI / ML model attempts to match the source information to fields in the target image 950. See Figure 9F. In this case, the source data 970 contains the same information as the source screen 940. In Figure 9G, the developer makes similar modifications as in Figure 9C, and the overall confidence score 960 improves.

[0113] The relationship between labels on the source and target screens can be used to determine what a given text field means, although the text fields may be similar or identical to one another. This can be achieved by assigning one or more anchors to a given text field. For example, because the field City appears directly to the left of the associated text field in the target screen 950 and no other text fields contain this label, the designer application and / or AI / ML model can determine that these fields are linked and assign the City label as the anchor for the target text field. If the label does not uniquely identify the text field, one or more other graphical elements can be assigned as anchors, and their geometric relationship can be used to uniquely identify the given target element. See, for example, U.S. Pat. No. 10,936,351 and U.S. Patent Application No. 17 / 100,146.

[0114] After the source screen or source data and target screen are mapped, the user can click the Create button 934 to automatically generate one or more activities in the RPA workflow that implement the desired mapping. See Figures 9C and 9G. This automatically creates the RPA workflow activities. In some embodiments, the RPA workflow is immediately executed after creation to perform the mapping task desired by the user.

[0115] To automatically create RPA workflows, a designer application can utilize a UI object repository. See, e.g., U.S. Patent Application Publication No. 16 / 922,289. A UI object repository (e.g., UiPath Object Repository™) is a collection of UI object libraries, which are themselves collections of UI descriptors (e.g., for a particular version of an application and one or more screens thereof). Unified target controls for similar graphical elements can be obtained from the UI object repository, which instructs the RPA robot how to interact with a given graphical element.

[0116] Such an example is shown in FIG. 10 , which illustrates an RPA designer application 1000 with automatically generated activities in an RPA workflow 1010 according to one embodiment of the present invention. The semantic matching AI / ML model has been trained to recognize associations between source screens or source data and target screens, as described above. In the example of FIGS. 9A-9G and 10 , the semantic matching AI / ML model can determine that data from fields in the source screen or source data should be copied to matching fields in the target screen. Thus, the RPA designer application 1000 knows to add an activity to the RPA workflow 1010 that retrieves the UI descriptors of the target elements from the UI object repository, clicks the target screen, clicks each target field, and uses these UI descriptors to enter text from the source screen or data source into each matching field in the target screen. The RPA designer application 1000 automatically generates one or more activities in the RPA workflow 1010 that implement this functionality. In some embodiments, a developer may not be allowed to modify these activities. However, in certain embodiments, a developer may be able to modify the activity's configuration, have full permissions to edit the activity, etc. In some embodiments, the RPA designer application automatically generates an RPA robot that implements the RPA workflow and executes the RPA robot so that information from the source screen or source data is automatically copied to the target screen without further instruction from the developer.

[0117] Some embodiments provide a semantic copy-and-paste feature that enables developers without substantial programming experience to perform semantic automation. FIG. 11A illustrates a semantic copy-and-paste interface 1100 according to one embodiment of the present invention. In some embodiments, the semantic copy-and-paste interface 1100 is part of an RPA designer application. However, in certain embodiments, the semantic copy-and-paste interface 1100 is part of a standalone application. The semantic copy-and-paste interface 1100 includes a data extraction button 1110, a data entry button 1120, a copy and paste button 1130, a view extracted data button 1140, and a close button 1150. Using the semantic copy-and-paste interface 1100, a user can extract data, enter data, or perform copy and paste from a source application to a target application.

[0118] Upon clicking the Extract Data button 1110, the designer application prompts the developer to open and indicate the application from which they want to extract data via the Data Extraction interface 1112. See FIG. 11B. When the user clicks the Show Applications button 1114 in the Data Extraction interface 1112, a display function is enabled (e.g., the same as or similar to that provided by UiPath Studio™). The user can then select Invoice 1116 as the data source.

[0119] After indicating the application as the source (i.e., in this example, invoice 1113), the semantic automation logic (i.e., semantic matching AI / ML model) can predict the type of source using a classification algorithm, and the data extraction interface 1112 displays the prediction of the type of source in a drop-down menu 1116. See FIG. 11C. The user can confirm the prediction using a confirm button 1117 or select a different type from the drop-down menu 1116. See FIG. 11D. A summary 1118 of the extracted data is then provided to the data extraction interface 1112. See FIG. 11E. The user can then select a back button 1119 to return to the semantic copy-and-paste interface 1100 to perform data entry.

[0120] Upon selecting the input data button 1120, the designer application prompts the developer to open and indicate the application from which they want to extract data via a data entry interface 1122. See Figure 11F. When the user clicks on the application button 1124 of the data entry interface 1122, it indicates that the on-screen functionality is enabled. The user can then select the web bill processing page 1123 as the target application.

[0121] After indicating the web invoice processing page 1123 as the target, the semantic automation logic can predict the type of target using a classification algorithm, and the data entry interface 1122 displays the prediction of the target type in a drop-down menu 1126. See FIG. 11G. The user can confirm the prediction using a confirm button 1127 or select a different type from the drop-down menu 1126. See FIG. 11H. After user confirmation, the designer application automatically populates the web browser 1123 with the extracted data. See FIG. 11I.

[0122] The user can also "copy and paste" data using copy and paste button 1130. Upon selecting copy and paste button 1130, the designer application prompts the developer to open and indicate the application into which they want to enter data via copy and paste interface 1132. See FIG. 11J. A user clicks indicating application button 1134 in copy and paste interface 1132, indicating that the on-screen functionality is enabled. The user can then select line item entry page 1133 as the target application.

[0123] After indicating the line item entry page 1133 as the target, the semantic automation logic can predict the type of target using a classification algorithm, and the copy and paste interface 1132 displays the prediction of the target type in a drop-down menu 1136. See FIG. 11K. The user can confirm the prediction using a confirm button 1137 or select a different type from the drop-down menu 1136. See FIG. 11L. The user can also select from a list of possibilities, such as a spreadsheet 1135. After user confirmation, the designer application automatically copies the data from the spreadsheet 1135 to the line item entry page 1133. See FIG. 11M.

[0124] In some embodiments, the designer application can prompt the user before entering a given line item into the target application via copy-and-paste interface 1132. Such an example is shown in FIG. 11N, where the user reviews and approves each line before it is entered into line-item entry page 1133. Copy-and-paste interface 1132 shows a preview 1138 of the data that will be entered from the next row in spreadsheet 1135. When the user clicks confirm row button 1139, the designer application enters the line shown in preview 1138 into the corresponding field on line-item entry page 1133.

[0125] 12 is an architecture diagram illustrating an architecture 1200 of an AI / ML model for performing semantic AI, according to one embodiment of the present invention. A CV model 1210 performs computer vision functions to identify graphical elements within a screen, and an OCR model 1220 performs text detection and recognition on the screen. In embodiments where both a source screen and a target screen are used, the CV model 1210 and OCR model 1220 perform the CV and OCR functions on both screens.

[0126] The CV model 1210 and OCR model 1220 then provide the type, position, size, text, etc. of the detected graphical elements and text in the target screen or both the target and source screens to a label matching model 1230, which matches labels from the OCR model 1220 with graphical elements from the CV model 1210. The matching labels and associated graphical elements from the screens are then passed to an input data matching model 1240, which matches input data from the data source or source screen with labels of graphical elements in the target screen. The matches and their respective confidence levels are then provided as output from the input data matching model 1240. In some embodiments, multiple AI / ML models can be used for input data matching that perform matching in different ways (e.g., they have different neural network architectures, employ different strategies, are trained with different training data, etc.).

[0127] In some embodiments, an AI / ML model can learn that fields with the same label can have different contexts. For example, the billing information and shipping information sections of a screen may both have an "address" label, but the AI / ML model can learn that the pattern of nearby elements in one is different from the pattern of nearby elements in the other. These sections of the screen can then be used as anchors in multi-anchor techniques, where the text field is the target and the section with the "address" label and the recognized pattern is the anchor. See, e.g., U.S. Pat. No. 10,936,351 and U.S. Patent Application No. 17 / 100,146.

[0128] 13 is a flowchart illustrating a process 1300 for performing semantic matching between a source screen or source data and a target screen using semantic AI for RPA workflows, according to one embodiment of the present invention. The process begins by receiving a selection of a source screen or source data at 1305 and a selection of a target screen at 1310. One or more AI / ML models trained to perform semantic matching between labels in the source screen and labels in the target screen, between data elements in the source data and labels in the target screen, or both, are then invoked at 1315. In some embodiments, the one or more AI / ML models are trained by providing words and phrases with semantic associations between the words and phrases and providing contextual labels for the screen on which the words and phrases appear, so that similar words and phrases for a given word or phrase can be identified. In some embodiments, the one or more AI / ML models include a CV model, an OCR model, a label matching model, and an input data matching model, where the label matching model matches labels detected by the OCR model with fields detected by the CV model, and the input data model receives matching labels from the label matching model and semantically matches data elements from the data source or fields associated with labels from the source screen with fields associated with semantically matching labels on the target screen.

[0129] At 1320, an indication of graphical elements (e.g., location, coordinates, type, etc.) associated with semantically matching labels in the target screen and their respective confidence scores from one or more AI / ML models is received. The graphical elements associated with the semantically matching labels, individual confidence scores, and overall confidence score are displayed on the target screen in a matching interface at 1325. For example, the target screen may be shown, and matching elements may be highlighted or otherwise made apparent to the developer. In some embodiments, a connection is drawn between matching fields in the source screen or source data and the target screen. In certain embodiments, elements in the source screen or source data for which no match was found are highlighted or otherwise indicated to the developer.

[0130] Modifications to graphical elements in the target screen identified by the one or more AI / ML models as having labels that match associated semantics, indications of new elements in the target screen that did not semantically match labels in the source screen by the one or more AI / ML models, or both, are received at 1330. At 1335, information about the modified and / or newly labeled graphical elements in the target screen and associated labels is collected and stored directly (i.e., stored directly in the computing system's memory) or indirectly (i.e., transmitted to an external system for storage). Steps 1330 and 1335 are performed when such modifications are provided by a developer.

[0131] One or more activities in the RPA workflow are automatically generated at 1340 to copy data from fields in the source screen having labels that the one or more AI / ML models identified as semantically matching fields in the target screen, copy data elements from the source data to fields in the target screen having labels that the one or more AI / ML models identified as semantically matching data elements from the source data, or both. An RPA robot that implements the one or more activities generated in the RPA workflow is generated and deployed at 1345.

[0132] At runtime, the deployed RPA robot accesses the UI descriptors of the graphical elements it is trying to identify to execute the automation according to the RPA workflow from the UI object repository and attempts to identify the graphical elements in the target screen using these UI descriptors. If all target graphical elements can be identified at 1355, the information is copied from the source screen or data source to the target screen at 1360. However, if all graphical elements are not found at 1355, the RPA robot invokes an AI / ML model to attempt to identify missing graphical elements and update the UI descriptors for each of these graphical elements at 1365. For example, the RPA robot can use the descriptor information provided by the AI / ML model to update the UI descriptors for each of the missing elements in the UI object repository so that other RPA robots do not encounter the same problem in the future. In this sense, the system is self-healing.

[0133] 14 is a flowchart illustrating a process 1400 for performing semantic matching between a source screen or source data and a target screen using semantic AI for use with an attended automation interface, according to one embodiment of the present invention. The process begins by providing a semantic copy-and-paste interface at 1405. Data is extracted from a source application or data source at 1410. The type of source is predicted using a classification algorithm at 1415. In some embodiments, the semantic copy-and-paste application waits at 1420 to receive confirmation of the prediction or changes to the prediction by the user.

[0134] An indication of the target application from which the user wants to extract data is received at 1425. The type of target is predicted using a classification algorithm at 1430. In some embodiments, the semantic copy and paste application waits at 1435 to receive confirmation of the prediction or changes to the prediction by the user.

[0135] In some embodiments, the user is prompted before each data entry at 1440. For example, before entering a given data item (e.g., a data line, an individual graphical element, etc.), the user can see the data to be entered as it appears in the target application. The user can then preview and approve the entry or reject it. Thereafter, the data from the source is entered into the target application at 1445.

[0136] 13 and 14 may be performed by a computer program encoding instructions for a processor to perform at least a portion of the processes described in FIGS. 13 and 14 in accordance with an embodiment of the present invention. The computer program may be embodied on a non-transitory computer-readable medium. The computer-readable medium may be, but is not limited to, a hard disk drive, a flash device, RAM, tape, and / or any other such medium or combination of media used to store data. The computer program may include encoded instructions, which may also be stored on the computer-readable medium, for controlling a processor of a computing system (e.g., processor 510 of computing system 500 of FIG. 5) to perform all or a portion of the process steps described in FIGS. 13 and 14.

[0137] The computer program may be implemented in hardware, software, or a hybrid implementation. The computer program may be composed of modules designed to operatively communicate with each other and pass information or instructions for display. The computer program may be configured to run on a general-purpose computer, an ASIC, or any other suitable device.

[0138] It will be readily understood that the components of the various embodiments of the present invention, as generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the detailed description of the embodiments of the present invention, as represented in the accompanying figures, is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention.

[0139] The features, structures, or characteristics of the invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, references throughout this specification to "certain embodiments," "some embodiments," or similar language mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Thus, the appearances of "certain embodiments," "some embodiments," "other embodiments," or similar language throughout this specification do not necessarily all refer to the same group of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0140] It should be noted that references to features, advantages, or similar language throughout this specification do not imply that all of the features and advantages that may be realized in the present invention should be in any single embodiment of the present invention, or that any single embodiment of the present invention is. Rather, language referring to features and advantages is understood to mean that the particular feature, advantage, or characteristic described in connection with one embodiment is included in at least one embodiment of the present invention. Thus, throughout this specification, descriptions of features and advantages and similar language can, but do not necessarily, refer to the same embodiment.

[0141] Furthermore, the described features, advantages, and characteristics of the invention may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the invention can be practiced without one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments of the invention.

[0142] Those skilled in the art will readily appreciate that the above-described invention may be practiced with steps in a different order and / or hardware elements in a different configuration than that disclosed. Thus, while the present invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative constructions will be apparent while remaining within the spirit and scope of the invention. Accordingly, reference should be made to the appended claims to determine the scope of the invention.

Claims

1. A non-transitory computer-readable medium storing a computer program, the computer program causing at least one processor to: Receive a selection of source screens or source data; Receive the selection of the target screen, invoking one or more artificial intelligence / machine learning (AI / ML) models trained to perform semantic matching between labels in the source screen and labels in the target screen, between data elements in the source data and the labels in the target screen, or both; receiving a display of graphical elements associated with semantically matching labels within the target screen and respective confidence scores from the one or more AI / ML models; a non-transitory computer-readable medium configured to cause the graphical elements associated with the semantically matching labels to be displayed on the target screen within a matching interface;

2. The computer program causes the at least one processor to:

10. The non-transitory computer-readable medium of claim 1, further configured to cause display of the respective confidence scores for potential matching graphical elements identified by the one or more AI / ML models in the matching interface.

3. The computer program causes the at least one processor to: receiving modifications to graphical elements in the target screen identified by the one or more AI / ML models as having labels that match associated semantics, or receiving representations of new elements in the target screen that did not semantically match labels in the source screen by the one or more AI / ML models, or both; collecting information regarding modified and / or newly labeled graphical elements and associated labels within the target screen; 10. The non-transitory computer-readable medium of claim 1, further configured to directly or indirectly store the collected information for retraining the one or more AI / ML models.

4. The computer program causes the at least one processor to:

10. The non-transitory computer-readable medium of claim 1, further configured to generate one or more activities in a robotic process automation (RPA) workflow that copies data from the fields in the source screen having labels that the one or more AI / ML models identified as semantically matching fields in the target screen, copies the data elements from the source data to the fields in the target screen having labels that the one or more AI / ML models identified as semantically matching the data elements from the source data, or both.

5. The computer program causes the at least one processor to: generating an RPA robot that performs the one or more generated activities in the RPA workflow; 5. The non-transitory computer-readable medium of claim 4, further configured to deploy the generated RPA robot in a runtime environment.

6. The computer program causes the at least one processor to: generating a composite confidence score from the confidence scores of the graphical elements associated with semantically matching labels in the target screen; The non-transitory computer-readable medium of claim 1 , further configured to cause the composite confidence score to be displayed in the matching interface.

7. 10. The non-transitory computer-readable medium of claim 1, wherein the one or more AI / ML models are trained by providing words and phrases having semantic associations between a given word or phrase and providing contextual labels for a screen on which the words and phrases are displayed, such that similar words and phrases to the word or phrase may be identified.

8. 10. The non-transitory computer-readable medium of claim 1, wherein the computer program is or includes a robotic process automation (RPA) designer application.

9. the one or more AI / ML models include a computer vision (CV) model, an optical character recognition (OCR) model, a label matching model, and an input data matching model; the label matching model matches the labels detected by the OCR model with the fields detected by the CV model; the input data model receives the matching label from the label matching model and semantically matches data from the data element from the data source or the field associated with the label from the source screen to the field associated with the semantically matching label on the target screen. The non-transitory computer-readable medium of claim 1 .

10. 1. A computer-implemented method for performing semantic artificial intelligence (AI) for robotic process automation (RPA), comprising: Invoking, by an RPA designer application, one or more artificial intelligence / machine learning (AI / ML) models trained to perform semantic matching between labels in a source screen and labels in a target screen, between data elements in the source data and the labels in the target screen, or both; receiving, by the RPA designer application, a representation of graphical elements associated with semantically matching labels in the target screen, as well as respective confidence scores from the one or more AI / ML models; displaying, by the RPA designer application, the graphical elements associated with the semantically matching labels on the target screen in a matching interface; automatically generating one or more activities in an RPA workflow that copies, by the RPA designer application, data from fields of the source screen or data elements of the source data into fields of the target screen having labels that the one or more AI / ML models have identified as semantically matching the labels from the source screen or the data elements from the source data.

11. displaying, by the RPA designer application, the respective confidence scores for potential matching graphical elements identified by the one or more AI / ML models in the matching interface.

11. The computer-implemented method of claim 10.

12. receiving, by the RPA designer application, modifications to graphical elements in the target screen identified by the one or more AI / ML models as having labels that match associated semantics, or receiving indications of new elements in the target screen that did not semantically match labels in the source screen by the one or more AI / ML models, or both; collecting, by the RPA designer application, information regarding modified and / or newly labeled graphical elements and associated labels in the target screen; storing, directly or indirectly, the collected information by the RPA designer application for retraining the one or more AI / ML models; 11. The computer-implemented method of claim 10, further comprising:

13. displaying, by the RPA designer application, the respective confidence scores for the graphical elements associated with semantically matching labels in the target screen; generating, by the RPA designer application, a composite confidence score from the confidence scores of the graphical elements associated with semantically matching labels in the target screen; displaying, by the RPA designer application, the composite confidence score in the matching interface; 11. The computer-implemented method of claim 10, further comprising:

14. 11. The computer-implemented method of claim 10, wherein the one or more AI / ML models are trained by providing words and phrases having semantic associations between a given word or phrase and providing contextual labels for a screen on which the words and phrases are displayed, so that similar words and phrases to the word or phrase can be identified.

15. the one or more AI / ML models include a computer vision (CV) model, an optical character recognition (OCR) model, a label matching model, and an input data matching model; the label matching model matches the labels detected by the OCR model with the graphical elements detected by the CV model; the input data model receives the matching label from the label matching model and semantically matches data from the data elements from a data source or graphical elements associated with the label from the source screen with the graphical elements associated with the semantically matching label on the target screen.

11. The computer-implemented method of claim 10.

16. 1. A computing system comprising: a memory storing computer program instructions for performing semantic artificial intelligence (AI) for robotic process automation (RPA); and at least one processor configured to execute the computer program instructions, the computer program instructions causing the at least one processor to: receiving, from one or more artificial intelligence / machine learning (AI / ML) models trained to perform semantic matching between labels in a source screen and labels in a target screen, between data elements in source data and the labels in the target screen, or both, indications of graphical elements associated with semantically matching labels in the target screen, and respective confidence scores; displaying the graphical elements associated with the semantically matching labels on the target screen within a matching interface; receiving modifications to graphical elements in the target screen identified by the one or more AI / ML models as having labels that match associated semantics, or receiving representations of new elements in the target screen that did not semantically match labels in the source screen by the one or more AI / ML models, or both; collecting information regarding modified and / or newly labeled graphical elements and associated labels within the target screen; a computing system configured to directly or indirectly store the collected information for retraining the one or more AI / ML models.

17. The computer program instructions may cause the at least one processor to:

17. The computing system of claim 16, further configured to automatically generate one or more activities in a robotic process automation (RPA) workflow that copies data from fields in the source screen having labels that the one or more AI / ML models have identified as semantically matching fields in the target screen, copies data elements from the source data to fields in the target screen having labels that the one or more AI / ML models have identified as semantically matching data elements from the source data, or both.

18. The computer program instructions may cause the at least one processor to: displaying on the target screen the respective confidence scores for the graphical elements associated with the semantically matching labels; generating a composite confidence score from the confidence scores of the graphical elements associated with semantically matching labels in the target screen; The computing system of claim 16 , further configured to cause the composite confidence score to be displayed in the matching interface.

19. 17. The computing system of claim 16, wherein the one or more AI / ML models are trained by providing words and phrases having semantic associations between a given word or phrase and providing contextual labels for a screen on which the words and phrases are displayed, such that similar words and phrases to the word or phrase may be identified.

20. the one or more AI / ML models include a computer vision (CV) model, an optical character recognition (OCR) model, a label matching model, and an input data matching model; the label matching model matches the labels detected by the OCR model with the fields detected by the CV model; the input data model receives the matching label from the label matching model and semantically matches data from the data element from the data source or the field associated with the label from the source screen to the field associated with the semantically matching label on the target screen.

17. The computing system of claim 16.

Images