Performing unified target techniques, default graphic element detection techniques, or both using a combination of serial and delayed parallel execution

Abstract

Disclosed is a graphical element detection that uses a combined serial and delayed parallel execution unified targeting technique that potentially uses multiple graphical element detection techniques, performs a default user interface (UI) element detection technique configuration at an application and / or UI type level, or both. Unified targeting merges multiple techniques that identify and automate UI elements into a single cohesive method. A unified targeting descriptor concatenates multiple types of UI descriptors together and uses them in parallel, or uses at least one technique for a period of time and then, in parallel or alternatively, runs at least one other technique if the first technique does not find a match within the period of time.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. nonprovisional patent application No. 17 / 014,171, filed September 8, 2020. The subject matter of that earlier application is incorporated herein by reference in its entirety. Technical Field

[0003] This invention relates generally to graphic element detection, and more specifically to graphic element detection using a combination of serial and deferred parallel execution of a unified target technique, performing a default UI element detection technique configuration at the application and / or UI type level, or both. Background Technology

[0004] For Robotic Process Automation (RPA) in UI, selectors, computer vision (CV), or optical character recognition (OCR) can be used to perform graphical element detection for each UI action. However, these techniques are often applied individually and are not optimal in all cases. Therefore, improved approaches may be beneficial. Summary of the Invention

[0005] Certain embodiments of the present invention can provide solutions to problems and needs in the art that have not yet been fully identified, understood, or resolved by current graphical element detection technologies. For example, some embodiments of the present invention relate to graphical element detection using a combination of serial and deferred parallel execution of a unified target technology. Some embodiments relate to a default UI element detection technology configuration at the application and / or UI type level. This configuration can then be used to detect UI elements at runtime.

[0006] In one embodiment, a computer-implemented method for detecting graphical elements in a UI includes: receiving, by a designer application, a selection of an activity in an RPA workflow, the selected activity being configured to perform graphical element detection using a unified targeting technique. The computer-implemented method also includes receiving, by the designer application, a modification to the unified targeting technique for the activity, and configuring the activity based on the modification. The unified targeting technique is configured to perform serial and deferred parallel execution of a combination of multiple graphical element detection techniques.

[0007] In another embodiment, a computer program is embodied on a non-transitory computer-readable medium. The computer program is configured to cause at least one processor to analyze a runtime UI to identify UI element attributes and compare the UI element attributes with the UI descriptor attributes of an activity in the RPA workflow using one or more initial graphical element detection techniques. When no match is found using one or more initial graphical element detection techniques during a first time period, the computer program is configured to cause at least one processor to execute one or more additional graphical element detection techniques in parallel with the one or more initial graphical element detection techniques.

[0008] In another embodiment, a computer program is embodied on a non-transitory computer-readable medium. The computer program is configured to cause at least one processor to analyze a runtime UI to identify UI element attributes, and to compare the UI element attributes with UI descriptor attributes using one or more initial graphic element detection techniques for activities within an RPA workflow. When no match is found using one or more initial graphic element detection techniques during a first time period, the computer program is configured to cause at least one processor to execute one or more additional graphic element detection techniques instead of one or more initial graphic element detection techniques.

[0009] In another embodiment, a computer-implemented method for detecting graphical elements in a UI includes receiving a selection of an application or UI type by an RPA designer application. The computer-implemented method further includes receiving and saving a default positioning method setting configuration by the RPA designer application. The computer-implemented method also includes receiving an instruction from the RPA designer application for a screen to be automated. This screen is related to the selected application or UI type. Additionally, the computer-implemented method includes automatically pre-configuring default positioning method settings for the selected application or UI type by the RPA designer application.

[0010] In another embodiment, a computer program is embodied on a non-transitory computer-readable medium. The computer program is configured to cause at least one processor to automatically pre-configure default location method settings for a selected application or UI type. The computer program is also configured to cause at least one processor to receive modifications to the default location method settings for the application or UI type and configure the default location method settings accordingly. The computer program is further configured to cause at least one processor to configure one or more activities in an RPA workflow using the default location method settings for the application or UI type.

[0011] In another embodiment, a computer program is embodied on a non-transitory computer-readable medium. The computer program is configured to cause at least one processor to be automatically pre-configured with default location method settings for an application or UI type by a designer application, and to configure one or more activities in an RPA workflow using those default location method settings. The computer program is also configured to cause at least one processor to generate an RPA robot to implement an RPA workflow including one or more configured activities. Attached Figure Description

[0012] To facilitate understanding of the advantages of certain embodiments of the invention, a more specific description of the invention briefly described above will be presented with reference to specific embodiments illustrated in the accompanying drawings. While it should be understood that these drawings depict only exemplary embodiments of the invention and are therefore not intended to be limiting of the scope of the invention, the invention will be described and explained with additional specificity and detail through the use of the drawings, in which:

[0013] Figure 1 This is an architecture diagram illustrating a robotic process automation (RPA) system according to an embodiment of the present invention.

[0014] Figure 2 This is an architecture diagram illustrating the deployment of an RPA system according to an embodiment of the present invention.

[0015] Figure 3 This is an architecture diagram illustrating the relationship between the designer, activities, and drivers according to an embodiment of the present invention.

[0016] Figure 4 This is an architecture diagram illustrating an RPA system according to an embodiment of the present invention.

[0017] Figure 5 This is an architecture diagram illustrating a computing system configured to perform graphical element detection using a combination of serial and delayed parallel execution of a unified targeting technique and / or one or more default positioning methods configured by an application or UI type, according to an embodiment of the present invention.

[0018] Figure 6A -G illustrates a unified target configuration interface for RPA designer applications according to an embodiment of the present invention.

[0019] Figure 7A -C illustrates a location method configuration interface according to an embodiment of the present invention for configuring a location method at the application and / or UI type level.

[0020] Figure 8 This is a flowchart illustrating a process for configuring unified target functionality for activities in an RPA workflow according to an embodiment of the present invention.

[0021] Figure 9A and 9B This is a flowchart illustrating a process for detecting graphical elements using a combined serial and delayed parallel execution of a unified target technique according to an embodiment of the present invention.

[0022] Figure 10A and 10B These are flowcharts illustrating the design-time and runtime portions of the process for configuring a default graphic element detection technique and performing graphic element detection at the application and / or UI type level, respectively, according to embodiments of the present invention.

[0023] Unless otherwise stated, similar reference numerals throughout the figures always denote corresponding features. Detailed Implementation

[0024] Some embodiments involve graphical element detection using a combination of serial and deferred parallel execution of a unified target technique, which potentially employs multiple graphical element detection techniques (e.g., selectors, CV, OCR, etc.). The terms "graphical element" and "UI element" are used interchangeably herein. At its core, a UI descriptor identifies UI elements (e.g., text fields, buttons, labels, menus, checkboxes, etc.). Certain types of UI descriptors include, but are not limited to, selectors, CV descriptors, image matching descriptors, OCR descriptors, and unified target descriptors that can use multiple different types of UI descriptors serially or in parallel. UI descriptors can be used to compare the properties of a given UI descriptor with the properties of UI elements found in the runtime UI.

[0025] In some embodiments, the UI descriptor stores attributes of the corresponding UI element and its parent element, for example, in an Extensible Markup Language (XML) fragment. Attributes of UI elements found in the runtime UI can be searched for matches with attributes of the corresponding RPA workflow activity, and if an exact match or a "sufficiently close" match within a matching threshold is found, the UI element can be identified and interacted with accordingly. Attributes can include text-based identifiers (IDs), classes, roles, etc. For CV, attributes can include the type of the target element and its relationship to one or more anchor elements that can be used in a multi-anchor matching method. For OCR, attributes can include, for example, text in the form of a stored string, and text found via OCR that fuzzily matches the stored string during execution. Any suitable attribute and graphical element detection techniques can be used without departing from the scope of the invention.

[0026] As used herein, a “screen” is an image of an application’s UI or a portion thereof at a specific point in time. In some embodiments, UI elements and screens can also be distinguished as specific types of UI elements (e.g., buttons, checkboxes, text fields, etc.) and screens (e.g., top window, modal window, pop-up window, etc.).

[0027] Some implementations use UI descriptors, which store the attributes of UI elements and their parent elements in XML fragments. In modern computing systems, operating systems typically represent each user interface as a hierarchical data structure, often called a UI tree. An example UI tree may include a Document Object Model (DOM), which web browser applications use to render web pages.

[0028] A selector is a type of UI descriptor that, in some embodiments, can be used to detect UI elements. In some embodiments, a selector has the following structure:

[0029] <node_1 / ><node_2 / > ...<node_N / >

[0030] The last node represents the GUI element of interest, and all previous nodes represent the parent elements of that element.<node_1> This is usually called the root node and represents the top window of the application.

[0031] Each node may have one or more attributes that help correctly identify a specific level of the selected application. In some embodiments, each node has the following format:

[0032] <ui_system attr_name_1='attr_value_1'...attr_name_N='attr_value_N' / >

[0033] Each property can be assigned a value, and properties with constant values ​​can be selected. This is because changes to property values ​​each time the application is launched may cause selectors to fail to correctly identify associated elements.

[0034] A UI descriptor is a set of instructions used to locate UI elements. In some embodiments, a UI descriptor is an encapsulated data / structure format that includes UI element selectors, anchor selectors, CV descriptors, OCR descriptors, unified target descriptors combining two or more types of UI descriptors, screen image capture (context), element image capture, other metadata (e.g., application and application version), and combinations thereof. The encapsulated data / structure format can be extended with future platform updates and is not limited to the above definitions. Any suitable UI descriptor can be used to identify UI elements on a screen without departing from the scope of the invention. UI descriptors can be extracted from activities in an RPA workflow and added to a structured architecture that groups UI descriptors by UI application, screen, and UI element.

[0035] In some embodiments, the UI descriptor can work with a unified target that includes multiple or all UI element detection mechanisms, through which image detection and definition can be performed. The unified target can combine multiple techniques for identifying and automating UI elements into a single cohesive approach. The unified target descriptor concatenates multiple types of UI descriptors together, using them in parallel, or first using at least one technique (e.g., a selector) for a period of time, and then, if the first technique does not find a match during that period, running at least one other technique in parallel or alternatively. In some embodiments, the unified target descriptor can function like a finite state machine (FSM), where a first UI descriptor mechanism is applied in a first context, a second UI descriptor in a second context, and so on. In some embodiments, if the first two mechanisms are unsuccessful, selector-based and driver-based UI detection mechanisms can be prioritized, relying on CV, image matching, and / or other mechanisms to find graphical elements.

[0036] In some embodiments, fuzzy matching may be employed, where string metrics (e.g., Levenshtein distance, Hamming distance, Jaro-Winkler distance, etc.), combinations thereof, etc., are used, and one or more attributes are to be matched within a certain range with a certain precision (e.g., 70% match, 80% match, 99% match, etc.). Those skilled in the art will understand that similarity metrics can quantify the amount of similarity and the amount of mismatch between two attribute values. Furthermore, in various embodiments, a similarity threshold may represent the maximum or minimum amount of mismatch required for a match.

[0037] Depending on the chosen method of calculating the similarity metric, the similarity threshold can have various interpretations. For example, the similarity threshold can indicate the maximum number of characters that can differ between two strings, or it can indicate a mismatch score calculated as a percentage of the total number of characters (e.g., the combined string length). In some embodiments, the similarity threshold can be rescaled to predetermined intervals, such as between 0 and 1, 0 and 100, 7 and 34, etc. In a non-limiting example, a relatively high similarity threshold (e.g., close to (e.g., 1 or 100%)) represents a requirement for an almost perfect match, meaning that the fuzzy attribute value in the runtime objective is only allowed to differ very little from the value of the corresponding attribute in the design-time objective. Conversely, when the similarity threshold is relatively low (e.g., close to 0), almost any value of the corresponding fuzzy attribute is considered a match.

[0038] In some embodiments, the matching tolerance can vary based on each attribute. For example, an exact match may be required for one or more attributes (e.g., finding a specific name might be desired), while fuzzy matching can be performed for one or more other attributes. In some embodiments, the number and / or type of attributes used by each graphical element detection technique can be customized and specified by the RPA developer.

[0039] In some embodiments, attributes may be stored as attribute-value pairs and / or attribute-value-tolerance pairs (e.g., fuzzy matching). In some embodiments, attribute-value pairs may indicate the name and type of the UI element represented by the corresponding node. However, those skilled in the art will understand that, without departing from the scope of the invention, there are various ways to represent the position of a particular node within the UI tree besides a list of attribute-value pairs.

[0040] In some embodiments, these attribute-value pairs and / or attribute-value tolerance pairs may be stored in tags, and each tag may include a sequence of characters that begins and ends with an implementation-specific delimiter (e.g., starting with "<" and ending with " / >"). In some embodiments, attribute value pairs may indicate the name and type of the UI element represented by the corresponding node. However, those skilled in the art will understand that, without departing from the scope of the invention, there are various ways to represent the position of a particular node within the UI tree besides a list of attribute value pairs.

[0041] To enable RPA robots to perform identification successfully and ideally without ambiguity, some embodiments use element IDs to represent each UI element. In some embodiments, the element ID indicates the location of a target node within the UI tree, where the target node represents the corresponding UI element. For example, the element ID can identify a target node / UI element as a member of a selected subset of nodes. The selected subset of nodes can form a family tree, i.e., a lineage through the UI tree, where each node is an ancestor or descendant of another node.

[0042] In some embodiments, the element ID comprises an ordered sequence of node indicators that traces a genealogical path through the UI tree and the path ending in the corresponding target node / UI element. Each node indicator may represent a member of the object hierarchy of the corresponding UI and its corresponding position in the sequence within the hierarchy. For example, each member of the sequence may represent a descendant (e.g., a child node) of a previous member, and a subsequent member may be a descendant (e.g., a child node). In one example of Hypertext Markup Language (HTML), the element ID representing an individual form field may indicate that the corresponding form field is a child of an HTML form, which in turn is a child of a specific section of a webpage, and so on. In some embodiments, the genealogy does not need to be complete.

[0043] Some implementations may use one or more multi-anchor matching attributes. Anchors are additional UI elements that can be used to help uniquely identify a target UI element. For example, if the UI includes multiple text fields, searching only the text fields is insufficient to uniquely identify a given text field. Therefore, some implementations look for additional information to uniquely identify a given UI element. As an example using text fields, a text field for entering a name might appear to the right of a label "Name". This name label can be set as an "anchor" to help uniquely identify the text field as the "target".

[0044] In some embodiments, a target can be uniquely identified using various locations and / or geometric associations between the target and anchor points, possibly within one or more tolerances. For example, the center of the bounding box for the anchor point and the target can be used to define a line segment. This line segment may then be required to have a certain length within tolerances and / or a slope within tolerances to uniquely identify the target using the target / anchor point pair. However, without departing from the scope of the invention, in some embodiments, any desired location associated with the target and / or anchor point can be used. For example, the point used to draw the line segment can be at the center of the bounding box, the top left corner, the top right corner, the bottom left corner, the bottom right corner, any other location on the border, any location within the bounding box, a location outside the bounding box as identified by bounding box properties, etc. In some embodiments, the target and one or more anchor points may have different locations within or outside their bounding boxes for geometric matching.

[0045] Based on the above, a single anchor point may not always be sufficient to uniquely identify a target element on the screen with a certain degree of confidence. For example, consider a web form where two text fields for entering a name appear to the right of a corresponding label "Name" in different locations on the screen. In this example, one or more additional anchor points can be used to uniquely identify a given target. Geometric properties between the anchor point and the target (e.g., line segment lengths, angles, and / or relative positions with tolerances) can be used to uniquely identify the target. The user may be prompted to continue adding anchor points until the target's match strength exceeds a threshold.

[0046] As used herein, the terms "user" and "developer" are used interchangeably. Users / developers may or may not have programming and / or technical knowledge. For example, in some embodiments, users / developers can create RPA workflows by configuring activities within the RPA workflow without manual coding. In some embodiments, this can be done, for example, by clicking and dragging and dropping various features.

[0047] In some embodiments, a default UI element detection technique (also referred to herein as a "location method") can be configured at the application and / or UI type level. A UI element detection technique that works well for a given application and / or UI type may not work well for another application and / or UI type. For example, for Techniques that are highly efficient for windows may not work well for web browser windows. Therefore, users can configure RPA bots to use the most efficient techniques for a given application and / or UI type.

[0048] Some implementations can be deployed for robotic process automation (RPA). Figure 1This is an architecture diagram illustrating an RPA system 100 according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that allows developers to design and implement workflows. The designer 110 can provide solutions for application integration and automation of third-party applications, management of information technology (IT) tasks, and business IT processes. The designer 110 can support the development of automation projects, which are graphical representations of business processes. In short, the designer 110 supports the development and deployment of workflows and robots.

[0049] Automation projects automate rule-based processes by giving developers control over the execution order and relationships between a set of custom steps (defined herein as "activities") developed within a workflow. A commercial example of an embodiment of Designer 110 is UiPath Studio. TM Each activity can include actions such as clicking a button, reading a file, or writing to a log panel. In some embodiments, workflows can be nested or embedded.

[0050] Some types of workflows may include, but are not limited to, sequences, flowcharts, flow charts (FSMs), and / or global exception handlers. Sequences may be particularly well-suited for linear processes, allowing flow from one activity to another without disrupting the workflow. Flowcharts may be particularly well-suited for more complex business logic, enabling the integration of decisions made in more distinct ways through multiple branching logic operators, as well as the connection of activities. FSMs may be particularly well-suited for larger workflows. FSMs can utilize a limited number of states during their execution, triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly well-suited for determining workflow behavior when execution errors are encountered and are especially suitable for debugging processes.

[0051] Once the workflow is developed in designer 110, the execution of the business process is orchestrated by conductor 120, which orchestrates one or more robots 130 to execute the workflow developed in designer 110. A commercial example of a conductor 120 is the UiPath Orchestrator. TM Commander 120 supports the creation, monitoring, and deployment management of resources in the environment. Utilizing third-party solutions and applications, Commander 120 can act as an integration point or one of the aggregation points.

[0052] Commander 120 can manage a fleet of robots 130, thereby connecting and executing the robots 130 from a central point. The types of robots 130 that can be managed include, but are not limited to: manned robots 132, unmanned robots 134, development robots (similar to unmanned robots 134, but used for development and testing purposes), and non-production robots (similar to manned robots 132, but used for development and testing purposes). Manned robots 132 can be triggered by user events or scheduled to occur automatically, and can operate on the same computing system as humans. Manned robots 132 can be used with commander 120 for centralized process deployment and recording media. Manned robots 132 can assist human users in completing various tasks and can be triggered by user events. In some embodiments, processes cannot be started from commander 120 on this type of robot and / or these processes cannot run under a locked screen. In some embodiments, manned robots 132 can only be started from a robot tray or upon command prompt. In some embodiments, manned robots 132 should operate under human supervision.

[0053] Unattended robot 134 operates unattended in a virtual environment or on a physical machine and can automate many processes. Unattended robot 134 can be responsible for remote execution, monitoring, scheduling, and supporting work queues. In some embodiments, debugging for all robot types can be run in designer 110. Both manned and unattended robots can automate various systems and applications, including but not limited to mainframes, web applications, VMs, and enterprise applications (e.g., those powered by...). Enterprise applications (such as those used in production) and computing system applications (such as desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0054] Commander 120 may have various capabilities, including but not limited to provisioning, deployment, versioning, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between robot 130 and commander 120 (e.g., a web application). Development may include ensuring that package versions are correctly delivered to the assigned robot 130 for execution. In some embodiments, versioning may include managing unique instances of a process or configuration. Configuration may include maintaining and delivering robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs to a database (e.g., an SQL database) and / or another storage mechanism (e.g., a web application). It provides the ability to store and quickly query large datasets. The Commander 120 can provide interconnectivity by acting as a central point for communication between third-party solutions and / or applications.

[0055] Robot 130 is an execution agent that runs workflows built into Designer 110. A commercial example of some embodiments of (multiple) robots 130 is UiPath Robots. TM In some embodiments, robot 130 has Microsoft installed by default. Services managed by the Service Control Manager (SCM). As a result, this type of robot 130 can open interactive windows under the local system account. Conversation, and has Service permissions.

[0056] In some embodiments, robot 130 can be installed in user mode. For such a robot 130, this means it has the same permissions as a user who has already installed the given robot 130. This feature can also be used with high-density (HD) robots, which ensure that each machine is fully utilized to its maximum potential. In some embodiments, any type of robot 130 can be configured in an HD environment.

[0057] In some embodiments, robot 130 is divided into several components, each dedicated to a specific automation task. In some embodiments, robot components include, but are not limited to, SCM-managed robot services, user-mode robot services, actuators, agents, and command lines. SCM-managed robot service management and monitoring. The session acts as a delegate between the controller 120 and the execution host (i.e., the computing system on which the robot 130 executes). These services are trusted by the robot 130 and manage the robot 130's credentials. The console application is launched by the SCM on the local system.

[0058] In some embodiments, user-mode robot service management and monitoring The session acts as a delegate between the command center 120 and the execution host. The user-mode robot service can be trusted by the robot 130 and manage the robot 130's credentials. Without an SCM-managed robot service installed, The application can be launched automatically.

[0059] The actuator can be A given job runs within a session (i.e., the executor can execute the workflow). The executor can know the dots per inch (DPI) setting for each monitor. An agent can be... Presentation Foundation (WPF) applications display available jobs in a system tray window. Agents can be clients of the service. Agents can request to start or stop jobs and change settings. The command line is a client of the service. The command line is a console application that can request to start a job and wait for its output.

[0060] As explained above, separating the components of robot 130 generally helps developers, support users, and the computing system to more easily run, identify, and track what each component is performing. Special behaviors can be configured per component in this way, such as setting different firewall rules for executors and services. In some embodiments, the executor can always know the DPI setting of each monitor. As a result, the workflow can be executed at any DPI, regardless of the configuration of the computing system that created the workflow. In some embodiments, projects from designer 110 can also be independent of browser zoom levels. In some embodiments, DPI can be disabled for applications that are unaware of or intentionally marked as unaware of the DPI.

[0061] Figure 2 This is an architecture diagram illustrating an RPA system 200 deployed according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be... Figure 1The RPA system 100 may be part of the RPA system. It should be noted that, without departing from the scope of the invention, the client side, server side, or both may include any desired number of computing systems. On the client side, the robot application 210 includes an actuator 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not run on the computing system 210. The actuator 212 is running the process. Figure 2 As shown, several business items can run simultaneously. In this embodiment, agent 214 (e.g., The service is a single point of contact for all executors 212. All messages in this embodiment are logged to the controller 230, which further processes these messages via a database server 240, an indexer server 250, or both. (As described above relative to...) Figure 1 The actuator 212 discussed may be a robot component.

[0062] In some embodiments, a robot represents an association between a machine name and a username. A robot can manage multiple actuators simultaneously. In computing systems that support multiple concurrently running interactive sessions (e.g., On a server (2012), multiple bots can run simultaneously, each using a unique username. During the conversation. This will be referred to as the HD robot above.

[0063] Agent 214 is also responsible for sending the robot's status (e.g., periodically sending "heartbeat" messages indicating that the robot is still operating) and downloading the required version of the package to be executed. In some embodiments, communication between agent 214 and controller 230 is always initiated by agent 214. In notification scenarios, agent 214 may open a WebSocket channel, which controller 230 later uses to send commands to the robot (e.g., start, stop, etc.).

[0064] On the server side, there are representation layers (web application 232, Open Data Protocol (OData) Representation State Transfer (REST) ​​Application Programming Interface (API) endpoint 234, and notification and monitoring 236), service layers (API implementation / business logic 238), and persistence layers (database server 240 and indexer server 250). The controller 230 includes web application 232, ODataREST API endpoint 234, notification and monitoring 236, and API implementation / business logic 238. In some embodiments, most actions performed by the user in the interface of the controller 220 (e.g., via browser 220) are performed by calling various APIs. Such actions may include, but are not limited to, starting a job on a robot, adding / removing data from a queue, scheduling a job to run unattended, etc., without departing from the scope of the invention. Web application 232 is the visual layer of the server platform. In this embodiment, web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the invention. In this embodiment, the user interacts with a webpage from web application 232 via browser 220 to perform various actions to control the controller 230. For example, the user can create robot groups, assign packages to robots, analyze logs of each robot and / or each process, start and stop robots, etc.

[0065] In addition to web application 232, the controller 230 also includes a service layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the invention. The REST API is consumed by both web application 232 and agent 214. In this embodiment, agent 214 is a supervisor of one or more bots on a client computer.

[0066] In this embodiment, the REST API covers configuration, logging, monitoring, and queuing functionality. In some embodiments, the configuration endpoint can be used to define and configure application users, permissions, bots, assets, publications, and environments. The logging REST endpoint can be used to log various information, such as errors, explicit messages sent by the bot, and other environment-specific information. If a start job command is used in the controller 230, the bot can use the deployment REST endpoint to query the package version that should be executed. The queuing REST endpoint can be responsible for queue and queue item management, such as adding data to the queue, retrieving transactions from the queue, and setting the status of transactions.

[0067] The monitoring REST endpoint can monitor web application 232 and agent 214. The notification and monitoring API 236 can be a REST endpoint used to register agent 214, deliver configuration settings to agent 214, and send / receive notifications from the server and agent 214. In some embodiments, the notification and monitoring API 236 can also use WebSocket communication.

[0068] In this embodiment, the persistence layer includes a pair of servers—a database server 240 (e.g., an SQL server) and an indexer server 250. In this embodiment, the database server 240 stores configurations for robots, robot groups, associated processes, users, roles, schedules, etc. In some embodiments, this information is managed via a web application 232. The database server 240 can manage queues and queue items. In some embodiments, the database server 240 can store messages logged by robots (in addition to or in place of the indexer server 250).

[0069] In some embodiments, an optional indexer server 250 stores and indexes information recorded by the robot logs. In some embodiments, the indexer server 250 can be disabled through configuration settings. In some embodiments, the indexer server 250 uses... It is an open-source full-text search engine project. Messages logged by the robot (such as activities using log messages or writing lines) can be sent to indexer server 250 via (multiple) logging REST endpoints, where these messages are indexed for future use.

[0070] Figure 3This is an architecture diagram illustrating the relationship 300 between designer 310, activities 320, 330, and driver 340 according to an embodiment of the present invention. As described above, developers use designer 310 to develop workflows executed by a robot. Workflows may include user-defined activities 320 and UI automation activities 330. Some embodiments are capable of identifying non-textual visual components in an image, referred to herein as computer vision (CV). Some CV activities associated with such components may include, but are not limited to, clicking, typing, retrieving text, hovering, element presence, refreshing range, highlighting, etc. In some embodiments, clicking identifies an element using, for example, CV, optical character recognition (OCR), fuzzy text matching, and multi-anchor, and the element is clicked. Typing can use the above to identify an element and the type within that element. Retrieving text can identify the location of specific text and scan that location using OCR. Hovering can identify an element and hover over it. Element presence can use the techniques described above to check if an element exists on the screen. In some embodiments, hundreds or even thousands of activities that can be implemented in designer 310 may exist. However, any number and / or type of activity may be available without departing from the scope of the invention.

[0071] UI automation activities 330 are a subset of specific low-level activities (e.g., CV activities) written in low-level code that facilitate interaction with the application through the UI layer. In some embodiments, UI automation activities 300 may “simulate” user input, for example, through window messages. UI automation activities 330 support these interactions via drivers 340 that allow the robot to interact with the desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc.

[0072] Driver 340 can perform low-level interactions with the OS to locate hooks, monitor keys, etc. The driver can facilitate communication with... Integration of features such as "click" activity, which plays the same role in these different applications via driver 340.

[0073] Figure 4 This is an architectural diagram illustrating an RPA system 400 according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be or include Figure 1 and / or Figure 2The RPA system 100 and / or 200. The RPA system 400 includes multiple client computing systems 410 that operate the robot. The computing systems 410 are capable of communicating with the command computing system 420 via a web application running thereon. The command computing system 420 is also capable of communicating with a database server 430 and an optional indexer server 440.

[0074] Compared to Figure 1 and Figure 3 It should be noted that while web applications are used in these embodiments, any suitable client and / or server software can be used without departing from the scope of the invention. For example, the controller can run a server-side application that communicates with non-web-based client software applications on the client computing system.

[0075] Figure 5 This is an architectural diagram illustrating a computing system 500 according to an embodiment of the present invention, which is configured to perform graphical element detection using a combination of serial and delayed parallel execution of a unified targeting technique and / or one or more default positioning methods configured by an application or UI type. In some embodiments, the computing system 500 may be one or more of the computing systems depicted and / or described herein. The computing system 500 includes a bus 505 or other communication mechanism for transmitting information, and processor(s) 510 coupled to the bus 505 to process information. The processor(s) 510 may be any type of general-purpose or special-purpose processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor(s) 510 may also have multiple processing cores, and at least some cores may be configured to perform specific functions. In some embodiments, multi-parallel processing may be used. In some embodiments, at least one of the processors 510 may be a neuromorphic circuit including processing elements that mimic biological nerves. In some embodiments, the neuromorphic circuit may not require typical components of a von Neumann computing architecture.

[0076] The computing system 500 also includes a memory 515 for storing information and instructions to be executed by the processor(s) 510. The memory 515 may consist of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage devices (such as disks or optical discs), or any other type of non-transitory computer-readable medium or any combination thereof. The non-transitory computer-readable medium may be any available medium accessible by the processor(s) 510 and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both.

[0077] Additionally, the computing system 500 includes a communication device 520 (such as a transceiver) to provide access to a communication network via a wireless and / or wired connection. In some embodiments, without departing from the scope of the invention, the communication device 520 may be configured to use Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiplexing (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), CDMA2000, Wideband CDMA (W-CDMA), High-Speed ​​Downlink Packet Access (HSD). High-speed uplink packet access (HSUPA), high-speed packet access (HSPA), Long Term Evolution (LTE), LTE Advanced (LTE-A), 802.11x, Wi-Fi, Zigbee, Ultra Wideband (UWB), 802.16x, 802.15, Home Node B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near Field Communication (NFC), 5G, New Radio (NR), any combination thereof and / or any other existing or future communication standards and / or protocols. In some embodiments, without departing from the scope of the invention, the communication device 520 may include one or more antennas, which may be singular, arrayed, phased, switched, beamformed, beamguided, combinations thereof and / or any other antenna configuration.

[0078] Multiple processors 510 are further coupled to a display 525 via a bus 505, such as a plasma display, liquid crystal display (LCD), light-emitting diode (LED) display, field emission display (FED), organic light-emitting diode (OLED) display, flexible OLED display, flexible substrate display, projection display, 4K display, high-definition display, etc. The display 525 may be a touch (haptic) display, an in-plane switching (IPS) display, or any other suitable display for displaying information to a user. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, etc., using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse recognition, suppressed total internal reflection, etc., employing resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse recognition, suppressed total internal reflection, etc., without departing from the scope of this invention. Any suitable display device and haptic I / O may be used without departing from the scope of this invention.

[0079] Keyboard 530 and cursor control devices 535 (such as a computer mouse, touchpad, etc.) are further coupled to bus 505 to enable a user to interface with computing system 500. However, in some embodiments, a physical keyboard and mouse may be absent, and the user may interact with the device solely through display 525 and / or touchpad (not shown). Any type and combination of input devices can be considered as a design choice. In some embodiments, there are no physical input devices and / or displays. For example, a user may interact remotely with computing system 500 via another computing system with which it communicates, or computing system 500 may operate autonomously.

[0080] Memory 515 stores software modules that provide functionality when executed by processor(s) 510. These modules include an operating system 540 of the computing system 500. These modules also include robot access control and management modules 545 configured to perform all or part of the processes described herein or derivatives thereof. Modules also include serial and parallel unified target / default localization method modules 545 configured to perform combinations of all or part of the processes described herein or derivatives thereof. The computing system 500 may include one or more additional functional modules 550, which include additional functionality.

[0081] Those skilled in the art will understand that, without departing from the scope of this invention, the "system" can be implemented as a server, embedded computing system, personal computer, console, personal digital assistant (PDA), mobile phone, tablet computing device, quantum computing system, or any other suitable computing device or combination of devices. Presenting the functions described above as being performed by the "system" is not intended to limit the scope of the invention in any way, but rather to provide an example of many embodiments of the invention. In fact, the methods, systems, and apparatuses disclosed herein can be implemented in a localized and distributed manner consistent with computing technologies including cloud computing systems. The computing system can be part of, or accessible from, a local area network (LAN), mobile communication network, satellite communication network, the Internet, public or private cloud, hybrid cloud, server farm, or any combination thereof. Any localized or distributed architecture can be used without departing from the scope of this invention.

[0082] It should be noted that some system features described in this specification have been presented as modules to more specifically emphasize their implementation independence. For example, modules can be implemented as hardware circuits that include custom-designed very large-scale integration (VLSI) circuits or gate arrays, off-the-shelf semiconductors (such as logic chips, transistors, or other discrete components). Modules can also be implemented in programmable hardware devices, such as field-programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, etc.

[0083] Modules can also be implemented, at least partially, in software for execution by various types of processors. For example, the identified units of executable code may include one or more physical or logical blocks of computer instructions, which may be organized, for example, into objects, procedures, or functions. However, the executable file of the identified module does not need to be physically located in one place, but may include entirely different instructions stored in different locations that, when logically combined, encompass the module and achieve its intended purpose. Furthermore, modules can be stored on a computer-readable medium, which, without departing from the scope of the invention, may be, for example, a hard disk drive, a flash memory device, RAM, magnetic tape, and / or any other such non-transitory computer-readable medium for storing data.

[0084] In practice, a module of executable code can be a single instruction or many instructions, and can even be distributed across several different code segments, different programs, and multiple memory devices. Similarly, operational data can be identified and illustrated herein as being within a module, and this operational data can be implemented in any suitable form and organized within any suitable type of data structure. Operational data can be collected as a single dataset, or it can be distributed across different locations including different storage devices, and can exist at least in part solely as electronic signals on a system or network.

[0085] Figure 6A -G illustrates a unified target configuration interface for an RPA designer application 600 according to an embodiment of the present invention. In this embodiment, RPA developers can customize the unified target functionality for activities in an RPA workflow. The RPA designer application 600 includes an RPA workflow development pane 610 with clickable activity 612. The RPA designer application 600 also includes a unified target configuration pane 620. When a user clicks on an activity that interacts with a graphical element in the UI, the unified target configuration pane 620 displays the unified target options for that activity.

[0086] like Figure 6A As shown, the serial execution selector 630 allows RPA developers to choose whether the unified target technology 612 for an activity will be executed serially, one by one, or in parallel with at least one stage of delayed parallel execution. If serial execution is selected, the execution order of the technology can be specified using the drop-down menu 632. See also Figure 6B The selected technology 634 appears after selection, and the timeout for each technology can be specified via the timeout field 636. See also Figure 6C .

[0087] If delayed parallel execution is desired, the value of serial execution selector 630 can be set to "No," and RPA developers can then select whether to use one or more initialization techniques via initialization technique selector 640. See also Figure 6D If the initial technology selector 640 is set to "No", a unified target graphical element detection technique can be executed in parallel. However, if the initial technology selector 640 is set to "Yes", a drop-down menu 642 appears for selecting the initial technology. See also Figure 6E However, in some embodiments, one or more initialization techniques are executed automatically and may not be configurable by the RPA developer. For example, a selector-based approach may be faster than others and may be tried at the start. In some embodiments, one or more default initialization techniques may be shown, and the RPA developer may change these default initialization techniques (e.g., by adding new techniques, removing default techniques, etc.). In some embodiments, the RPA developer cannot configure the initialization techniques, but can configure delayed parallelization techniques.

[0088] After the initial technology 644 is selected, a parallel technology drop-down menu 646 appears, listing the remaining technologies. See also Figure 6F In some embodiments, the type of graphical element detection technology is automatically selected based on the following: the action performed by the activity (e.g., click, text capture, hover, etc.), the type of graphical element (e.g., button, text field, etc.), and / or the specific graphical element indicated by the RPA developer (i.e., which element the user selected on the screen, and what other elements exist in the application). For a specific graphical element, for example, if an RPA developer clicks an "OK" button, but there are two "OK" buttons on the screen, some attributes can be automatically added to distinguish these two otherwise identical "OK" buttons. For example, when using a UI tree, it is typically constructed such that when the RPA developer indicates a graphical element on the screen, at least some attributes of that graphical element in the UI tree are different compared to other graphical elements.

[0089] RPA developers can still add more initial technologies via Add link 645, or they can delete previously selected initial technologies. In some embodiments, if the RPA developer selects "No" for the initial technology selector 640, the parallel technology dropdown menu 646 still appears, and the RPA developer can customize the selection of technologies to be executed in parallel. After parallel technology 647 is selected, the RPA developer can still add more parallel technologies via Add link 648, or they can delete previously selected parallel technologies. See also Figure 6GParallel techniques can be executed by specifying a delay, such as how long to wait after starting one or more initial techniques, via the delay field 649. In some embodiments, a multi-anchor technique is used to identify the target and one or more anchors. Uniform target settings can be customized for the target and each anchor, or the same settings can be applied to the target and one or more anchors.

[0090] In some embodiments, multiple delay periods may be used. For example, an initial technique may be used for up to one second, and if no match is found, one or more other techniques may be used in parallel with the initial technique, and if no match is found in a second period, other techniques may be applied in parallel again, and so on. Without departing from the scope of the invention, any number of delay periods and / or any number of techniques may be used during each delay period.

[0091] In some embodiments, mutually exclusive sequential phases can be employed. For example, an initial technique can be used for up to one second, and if no match is found, one or more other techniques can be used to replace the initial technique, and if no match is found in a second time period, other techniques can be applied to replace the initial and second time period techniques, and so on. In this way, techniques that appear unsuccessful can be stopped, thereby potentially reducing resource requirements.

[0092] Figure 7A A configuration interface 700 for a collapsible positioning method at the application and / or UI type level is illustrated according to an embodiment of the present invention. In this embodiment, the user can configure a collapsible positioning method at the web browser, UI Automation (UIA) desktop, Active Accessibility (AA) desktop, and Win32 desktop configuration location methods. Each of these applications and / or display types can be configured individually by the user.

[0093] Turn Figure 7B Users have already used The "Location Method" tab 710 is for... A default location method has been configured. Because... The current selector is powerful and reliable, so the running value of the full selector 712 will be set to "true". The running values ​​of the blur selector 714, the image selector 716, and the enabled anchor point 718 (i.e., the ability to identify targets using the target / anchor point function) will be set to "false".

[0094] However, this technique may not work for all applications and / or UI types. For example, selectors may not work in many modern web browsers because attribute values ​​often change dynamically. [Go to...] Figure 7C The user has already configured the default location method for the web browser using the web location method tab 720. Input mode 722 is set to "Simulate" via drop-down menu 723. This is a web-specific setting and, without departing from the scope of the invention, may include other application and / or UI type-specific settings for the web application and / or other application / UI types. In some embodiments, when the application is automated, different mechanisms (e.g., providing mouse clicks, key presses, etc.) can be used to interact with the application. "Simulate" simulates the input that the web browser will receive from the system when the user performs similar interactions.

[0095] Because using the full selector is inaccurate for many web browsers, the full selector's running value 724 is set to "false". On the other hand, the running values ​​for the fuzzy selector 725, the image selector 726, and the enabled anchor 727 are set to "true". This is consistent with... Figure 7B In The opposite configuration is in the Location Method tab 710.

[0096] By configuring the positioning method based on each application and / or each UI type, users do not need to reconfigure each graphical element when they point to it on the screen. For example, if a user points to a text field in a web browser, the positioning method used for that text field will be a pre-configured method, without requiring the user to type it in each time and set the run value of the full selector to "false," the run value of the fuzzy selector to "true," and so on. However, it should be noted that in some embodiments, if there are specific UI elements that can be detected more accurately using a different positioning method compared to the default configuration, the user can modify these default values.

[0097] It should be noted that other positioning methods are also possible. For example, in some embodiments, CV, OCR, or a combination thereof may be used. In fact, any suitable graphic element detection technique may be used without departing from the scope of the invention.

[0098] Figure 8 This is a flowchart illustrating a process 800 for configuring unified target functionality for activities in an RPA workflow according to an embodiment of the present invention. In some embodiments, process 800 may be provided by Figure 6A The RPA designer application 600 executes with the -G nameplate. The process begins at 810 by receiving a selection of an activity in the RPA workflow, configured to perform graphical element detection using a unified target. In some embodiments, at 820, unified target functionality is automatically pre-configured. In some embodiments, this pre-configuration may be based on the type of graphical element detection technology, the type of graphical element, and / or a specific graphical element indicated by the RPA developer.

[0099] At step 830, the RPA designer application receives modifications to the unified goal functionality from the RPA developer to customize the unified goal functionality configuration for the activity. Then, at step 840, the RPA designer application configures the activity based on the unified goal configuration. If more activities need to be configured, the RPA developer can select another activity, and the process returns to step 810. Once the desired activities are configured, at step 850, the RPA designer application generates an RPA robot to implement the RPA workflow, including the configured activities. The process then ends or proceeds to... Figure 9A .

[0100] Figure 9A and 9B This is a flowchart illustrating a process 900 for performing a unified target technique in parallel using a combination of serial and delayed execution according to an embodiment of the present invention. In some embodiments, process 900 may be performed via... Figure 6A The RPA designer application (-G) creates an RPA robot that is implemented at runtime. The process begins at step 910 by analyzing the UI (e.g., screenshots, images of application windows, etc.) to identify UI element attributes. UI element attributes can include, but are not limited to, images, text, relationships between graphic elements, and hierarchical representations of graphic elements within the UI. This identification can be performed via CV, OCR, API calls, analysis of text files (e.g., HTML, XML, etc.), and combinations thereof.

[0101] After analyzing the UI, at 920, use Uniform Target to analyze the UI element attributes used for the activity. (Go to...) Figure 9B At 922, one or more initial graphic element detection techniques are executed (potentially in parallel). If a match is found in the first time period of the initial technique at 924 (e.g., within one-tenth of a second, two seconds, ten seconds, etc.), then at 926, the result of that match is selected.

[0102] If no match is found at 924 within the initial technical time period, then at 928, one or more additional graphic element detection techniques are executed in parallel. If a match is found at 929 within the second time period, then at 926, the result of the first technique that found the match is selected from all the initial techniques and subsequent parallel techniques, and the process proceeds to step 930. In some embodiments, finding a match for a graphic element may involve finding a match for the target graphic element itself and its anchor points, and a first technique that finds a match may be selected for that corresponding target / anchor point. If no match is found at 929 within the second time period, the process also proceeds to step 930. In some embodiments, multiple phases of delayed parallel execution are performed. In some embodiments, different techniques are executed at each phase, and the previous technique is stopped.

[0103] If a matching UI element is found via a unified target at 930, then at 940, the action associated with the activity involving the UI element is executed (e.g., clicking a button, entering text, interacting with a menu, etc.). If more activities are found at 950, the process proceeds to step 920 to proceed to the next activity. However, if no UI element matching the attributes of the graphical element detection technology is found at 930, then at 960, an exception is thrown or the user is asked how he or she would like to proceed (e.g., whether to continue), and the process ends.

[0104] Figure 10A and 10B These are flowcharts illustrating the design-time and runtime portions of a process 1000 for configuring a graphical element detection technique and performing graphical element detection at the application and / or UI type level, according to embodiments of the present invention. In some embodiments, the design-time portion of process 1000 can be... Figure 7A The location method configuration interface 700 (-C) is used to execute this process. The process begins at 1005 by receiving a selection of an application or UI type for use as the default location method configuration. The developer then configures the default location method settings (e.g., for a web browser, Win32 desktop, etc.), and at 1010, receives and saves this default configuration. In some embodiments, the default configuration may be saved in a repository of UI objects accessible to multiple users. Then, if needed, the user can select a different application or UI type for the default location method configuration.

[0105] Then, the same or different developers instruct the screen in the UI, and at 1015, receive screen instructions (e.g., by a developer such as UiPath Studio). TM (e.g., RPA designer applications). In other words, different instances of the RPA designer application can be used to configure the default positioning method settings, and subsequently modify the default positioning method settings. In practice, in some embodiments, these instances may not be on the same computing system. Then, at 1020, the default positioning method settings are pre-configured for the detected application or UI type associated with the screen. At 1025, the RPA designer application can receive modifications to the default positioning method settings and configure the settings accordingly. For example, a user can choose which positioning methods he or she wants to use on screens where the default technology is not ideal. For example, a screen may have visual characteristics substantially different from other screens in the application or desktop.

[0106] After the target localization method settings have been configured by default or modified from the default configuration, the user will develop an RPA workflow, where, at 1030, these target localization method settings are used to configure activities. In some embodiments, the user can still modify the default localization method settings while designing the RPA workflow. After the user completes the RPA workflow, at 1035, the RPA designer applies a generated RPA robot to implement the RPA workflow including the configured activities. The process then ends or proceeds to... Figure 10B .

[0107] Turn Figure 10B In section 1040, the UI (e.g., screenshots, images of application windows, etc.) is analyzed to identify UI element attributes. UI element attributes may include, but are not limited to, images, text, relationships between graphic elements, and hierarchical representations of graphic elements in the UI. This identification can be performed via CV, OCR, API calls, analysis of text files (e.g., HTML, XML, etc.), and combinations thereof.

[0108] After analyzing the UI, unless the user overridden the default positioning method configuration for a given UI element during design-time development, at step 1045, the UI element properties are analyzed using the default positioning method configuration for use with the activity. If a matching UI element is found using the positioning method at step 1050, then at step 1055, the action associated with the activity involving the UI element is executed (e.g., clicking a button, entering text, interacting with a menu, etc.). If more activities are found at step 1060, the process proceeds to step 1045 for the next activity. However, if no UI element is found using the configured positioning method at step 1050, then at step 1065, an exception is thrown or the user is asked how they wish to proceed (e.g., whether to continue), and the process ends.

[0109] According to an embodiment of the present invention, Figures 8 to 10B The process steps executed in the process can be performed by a computer program that is encoded with instructions that cause (multiple) processors to execute them. Figures 8 to 10B The computer program may be implemented on a non-transient computer-readable medium, including but not limited to hard disk drives, flash memory devices, RAM, magnetic tape, and / or any other such medium or combination of media for storing data. The computer program may include processors (e.g., multiple processors) for controlling a computing system. Figure 5 The computing system 500 has (multiple) processors 510 to achieve Figures 8 to 10B The coded instructions, in whole or in part, of the process steps described herein, may also be stored on a computer-readable medium.

[0110] Computer programs can be implemented in hardware, software, or a hybrid implementation. A computer program can consist of modules that can operatively communicate with each other and are designed to exchange information or instructions for display. A computer program can be configured to operate on a general-purpose computer, an ASIC, or any other suitable device.

[0111] It will be readily understood that, as generally described and illustrated in the figures herein, the components of various embodiments of the invention can be arranged and designed in a variety of different configurations. Therefore, the detailed description of embodiments of the invention as illustrated in the figures is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention.

[0112] The features, structures, or characteristics of the invention described throughout this specification can be combined in any suitable manner in one or more embodiments. For example, throughout this specification, references to "certain embodiments," "some embodiments," or similar language mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Therefore, throughout this specification, the phrases "in some embodiments," "in some embodiments," "in other embodiments," or similar language do not necessarily all refer to the same set of embodiments, and the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0113] It should be noted that throughout this specification, references to features, advantages, or similar language do not imply that all features and advantages achievable with the invention should be, or all exist, in any single embodiment of the invention. Rather, language relating to features and advantages is understood to mean that a particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Therefore, throughout this specification, discussions of features and advantages, as well as similar language, may, but are not necessarily, of the same embodiments.

[0114] Furthermore, the features, structures, and characteristics described in this invention can be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the invention can be practiced without having one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages that may not be present in all embodiments of the invention may be identified in some embodiments.

[0115] It will be readily understood by those skilled in the art that the invention as described above can be practiced using steps in different sequences and / or hardware elements with configurations different from those disclosed. Therefore, although the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative constructions will be apparent while still remaining within the spirit and scope of the invention. Therefore, reference should be made to the appended claims to determine the limits and boundaries of the invention.

Claims

1. A computer-implemented method for detecting graphical elements in a user interface (UI), comprising: The designer application receives selections of activities in the Robotic Process Automation (RPA) workflow, which will be configured to perform graphical element detection using a unified target technology. The unified target technology can be customized by the RPA developer to specify one or more initial graphical element detection technologies, one or more additional graphical element detection technologies, and a delay time after initiating the one or more initial graphical element detection technologies and before executing the one or more additional graphical element detection technologies. The designer application receives modifications to the unified target technology for the activity; The designer applies the configuration of the activity based on the modifications; The designer application generates an RPA robot configured to perform the configured activities; The RPA robot analyzes the runtime UI to identify UI element attributes; The RPA robot uses one or more initial graphical element detection techniques to compare the UI element attributes with the UI descriptor attributes used for the activity. as well as When a match is not found during the first time period defined by the delay time using the one or more initial graphic element detection techniques: the one or more additional graphic element detection techniques are executed by the RPA robot in parallel with the one or more initial graphic element detection techniques. The unified target technology is configured to perform serial and delayed parallel execution of the unified target technology by employing a combination of multiple graphics element detection technologies.

2. The computer-implemented method according to claim 1 further includes: The designer application automatically configures one or more graphic element detection technologies among the graphic element detection technologies for the unified target technology.

3. The computer-implemented method of claim 2, wherein the automatic configuration of the one or more graphic element detection techniques is performed by the designer based on the action performed by the activity, the type of the target graphic element, the presence of one or more other graphic elements in the UI, or a combination thereof.

4. The computer-implemented method of claim 1, wherein the method is repeated for at least one additional activity in the robotic process automation (RPA).

5. The computer-implemented method of claim 1, wherein when a match is not found during the second time period using the one or more initial graphic element detection techniques and the one or more additional graphic element detection techniques, the method further comprises: One or more supplementary graphic element detection techniques are performed in parallel by the RPA robot and the one or more initial graphic element detection techniques and the one or more additional graphic element detection techniques.

6. The computer-implemented method according to claim 1, further comprising: The RPA robot analyzes the runtime UI to identify UI element attributes; The RPA robot uses one or more initial graphical element detection techniques to compare the UI element attributes with the UI descriptor attributes of the activity. as well as When a match is not found during the first time period using one or more of the initial graphic element detection techniques: The RPA robot performs one or more additional graphic element detection techniques to replace the one or more initial graphic element detection techniques.

7. The computer-implemented method of claim 6, wherein when a match is not found during the second time period using the one or more additional graphic element detection techniques, the method further comprises: The RPA robot performs one or more supplementary graphic element detection techniques to replace the one or more initial graphic element detection techniques and the one or more additional graphic element detection techniques.

8. The computer-implemented method of claim 1, wherein the method further comprises: only when a match is found: The RPA robot takes actions associated with the activities involving the UI elements.

9. The computer-implemented method according to claim 1, wherein the plurality of graphic element detection techniques includes two or more of selector techniques, computer vision (CV) techniques, image matching techniques, and optical character recognition (OCR) techniques.

10. A non-transitory computer-readable medium storing a computer program configured to cause at least one processor to: Analyze the runtime user interface (UI) to identify UI element attributes; The UI element attributes are compared with UI descriptor attributes for activities in a Robotic Process Automation (RPA) workflow using one or more initial graphical element detection techniques; and When a match is not found using one or more of the initial graphic element detection techniques during the first time period defined by the delay time: One or more additional graphic element detection techniques are performed in parallel with the aforementioned one or more initial graphic element detection techniques, wherein... Unified Targeting Technology is a unified targeting technology that combines serial and delayed parallel execution, configured to employ a combination of multiple graphics element detection techniques. The unified target technology can be customized by RPA developers to specify one or more initial graphic element detection technologies, one or more additional graphic element detection technologies, and the delay time after initiating the one or more initial graphic element detection technologies and before executing the one or more additional graphic element detection technologies.

11. The non-transitory computer-readable medium of claim 10, wherein when a match is not found during the second time period using the one or more initial graphic element detection techniques and the one or more additional graphic element detection techniques, the computer program is further configured to cause the at least one processor to: One or more supplementary graphic element detection techniques are performed in parallel with the one or more initial graphic element detection techniques and the one or more additional graphic element detection techniques.

12. The non-transitory computer-readable medium of claim 10, wherein the computer program is further configured to cause the at least one processor only when a match is found: Take the action associated with the activity involving the UI element.

13. The non-transitory computer-readable medium of claim 10, wherein at least one additional activity in the robotic process automation (RPA) is repeated.

14. The non-transitory computer-readable medium of claim 10, wherein the UI descriptor attribute includes two or more of selector attributes, computer vision (CV) attributes, image matching attributes, and optical character recognition (OCR) attributes.

15. A non-transitory computer-readable medium storing a computer program configured to cause at least one processor to: The designer application receives selections of activities in the Robotic Process Automation (RPA) workflow, which will be configured to perform graphical element detection using a unified target technology. The unified target technology can be customized by the RPA developer to specify one or more initial graphical element detection technologies, one or more additional graphical element detection technologies, and a delay time after initiating the one or more initial graphical element detection technologies and before executing the one or more additional graphical element detection technologies. Analyze the runtime user interface (UI) to identify UI element attributes; The UI element attributes are compared with UI descriptor attributes for activities in a Robotic Process Automation (RPA) workflow using one or more of the aforementioned initial graphical element detection techniques; and When a match is not found during the first time period defined by the delay time using the one or more initial graphic element detection techniques: Perform one or more additional graphic element detection techniques to replace one or more initial graphic element detection techniques. in, The unified target technique is configured to perform the unified target technique in both serial and delayed parallel execution using a combination of multiple graphics element detection techniques.

16. The non-transitory computer-readable medium of claim 15, wherein when a match is not found during the second time period using the one or more additional graphic element detection techniques, the computer program is further configured to cause the at least one processor to: One or more supplementary graphic element detection techniques are performed to replace the one or more initial graphic element detection techniques and the one or more additional graphic element detection techniques.

17. The non-transitory computer-readable medium of claim 15, wherein the computer program is further configured to cause the at least one processor only when a match is found: Take the action associated with the activity involving the UI element.

18. The non-transitory computer-readable medium of claim 15, wherein at least one additional activity in the robotic process automation (RPA) is repeated.

19. The non-transitory computer-readable medium of claim 15, wherein the UI descriptor attributes include two or more of selector attributes, computer vision (CV) attributes, image matching attributes, and optical character recognition (OCR) attributes.

Images