Computer program product and method for creating a user interface for robotic process automation

Abstract

A user interface (UI) object descriptor, a UI object library, a UI object repository, and a UI object browser for robotic process automation (RPA) are disclosed. The UI object browser can be used to manage, reuse, and increase the reliability of UI descriptors in a project. UI descriptors can be added to a UI object library and published or republished as a UI object library for global reuse in a UI object repository. The UI object browser, UI object library, and UI object repository can promote the reusability of a UI element identification framework and its derivatives.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Non-Provisional Patent Application No. 16 / 922,289, filed on July 7, 2020. The subject matter of that previously filed application is hereby incorporated by reference in its entirety. Technical Field

[0003] The present invention generally relates to Robotic Process Automation (RPA), and more particularly, to a User Interface (UI) object descriptor, a UI object library, a UI object repository, and / or a UI object browser for RPA. Background Art

[0004] For RPA automation in user interfaces (UIs), developers traditionally build descriptors (e.g., selectors) for each UI action. Traditionally, a default descriptor is generated. However, building reliable descriptors may consume developer time for troubleshooting and trial-and-error development, which may not be efficient. There is also a lack of reusability of descriptors when the same descriptor is needed. In addition, the process upgrade path may not be simple when systems and applications are changed or upgraded. Therefore, an improved approach may be beneficial. Summary of the invention

[0005] Certain embodiments of the present invention may provide solutions to problems and needs in the prior art that have not been fully identified, understood, or solved by current RPA technology. For example, some embodiments of the present invention relate to UI descriptors, UI object libraries, UI object repositories, and / or UI object browsers for RPA.

[0006] In an embodiment, a computer program for creating a user interface (UI) object library for RPA is embodied on a non-transitory computer readable medium. The program is configured to cause at least one processor to detect an application associated with a screen of the UI and detect UI elements on the screen. The program is also configured to cause at least one processor to receive a selection of a target UI element and generate a UI descriptor for the selected UI element and add the generated UI descriptor of the selected UI element to a corresponding RPA workflow activity. The program is further configured to cause at least one processor to create and publish a UI object library including UI descriptors in a UI object repository.

[0007] In another embodiment, a computer program is embodied on a non-transitory computer readable medium. The program is configured to cause at least one processor to receive a selection of a target UI element on a screen of an application and to generate a UI descriptor for the selected UI element. The program is also configured to cause at least one processor to add the generated UI descriptor of the selected UI element to a corresponding RPA workflow activity. The RPA workflow activity is or includes a CV-based identification activity or a unified target identification activity.

[0008] In another embodiment, the RPA developer application is embodied on a non-transitory computer-readable medium. The RPA developer application is configured to cause at least one processor to detect an application associated with a screen of a UI and detect UI elements on the screen. The RPA application is also configured to cause at least one processor to receive a selection of a target UI element and generate a UI descriptor for the selected UI element. The RPA developer application is also configured to cause at least one processor to add the generated UI descriptor of the selected UI element to a corresponding RPA workflow activity. The RPA workflow activity is or includes a selector-based identification activity, a CV-based identification activity, a unified target identification activity, an image matching identification activity, or an identification activity based on a selector and OCR. The RPA application includes a UI object browser that displays UI descriptors in one or more UI object libraries of a UI object repository, and the one or more UI object libraries are organized by application and version. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to facilitate an understanding of the advantages of certain embodiments of the present invention, a more particular description of the invention briefly described above will be presented by reference to specific embodiments shown in the accompanying drawings. Although it should be understood that these drawings depict only conventional embodiments of the invention and are not to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0010] Figure 1 is an architectural diagram illustrating a robotic process automation (RPA) system according to an embodiment of the present invention.

[0011] Figure 2 is an architectural diagram illustrating a deployed RPA system according to an embodiment of the present invention.

[0012] Figure 3 is an architectural diagram illustrating the relationship between designers, activities, and drivers according to an embodiment of the present invention.

[0013] Figure 4 is an architectural diagram illustrating an RPA system according to an embodiment of the present invention.

[0014] Figure 5is an architectural diagram illustrating a computing system configured to provide a UI descriptor, an object library, a UI object repository, and / or an object browser for RPA according to an embodiment of the present invention.

[0015] Fig. 6A is a screenshot illustrating an RPA designer application according to an embodiment of the present invention.

[0016] Figure 6B It is a graphic 7Screenshot of the calculator standard views screen and the view selector tab.

[0017] Figure 6C is a diagram illustrating an embodiment of the present invention. Fig. 6A An enlarged screenshot view of a sequence in the outline pane of an RPA development application is shown.

[0018] Fig.6D is a diagram illustrating an embodiment of the present invention. Fig. 6A An enlarged screenshot view of the snippet pane of the RPA development application is shown.

[0019] Fig. 6E and 6F are screenshots illustrating activities of an RPA workflow according to an embodiment of the present invention, each activity having a screenshot of a corresponding UI element.

[0020] Figure 6G is a screen shot illustrating a workflow activity options menu according to an embodiment of the present invention.

[0021] Figure 6H is a screenshot illustrating an RPA designer application with properties of a click activity shown in a properties pane in accordance with an embodiment of the present invention.

[0022] Fig.6I is a diagram illustrating an embodiment of the present invention. Figure 6H An enlarged screenshot of the properties pane of the RPA development application is shown.

[0023] Figure 6J is a screenshot illustrating an RPA designer application with a UI object browser pane according to an embodiment of the present invention.

[0024] Figure 6K is a diagram illustrating an embodiment of the present invention. Figure 6J An enlarged screenshot of the UI descriptor view of the UI object browser pane of the RPA development application is shown.

[0025] Figure 6L is a diagram illustrating an embodiment of the present invention. Figure 6JAn enlarged screenshot of the UI descriptor view of the UI object browser pane of the RPA development application is shown.

[0026] Figure 6M is a screenshot illustrating a UI activity view of a UI object browser pane according to an embodiment of the present invention.

[0027] Figure 6N is a screenshot illustrating drag and drop functionality for dropping a selector into a desired activity of an RPA workflow in accordance with an embodiment of the present invention.

[0028] Fig.6O It is a graphic 10 Calculator screenshots.

[0029] Figure 6P is a diagram showing a 10Screenshot of the UI Activity view of the UI Object Browser pane of the Calculator selector.

[0030] Figure 6Q is a screenshot illustrating some of the activities of an RPA workflow according to an embodiment of the present invention, with 10 Updated screenshots of the corresponding UI elements for each activity of the calculator.

[0031] Figure 6R is a screenshot illustrating a publish UI object library window according to an embodiment of the present invention.

[0032] Figure 6S is a screen shot illustrating a Manage Packages window according to an embodiment of the present invention.

[0033] Figure 6T is a screenshot of a UI activity view of a UI object browser pane according to an embodiment of the present invention, wherein 10Calculator selector added to UI object library.

[0034] Fig. 7A is a screenshot illustrating an RPA designer application according to an embodiment of the present invention.

[0035] Figure 7B are screenshots illustrating an RPA workflow that would be configured to use three different UI element detection techniques according to an embodiment of the present invention.

[0036] Figure 7C It is a diagram showing that according to an embodiment of the present invention, after the developer selects "Indicate on Screen", UiPathStudio TM Modified 10 Calculator screenshots.

[0037] Fig.7D is a diagram illustrating a method of generating a 10 screenshots of selected UI elements of the calculator Figure 7B Screen capture of the RPA workflow.

[0038] Fig. 7E is a screen shot illustrating an activity options menu according to an embodiment of the present invention.

[0039] Figure 7F is a screenshot illustrating a selector editor according to an embodiment of the present invention.

[0040] Figure 7G is a screenshot illustrating Add to UI Object Library functionality according to an embodiment of the present invention.

[0041] Figure 7H is a screenshot illustrating a UI descriptor pane according to an embodiment of the present invention.

[0042] Fig.7I is a screenshot illustrating an RPA workflow with detecting screen elements and tagging messages according to an embodiment of the present invention.

[0043] Figure 7J is a screenshot illustrating a search interface according to an embodiment of the present invention.

[0044] Figure 7K are screenshots illustrating CV-based identification activities according to an embodiment of the present invention.

[0045] Figure 7L is a diagram showing that according to an embodiment of the present invention, after the developer selects "Indicate on Range" through the CV click interface, UiPath Studio TM Modified 10 Calculator screenshots.

[0046] Figure 7M is a diagram illustrating an embodiment of the present invention, using UiPath Studio TM Modified 10 Screen shot of the calculator and the CV click interface after the developer makes a target selection.

[0047] Figure 7N is a diagram illustrating an embodiment of the present invention, using UiPath Studio TM Modified 10 Screenshot of the calculator and CV click interface after the developer makes target selection and anchor selection.

[0048] Fig.7Ois a screenshot illustrating a CV-based identification activity in a workflow after the CV-based identification activity is configured with a target and an anchor point according to an embodiment of the present invention.

[0049] Figure 7P is a screenshot illustrating a properties tab with a CV click with descriptor attributes in accordance with an embodiment of the present invention.

[0050] Figure 7Q are screenshots illustrating unified goal-based identification activities according to an embodiment of the present invention.

[0051] Figure 7R is a screenshot illustrating a search interface with a highlighted unified target click activity in accordance with an embodiment of the present invention.

[0052] Figure 7S is a screenshot illustrating a unified target-based identification activity with a nested click activity in accordance with an embodiment of the present invention.

[0053] Figure 7T is a diagram illustrating a method of using UiPath Studio according to an embodiment of the present invention. TM Modified 10 Screenshot of the calculator and unified target selection options window.

[0054] Figure 7U is a diagram illustrating a process of executing a task by UiPath Studio with a selected target according to an embodiment of the present invention. TM Modified 10 Screenshot of the calculator and unified target selection options window.

[0055] Figure 7V is a diagram illustrating a diagram of a process of executing a program by UiPath Studio with a selected target and a selected anchor according to an embodiment of the present invention. TM Modified 10 Screenshot of the calculator and unified target selection options window.

[0056] Figure 7W is a diagram illustrating a method of using UiPath Studio according to an embodiment of the present invention. TM Modified 10 Screen shot of the calculator with the selected target and the selected anchor point verified via the unified target selection options window.

[0057] Figure 7X is a screenshot of an identification activity based on a unified target with screenshots of targets and anchors in a nested click activity in accordance with an embodiment of the present invention.

[0058] Figure 7Yis a screenshot illustrating a click-to-image activity added to a workflow in accordance with an embodiment of the present invention.

[0059] Figure 7Z The diagram shows an embodiment of the present invention, which is implemented by UiPath Studio TM Modified 10 Screenshot of calculator with a box drawn around the image to be clicked and a screenshot of the image being selected from the selection box in the click image activity.

[0060] Figure 7a is a screenshot illustrating a properties tab with click image activity properties in accordance with an embodiment of the present invention.

[0061] Figure 7b is a screenshot illustrating a selector editor of a selector of an image to be clicked according to an embodiment of the present invention.

[0062] Figure 7c is a screenshot illustrating a click OCR text activity in a workflow according to an embodiment of the present invention.

[0063] Figure 7d is a screenshot illustrating a Click OCR Text activity with a screenshot of a button to be clicked in accordance with an embodiment of the present invention.

[0064] Figure 8 is a flow chart illustrating a process for generating a selector-based activity according to an embodiment of the present invention.

[0065] Fig. 9 is a flow chart illustrating a process for generating activities based on CV descriptors according to an embodiment of the present invention.

[0066] Fig.10 is a flow chart illustrating a process for generating unified goal-based activities according to an embodiment of the present invention.

[0067] Fig.11 is a flow chart illustrating a process for generating an image matching activity according to an embodiment of the present invention.

[0068] Fig.12 is a flow chart illustrating a process for generating selector and OCR based activities according to an embodiment of the present invention.

[0069] Fig.13 is a flow chart illustrating a process for generating an RPA workflow using multiple identification techniques according to an embodiment of the present invention.

[0070] Fig.14is a flow chart illustrating a process for creating a UI object library and configuring a project to work with a new version of an application according to an embodiment of the present invention.

[0071] Unless otherwise indicated, like reference numerals refer to corresponding features throughout the drawings. DETAILED DESCRIPTION

[0072] Some embodiments relate to UI descriptors, UI object libraries, UI object repositories, and / or UI object browsers for RPA. UI elements on a screen (e.g., text fields, buttons, labels, menus, check boxes, etc.) can be grouped by application, version of application, application screen, and collection of UI elements, and each screen has multiple UI elements. As used herein, a "screen" is an image of an application UI or a portion of an application UI at a certain point in time. In this context, an "application" or a version of a given application can be a union of screens. In some embodiments, each UI element can be described by one or more UI descriptors. UI elements, UI descriptors, applications, and application screens are all UI objects. In some embodiments, UI elements and screens can be further distinguished into specific types of UI elements (e.g., buttons, check boxes, text fields, etc.) and screens (e.g., top windows, modal windows, pop-up windows, etc.).

[0073] The UI object browser may access UI object libraries that may be grouped by application, version of application, application screen, collection of UI elements, and combinations thereof, etc. In some embodiments, the UI object libraries may be stored in a UI object repository. As used herein, a UI object repository is a collection of UI object libraries. In some embodiments, the UI object repository may be a NuGet repository. TM Feeds, web services, etc.

[0074] The object browser can be used to navigate, manage and edit the UI object library in the UI object repository in some embodiments. In some embodiments, the UI object repository of the UI object library can promote the reliability of the UI descriptors in the management, reuse and increase the project. In some embodiments, the UI descriptor can be added to the UI library and published or republished for global reuse in the UI object repository. The UI object browser can promote the reusability of the UI element identification framework and its derivatives by providing access to the UI object repository and its UI object library.

[0075] To make UI objects reusable, they can be extracted into a UI object library that can be referenced by RPA processes. For example, when a selector or other UI descriptor is modified due to a new version of an application, the library can be recreated (or republished) to include the modified UI descriptor. The RPA process that uses the UI object library can then call the modified version of the UI descriptor. In some embodiments, for the RPA process, references to new application programming interfaces (APIs) associated with the new library can be modified manually by the developer, via a software tool (e.g., a tool that goes through the RPA process and updates UI descriptor references when a selector or other component thereof changes), or automatically.

[0076] In some embodiments, a package of UI objects can be attached as a dependency (e.g., NuGet TM However, in some embodiments, the UI object can be obtained via one or more API calls to a web service. This enables the UI object to be stored remotely and retrieved and used during automation.

[0077] The initial construction of UI descriptors previously required developers to have technical knowledge of what was happening behind the scenes in the UI descriptors. However, many RPA processes are no longer built by software developers. For example, these people can use UiPath StudioX TM Creating RPA workflows that do not require in-depth programming knowledge. Some embodiments allow individuals creating RPA workflows to reuse descriptors, which provides time savings because if a suitable UI descriptor is available to the individual creating the RPA workflow, there is no need for a software developer to build a new UI descriptor from scratch.

[0078] UI elements added to the UI object library can represent selectors for RPA workflows. To automate specific operations in the UI, RPA robots can interact with various windows, buttons, drop-down lists, and / or other graphical elements. Conventionally, this is done using the expected screen position of the UI element. However, this is unreliable.

[0079] Some embodiments use selectors to overcome the problems associated with fixed coordinate identifiers by storing the properties of UI elements and their parent elements in XML fragments. Although selectors can be automatically generated in some embodiments where the UI is static, some software programs, such as some web applications, have layouts and property nodes that have variable values. These changes may not be easy to predict, and this previously required some selectors to be manually generated. However, the object browser of some embodiments can overcome this problem.

[0080] In some embodiments, the selector is a type of UI descriptor that can be used to detect UI elements. In some embodiments, the selector has the following structure:

[0081] <node_1 / ><node_2 / > ...<node_N / >

[0082] The final node represents the GUI element of interest, and all previous nodes represent the parents of that element.<node_1> Often referred to as the root node, and represents the top window of an application.

[0083] Each node may have one or more characteristics that help correctly identify a specific level of the selected application. In some embodiments, each node has the following format:

[0084] <ui_system attr_name_1='attr_value_1'...attr_name_N='attr_value_N' / >

[0085] Each property can have an assigned value, and properties with constant values ​​can be selected. This is because changing the property value each time the app is launched might prevent the selector from correctly identifying the associated element.

[0086] UI object library descriptors can be added directly to RPA workflow activities, saving developers the time required to create custom selectors for activities. The object browser can provide a database for storing created selectors in the object library to achieve reusability of UI descriptors. The object library is defined herein as a collection of UI descriptors corresponding to one or more screens from a version of an application. A UI descriptor is a set of instructions for finding UI elements. In some embodiments, a UI descriptor is an encapsulated data / structure format that includes (multiple) UI element selectors, (multiple) anchor selectors, (multiple) computer vision (CV) descriptors, (multiple) unified target descriptors, (multiple) screen image captures (contexts), element image captures, other metadata (e.g., applications and application versions), combinations thereof, and the like. The encapsulated data / structure format can be expanded with future updates of the platform and is not limited to the above definition. Without departing from the scope of the present invention, any suitable UI descriptor for identifying UI elements on the screen may be used. The unified target descriptor links multiple types of UI descriptors together. The unified target descriptor may function like a finite state machine (FSM), where in a first context, a first UI descriptor mechanism applies, in a second context, a second UI descriptor applies, and so on.

[0087] In some embodiments, the RPA designer application may ask the user what type of application(s) the user plans to automate. For example, the user may specify Etc. Since RPA applications already include screen descriptors in the UI object library for those applications, the applications can be programmed with logic on how to automate these applications. Out-of-the-box selectors for various versions of these applications may be available, and the user may be able to specify the version(s) to be automated.

[0088] As used herein, the terms "user" and "developer" are used interchangeably. A user / developer may or may not have programming and / or technical knowledge. For example, in some embodiments, a user / developer may create an RPA workflow by configuring the activities in the RPA workflow without manual coding. In certain embodiments, this may be accomplished by, for example, clicking and dragging various features.

[0089] Conventionally, making selectors reliable requires testing, understanding the inner details of the selector, etc. This makes it difficult or impossible for people without technical knowledge to create / improve selectors. By providing a library of functioning selectors for common UI elements and tasks, some embodiments allow non-technical people to develop automations that work in production. A user can indicate a screen of the application version to be automated, and the RPA development application can fetch that screen. The user can then interact with the screen to define what he or she wants to do.

[0090] In some embodiments, as new UI descriptors are created and / or existing UI descriptors are modified, a global database of UI object libraries can be established that can be shared, collaborative, and potentially open source. In some embodiments, taxonomies and ontologies can be used. Applications, versions, screens, UI elements, descriptors, etc. can be defined as a taxonomy, which is a hierarchical structure of subcategories.

[0091] However, many real-world concepts are not easily organized by classification. Instead, they may be closer to the concept of mathematical ontology. In an ontology, the relationship between categories is not necessarily hierarchical. For example, the situation where a button on a screen takes the user to another screen when clicked cannot be easily captured by the classification of that screen because the next screen is not in the hierarchical structure. When building a graph representing this situation, the object browser of the application can be modeled as an ontology that allows interactions to be created between UI elements on the same screen or on different screens, and provides more information about how UI elements are related to each other.

[0092] Consider the example of clicking the OK button to proceed to the employee screen. The structure of the ontology allows the designer application to suggest to the user to screen employees on the next screen. The ontological information of the relationships between these screens allows the designer application to do so via the OK button. By defining a graph structure that is not necessarily a tree, but rather a graph structure that is relevant to what the application is actually doing, more complex and rich relationships can be captured.

[0093] Some embodiments pertain to or include a recorder that allows for faster acquisition of screens and UI elements from an application. For example, if a given screen has 250 different UI elements, it may take some time to acquire them one by one. A screen pointer can be used to acquire all editable UI elements. This may be helpful in creating a first version of a tree or ontology.

[0094] In some embodiments, the object browser can be exposed as a web service. If the user updates the selector or other UI descriptor at runtime, the RPA robot may call the latest version of the service if it is properly configured. This allows the RPA robot to always use the current version of the UI descriptor.

[0095] UI descriptors can be extracted from activities in RPA workflows and added to a structured pattern that groups UI descriptors by UI application, screen, and UI element. In some embodiments, the UI descriptor can be part of a project for extensive reuse, part of a global repository for testing purposes, or part of a UI object library for global cross-project sharing. In some embodiments, the object library is an encapsulation of UI descriptors grouped by application, version of the application, and screen. UI descriptors can be defined and added to an object library, which in some embodiments can be installed as a dependency in other projects after publishing. This makes the object library of UI descriptors available for reuse. In some embodiments, the UI application is a target application with multiple versions, each version having multiple screens.

[0096] UI object repositories and UI descriptor reuse may be beneficial for a variety of reasons. For example, if the location, appearance, and / or functionality of a UI element in an application changes, the UI descriptor may be changed, and the change may then be propagated to the activities that use the UI descriptor. Thus, reusability may be provided at the level of identifying graphical elements in a UI.

[0097] This may be particularly advantageous for adapting to new versions of applications. For new versions, software developers may only need to update UI descriptors and / or develop some new selectors from scratch, thereby greatly reducing development time. For example, for web pages, hypertext markup language (HTML) can be checked to obtain the path of the UI element for the corresponding selector. The selector of the corresponding UI element may be different in different web browsers and / or different versions of the same web browser. This concept may also be applicable to visual desktops, servers, smart phones and tablet computer applications. The UI object library may include the actual selector of the application, which is just one type of UI descriptor according to the above description. For one or more applications, the UI object library may have multiple different types of UI descriptors. The UI descriptor may also refer to different technologies for building applications. For example, different UI descriptors can be created and used for independent desktops, web and mobile applications. For a technology, you may wish to use a selector, and for another technology, you may wish to use a CV descriptor, etc.

[0098] In some embodiments, the UI descriptor can work with a unified target by performing image detection and definition in a unified target that covers all UI detection mechanisms. The unified target can merge multiple techniques for identifying and automating UI elements into a single cohesive approach. The unified target can prioritize selector-based and driver-based UI detection mechanisms, and fall back to CV to find an image if the first two mechanisms are unsuccessful. In some embodiments, the selector editor and UI browser can be configured to support the unified target.

[0099] In some embodiments, the designer application includes a portion or panel that is an "object browser" that loads object libraries from one or more sources. The object browser may allow the user to visualize the entire collection of UI objects and use any of the UI descriptors via drag and drop and wizards or contextual actions. This may allow the user to select an action to be performed in an RPA workflow activity, such as click, get text, type, etc.

[0100] An example of an object library structure is provided below.

[0101] ·SAP

[0102] Version 1

[0103] Screen 1

[0104] Subject 1

[0105] Object 2

[0106] ·…

[0107] ·Screen 2…

[0108] ·…

[0109] Version 2…

[0110] ·…

[0111] Salesforce…

[0112] ·…

[0113] It should be noted that the above hierarchical structure is provided as an example only. Without departing from the scope of the present invention, any desired number of levels in the hierarchical structure and the elements associated with each level may be used. In some embodiments, the user may define the application map according to his or her needs. In addition, in some embodiments, some nodes in the tree may only play an administrative role, rather than a functional role. For example, UI elements may be combined into a container without a UI descriptor. In some embodiments, the container may exist only for the purpose of grouping.

[0114] In some embodiments, a user can define a scope in the designer application by selecting from a set of screens available in an object library. When a scope is bound one-to-one to a screen, the user can add UI actions (e.g., click, get text, etc.), and then use IntelliSense for potential candidate options to map these actions to UI objects. In some embodiments, the object set can be limited to a list of screen definitions within a UI object repository (e.g., a database of UI object libraries).

[0115] For example, once the structure of the application screens, UI elements, etc. is determined, a list of identified applications may be provided. In some embodiments, the user may then select an application for automation, drag the selected application onto the canvas, write "SAP", press ".", and all applications may be listed. Screens. After selecting a screen, additional candidate elements, functions, etc. may appear. This allows the user to select predicted UI elements, functions, applications, and screens.

[0116] In some embodiments, for example, when in UiPath Studio TM When OLDB is configured in , "Indicate on Screen" can look at OLDB and pick up UI descriptors if available. If UI descriptors are not available, UI descriptors can be defined by the user by indicating the missing UI elements. UI descriptors can then be generated and published in the UI object library.

[0117] In some embodiments, the use of UI descriptors and UI object repositories creates relationships and data that can be further expanded. In some embodiments, UI steps from an automated process can be mapped to a single screen. For example, a screen that includes a form where a user adds and retrieves some data and then clicks submit can be recorded using a single screenshot that includes a screen capture and each UI step performed on that screen, such as: (1) copy quantity; (2) copy number; (3) add name; and (4) click "OK".

[0118] In the case of upgrade processes affected by system and application upgrades, a "find reference" service can be implemented. The find reference service provides the ability to scan a collection of items from a repository and detect processes that are using UI descriptors belonging to a version of an application. In some embodiments, the application version can be determined based on the corresponding screen of the UI descriptor. In addition, a service for detecting differences between UI descriptors with matching elements from two different versions of an application can be implemented. The detected differences in processes and UI descriptors provide information about processes that may be interrupted when the application or system is upgraded. These processes can be automatically updated to use UI descriptors from the new application version.

[0119] In some embodiments, OLDB analysis can be used to enhance the OLDB UI object repository. A mechanism for extracting UI descriptors can be implemented. The extracted UI descriptor data can be sent via OLDB analysis before or after the creation / definition time. In addition, in some embodiments, the extracted UI descriptor data can give a new data set that can be used to use artificial intelligence (AI) and machine learning (ML) techniques to improve the strength of the UI descriptors for each UI element of any screen from any version of any application.

[0120] Certain embodiments may be used for Robotic Process Automation (RPA). Figure 1 1 is an architectural diagram illustrating an RPA system 100 according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that allows developers to design and implement workflows. The designer 110 may provide solutions for application integration and automation of third-party applications, management of information technology (IT) tasks, and business IT processes. The designer 110 may facilitate the development of automation projects, which are graphical representations of business processes. In short, the designer 110 facilitates the development and deployment of workflows and robots.

[0121] Automation projects enable automation of rule-based processes by providing developers with control over the execution order and relationships between a custom set of steps developed in a workflow, defined herein as "activities". One commercial example of an embodiment of the designer 110 is UiPath Studio TMEach activity may include an action, such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0122] Some types of workflows may include, but are not limited to, sequences, flow charts, FSMs, and / or global exception handlers. Sequences may be particularly suitable for linear processes, supporting the flow from one activity to another without cluttering the workflow. Flow charts may be particularly suitable for more complex business logic, implementing the integration of decisions and the connection of activities in a more diverse manner through multiple branching logic operators. FSMs may be particularly suitable for large workflows. FSMs may use a limited number of states in their execution, which are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suitable for determining workflow behavior and debugging processes when execution errors are encountered.

[0123] Once a workflow is developed in the designer 110, the execution of the business process is coordinated by the commander 120, which coordinates one or more robots 130 that execute the workflow developed in the designer 110. One commercial example of an embodiment of the commander 120 is the UiPath Orchestrator TM Director 120 facilitates the creation, monitoring, and deployment of resources in a management environment. Director 120 may serve as one of the integration or aggregation points for third-party solutions and applications.

[0124] The commander 120 can manage a queue of robots 130, connecting and executing robots 130 from a centralized point. The types of robots 130 that can be managed include, but are not limited to, attended robots 132, unattended robots 134, development robots (similar to unattended robots 134, but for development and testing purposes), and non-production robots (similar to attended robots 132, but for development and testing purposes). Attended robots 132 can be triggered by user events or scheduled to occur automatically and operate with humans on the same computing system. Attended robots 132 can be used with commanders 120 for centralized process deployment and recording media. Attended robots 132 can help human users complete various tasks and can be triggered by user events. In some embodiments, processes cannot be started from the commander 120 on this type of robot, and / or they cannot be run under a locked screen. In some embodiments, attended robots 132 can only be started from the robot tray or from a command prompt. In some embodiments, attended robots 132 should be run under human supervision.

[0125] Unattended robots 134 run unattended in a virtual environment or on a physical machine and can automate multiple processes. Unattended robots 134 can be responsible for remote execution, monitoring, scheduling, and providing support for work queues. In some embodiments, debugging for all robot types can be run from the designer 110. Both attended and unattended robots can automate various systems and applications, including but not limited to mainframes, web applications, virtual machines, enterprise applications (e.g., and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0126] The commander 120 may have various capabilities, including, but not limited to, provisioning, deploying, versioning, configuring, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between robots 130 and a commander 120 (e.g., a web application). Deployment may include ensuring that package versions are correctly delivered to designated robots 130 for execution. In some embodiments, versioning may include the management of unique instances of some processes or configurations. Configuration may include the maintenance and delivery of robot environments and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs into a database (e.g., a SQL database) and / or another storage mechanism (e.g., a SQL database) that provides the ability to store and quickly query large data sets. ). The director 120 may provide interconnectivity by acting as a centralized communication point for third-party solutions and / or applications.

[0127] Robots 130 are execution agents that run workflows built into designer 110. One commercial example of some embodiments of robot(s) 130 is UiPath Robots TM In some embodiments, the robot 130 installs Microsoft Services managed by the Service Control Manager (SCM). As a result, such robots 130 can open interactive session, and has Permissions for services.

[0128] In some embodiments, the robots 130 can be installed in user mode. For such robots 130, this means that they have the same permissions as the user if a given robot 130 is installed. This feature is also available for high density (HD) robots, which ensures that the maximum potential of each machine is fully utilized. In some embodiments, any type of robot 130 can be configured in an HD environment.

[0129] The robot 130 in some embodiments is divided into several components, each dedicated to a specific automation task. The robot components in some embodiments include but are not limited to SCM-managed robot services, user mode robot services, executors, agents, and command lines. SCM-managed robot service management and monitoring The SCM in the local system starts the console application.

[0130] In some embodiments, user-mode robot services manage and monitor The user-mode robot service can be trusted and manage the credentials of the robot 130. If the SCM-managed robot service is not installed, then The app may start automatically.

[0131] The actuator can be The executor may be aware of the dots per inch (DPI) setting of each monitor. The agent may be a window that displays available jobs in the system tray. Presentation Foundation (WPF) applications. Agents can be clients of services. Agents can request to start or stop jobs and change settings. Command lines are clients of services. Command lines are console applications that can request to start jobs and wait for their output.

[0132] Splitting the components of the robot 130 as described above helps developers, support users, and computing systems to more easily run, identify, and track what each component is executing. Special behaviors can be configured for each component in this way, such as setting different firewall rules for executors and services. In some embodiments, the executor may always know the DPI setting of each monitor. As a result, workflows can be executed at any DPI, regardless of the configuration of the computing system on which they were created. In some embodiments, projects from the designer 110 can also be independent of the browser zoom level. For applications that are DPI-unaware or intentionally marked as DPI-unaware, DPI can be disabled in some embodiments.

[0133] Figure 2 2 is an architectural diagram illustrating an RPA system 200 deployed according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be Figure 1 The RPA system 100 may be Figure 1It should be noted that the client, server, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 210 includes an executor 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not be running on the computing system 210. The executor 212 is running a process. Several business projects may be running at the same time, such as Figure 2 In this embodiment, the agent 214 (e.g., The Executor 212 service is the single point of contact for all Executors 212. All messages in this embodiment are logged to the Director 230, which further processes them via the Database Server 240, the Indexer Server 250, or both. Figure 1 As discussed, effector 212 may be a robotic component.

[0134] In some embodiments, a robot represents an association between a machine name and a user name. A robot can manage multiple executors simultaneously. In a computing system that supports multiple interactive sessions running simultaneously (e.g., Server2012), multiple robots can run simultaneously, each with a unique username in a separate This is the HD robot mentioned above.

[0135] The agent 214 is also responsible for sending the status of the robot (e.g., periodically sending a "heartbeat" message indicating that the robot is still running) and downloading the required version of the package to be executed. In some embodiments, communication between the agent 214 and the commander 230 is always initiated by the agent 214. In the notification scenario, the agent 214 can open a WebSocket channel that is later used by the commander 230 to send commands to the robot (e.g., start, stop, etc.).

[0136] On the server side, it includes a presentation layer (web application 232, Open Data Protocol (OData) Representational State Transfer (REST) ​​application programming interface (API) endpoint 234, and notification and monitoring 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240 and indexer server 250). The commander 230 includes a web application 232, an OData REST API endpoint 234, notification and monitoring 236, and an API implementation / business logic 238. In some embodiments, most actions performed by a user in the interface of the commander 230 (e.g., via a browser 220) are performed by calling various APIs. Without departing from the scope of the present invention, such actions may include, but are not limited to, starting jobs on a robot, adding / removing data in a queue, scheduling jobs to run unattended, etc. The web application 232 is the visual layer of the server platform. In this embodiment, the web application 232 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, without departing from the scope of the present invention, any desired markup language, scripting language, or any other format may be used. In this embodiment, the user interacts with a web page from web application 232 via browser 220 to perform various actions to control commander 230. For example, the user can create robot groups, assign packages to robots, analyze logs for each robot and / or each process, start and stop robots, etc.

[0137] In addition to the web application 232, the director 230 also includes a service layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the present invention. The REST API is used by both the web application 232 and the agent 214. In this embodiment, the agent 214 is a supervisor of one or more robots on a client computer.

[0138] The REST API in this embodiment covers configuration, logging, monitoring, and queuing functionality. In some embodiments, the configuration endpoint can be used to define and configure application users, permissions, robots, assets, versions, and environments. The logging REST endpoint can be used to log different information, such as errors, explicit messages sent by the robot, and other environment-specific information. If the start job command is used in the commander 230, the robot can use the deployment REST endpoint to query the package version that should be executed. The queue REST endpoint can be responsible for queue and queue item management, such as adding data to the queue, getting transactions from the queue, setting the status of transactions, etc.

[0139] The monitoring REST endpoint can monitor the web application 232 and the agent 214. The notification and monitoring API 236 can be a REST endpoint for registering the agent 214, delivering configuration settings to the agent 214, and sending / receiving notifications from the server and the agent 214. In some embodiments, the notification and monitoring API 236 can also use WebSocket communication.

[0140] In this embodiment, the persistence layer includes a pair of servers - a database server 240 (e.g., SQL server) and an indexer server 250. The database server 240 in this embodiment stores configurations for robots, robot groups, associated processes, users, roles, schedules, etc. In some embodiments, this information is managed by the web application 232. The database server 240 can manage queues and queue items. In some embodiments, the database server 240 can store messages logged by the robot (in addition to or in place of the indexer server 250).

[0141] In some embodiments, the optional indexer server 250 stores and indexes the information recorded by the robot. In some embodiments, the indexer server 250 can be disabled through configuration settings. In some embodiments, the indexer server 250 uses It is an open source project full-text search engine. Messages logged by the robot (e.g., using activities like log message or write line) can be sent through the logging REST endpoint(s) to the indexer server 250 where they are indexed for future use.

[0142] Figure 3310, activities 320, 330 and drivers 340 according to an embodiment of the present invention. As described above, developers use the designer 310 to develop workflows executed by robots. The workflow may include user-defined activities 320 and UI automation activities 330. Some embodiments are capable of identifying non-textual visual components in images, which are referred to as computer vision (CV) in this article. Some CV activities related to these components may include, but are not limited to, clicking, typing, getting text, hovering, element existence, refreshing range, highlighting, etc. In some embodiments, clicking uses, for example, CV, optical character recognition (OCR), fuzzy text matching and multiple anchors to identify an element and click it. Typing can use the above and type in the element to identify the element. "Get text" can identify the location of a specific text and use OCR to scan it. Hovering can identify an element and hover over it. Element existence can use the above technology to check whether the element exists on the screen. In some embodiments, there may be hundreds or even thousands of activities that can be implemented in the designer 310. However, without departing from the scope of the present invention, any number and / or type of activities are available.

[0143] UI automation activities 330 are a subset of special low-level activities (e.g., UI activities) written in low-level code, and facilitate interaction with applications through the UI layer. In some embodiments, for example, UI automation activities 300 can simulate "user input through window messages, etc.". UI automation activities 330 facilitate these interactions via drivers 340 that allow the robot to interact with the desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc.

[0144] Drivers 340 can interact with the OS at a low level to find hooks, monitor keys, etc. They can facilitate For example, the "click" activity performs the same role in these different applications via the driver 340.

[0145] Figure 4 4 is an architectural diagram illustrating an RPA system 400 according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be or include Figure 1 The RPA system 100 and / or 200 of and / or 2. The RPA system 400 includes a plurality of client computing systems 410 running robots. The computing system 410 can communicate with a commander computing system 420 via a web application running thereon. The commander computing system 420 can in turn communicate with a database server 430 and an optional indexer server 440.

[0146] about Figure 1and Figure 3 It should be noted that although web applications are used in these embodiments, any suitable client and / or server software may be used without departing from the scope of the present invention. For example, the director may run a server-side application that communicates with a non-web-based client software application on a client computing system.

[0147] Figure 5 1 is an architectural diagram of a computing system 500 configured to provide a UI descriptor, UI object library, UI object repository, and / or object browser for RPA according to an embodiment of the present invention. In some embodiments, the computing system 500 may be one or more computing systems depicted and / or described herein. The computing system 500 includes a bus 505 or other communication mechanism for transmitting information, and a (multiple) processor 510 coupled to the bus 505 for processing information. The (multiple) processors 510 may be any type of general or special-purpose processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The (multiple) processors 510 may also have multiple processing cores, and at least some of the cores may be configured to perform specific functions. In some embodiments, multi-parallel processing may be used. In some embodiments, at least one of the (multiple) processors 510 may be a neuromorphic circuit including a processing element that simulates a biological neuron. In some embodiments, the neuromorphic circuit may not require traditional components of the von Neumann computing architecture.

[0148] The computing system 500 further includes a memory 515 for storing information and instructions to be executed by the processor(s) 510. The memory 515 may be composed of any combination of random access memory (RAM), read-only memory (ROM), flash memory, cache, static memory such as a magnetic disk or optical disk, or any other type of non-transitory computer-readable medium or a combination thereof. The non-transitory computer-readable medium may be any available medium that can be accessed by the processor(s) 510, and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both.

[0149] In addition, the computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communication network via wireless and / or wired connections. In some embodiments, the communication device 520 may be configured to use frequency division multiple access (FDMA), single carrier FDMA (SC-FDMA), time division multiple access (TDMA), code division multiple access (CDMA), orthogonal frequency division multiple access (OFDM), orthogonal frequency division multiple access (OFDMA), global system for mobile communications (GSM), general packet radio service (GPRS), universal mobile telecommunications system (UMTS), cdma2000, wideband CDMA (W-CDMA), high speed downlink packet access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), High Speed ​​Packet Access (HSPA), Long Term Evolution (LTE-A), Advanced LTE (LTE-A), 802.11x, Wi-Fi, Zigbee, Ultra Wideband (UWB), 802.16x, 802.15, Home Node B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near Field Communication (NFC), Fifth Generation (5G), New Radio (NR), any combination thereof, and / or any other currently existing or future implemented communication standards and / or protocols. In some embodiments, without departing from the scope of the present invention, the communication device 520 may include one or more antennas, which are single antennas, array antennas, phased antennas, switched antennas, beamforming antennas, beam steering antennas, combinations thereof, and / or any other antenna configurations.

[0150] The processor(s) 510 are also coupled to a display 525 via the bus 505, such as a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, Display, in-plane switching (IPS) display, or any other suitable display for displaying information to a user. Display 525 can be configured as a touch (tactile) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, etc. using resistance, capacitance, surface acoustic wave (SAW) capacitance, infrared, optical imaging, dispersion signal technology, acoustic pulse recognition, frustrated total internal reflection, etc. Any suitable display device and tactile I / O can be used without departing from the scope of the present invention.

[0151] A keyboard 530 and a cursor control device 535, such as a computer mouse, touchpad, etc., are further coupled to bus 505 to enable a user to interact with computing system 500. However, in some embodiments, a physical keyboard and mouse may not be present, and a user may interact with the device only through display 525 and / or touchpad (not shown). Any type and combination of input devices may be used, depending on design choice. In some embodiments, there is no physical input device and / or display. For example, a user may interact with computing system 500 remotely via another computing system in communication with it, or computing system 500 may operate autonomously.

[0152] The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. These modules include an operating system 540 of the computing system 500. These modules further include a UI object operation module 545, which is configured to perform all or part of the processes described herein or derivatives thereof. The computing system 500 may include one or more additional functionality modules 550 that include additional functionality.

[0153] Those skilled in the art will appreciate that, without departing from the scope of the present invention, a "system" may be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device or combination of devices. Presenting the above functions as being performed by a "system" is not intended to limit the scope of the present invention in any way, but is intended to provide an example of many embodiments of the present invention. In fact, the methods, systems, and devices disclosed herein may be implemented in localized and distributed forms consistent with computing technology (including cloud computing systems).

[0154] It should be noted that some of the system features described in this specification have been presented as modules in order to more specifically emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising a custom very large scale integrated (VLSI) circuit or gate array, an off-the-shelf semiconductor such as a logic chip, a transistor, or other discrete components. A module may also be implemented in a programmable hardware device such as a field programmable gate array, a programmable array logic, a programmable logic device, a graphics processing unit, etc.

[0155] Modules may also be implemented at least in part in software for execution by various types of processors. An identified unit of executable code may, for example, include one or more physical or logical blocks of computer instructions, which may, for example, be organized as objects, processes, or functions. However, the executable code of the identified module need not be physically located together, but may include different instructions stored in different locations, which, when logically combined together, constitute the module and implement the stated purpose of the module. In addition, without departing from the scope of the present invention, the module may be stored on a computer-readable medium, which may be, for example, a hard drive driver, a flash memory device, a RAM, a magnetic tape, and / or any other such non-transitory computer-readable medium for storing data.

[0156] In practice, an executable code module may be a single instruction or multiple instructions, and may even be distributed across several different code segments, different programs, and several storage devices. Similarly, operational data may be identified and illustrated herein within a module, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed across different locations including different storage devices, and may exist, at least in part, simply as an electronic signal on a system or network.

[0157] Some embodiments store an object library corresponding to a set of UI descriptors for screens from a version of an application in a database. When a user develops RPA UI automation, the object library can be applied by an RPA designer (e.g., UiPathStudio TM ) is created. This can help create a more extensible and adaptable collection of UI objects that can be used to accommodate post-design-time changes (such as a new version of an application).

[0158] Consider a situation where employees use one or more RPA robots to perform tasks on a software application running on their computers. Then, the Information Technology (IT) department upgrades the software application with which the RPA robots interact. If the company's RPA team did not make changes to the modified UI descriptors, called RPA robot workflows, ahead of time, some RPA processes may now break because the RPA robots can no longer find certain UI elements in the application due to changes in the new version (e.g., changed UI element functionality and / or appearance, UI elements were completely removed or moved to new screens, etc.). If UI descriptors for both versions of the UI elements already exist, the new version of the UI descriptor can be swapped with the previous version of the UI descriptor in the appropriate activity in the RPA workflow, and the RPA robots should then work on the new version.

[0159] Fig. 6A6 is a screenshot illustrating an RPA designer application 600 according to an embodiment of the present invention. The RPA designer application 600 includes a main RPA development pane 610, which includes an RPA workflow 612 (in this example, a sequence). The "Excerpt" pane 620 includes reusable components or common automation items that can be reused and applied to multiple workflows. The properties pane 640 displays the properties of the selected activity 614 of the RPA workflow 612.

[0160] In this embodiment, the RPA workflow 612 is 7 A series of actions are performed on the standard screen (or window) 670 of the calculator. Figure 6B The user can select from different screens for different calculator settings using the View tab 672. The sequence of actions in the RPA workflow 612 is click button 2, click the plus button, click button 3, click the equals button, get the text of the result, click the clear button, and write the result via the "write row" operation. Figure 6C An enlarged view of the sequence in outline pane 630 is shown.

[0161] Fig.6D An enlarged view of the snippet pane 620 is shown. The snippet pane 620 includes a workflow snippet, which can be a ready-made workflow that a developer can call / include in his or her own workflow (e.g., RPA workflow 612) in the RPA designer application 600. In some embodiments, the snippet pane 620 can provide a sandbox in which a team company, all interested developers, or any other group of developers can save UI descriptors of elements to be automated. The snippet pane 620 includes UI descriptors 621 arranged by application. Under the calculator application 622, 10 Selectors are organized in 10 version 623, standard window 624, target UI element 625 and corresponding UI element selector 626. 7 Selectors are organized in 7 version 623A, standard window 624A, target UI element 625A, and corresponding UI element selector 626A. Although a selector is used in this embodiment, in some embodiments, a CV descriptor, a unified target descriptor, or both may be included in addition to or instead of a UI element selector.

[0162] Fig. 6E and Fig. 6F The activities of an RPA workflow sequence 612 are shown. 7 Screenshots of the portion of the standard calculator screen corresponding to each of the activities are displayed for that activity and highlighted if applicable. For example, UiPath Studio TM The "indication" functionality in selects the corresponding UI element of the activity, which will be discussed in more detail below regarding the use of multiple UI descriptor types for activities. In this way, the user can intuitively verify whether the RPA workflow should interact with the expected UI element. By clicking the button 615 of the activity that interacts with the UI element, a menu 616 appears. See Figure 6G The user may add the corresponding UI object to the object browser by selecting the Add to UI Object option 618 .

[0163] Figure 6H The RPA designer application 600 is illustrated with properties of a click activity shown in the properties pane 640 . Fig.6I An enlarged view of the properties pane 640 is shown. At the bottom of the properties pane 640, the user can select the UI object browser via the UI object browser tab 652.

[0164] Figure 6J The RPA designer application 600 is illustrated with a UI descriptor view of a UI object browser pane 650, and Figure 6K An enlarged view of a UI object browser pane 650 with a UI descriptor view is shown. The UI descriptor view includes a project UI descriptor 654A and a UI object library 656A. Figure 6K No UI descriptor has been selected for the project.

[0165] To add UI descriptors, the developer can add these descriptors using the excerpt pane 620. For example, the developer can add the UI descriptors in the excerpt pane 620. 7 version 623A, which causes menu 627 to appear. The developer can choose to add these UI descriptors to the project, which causes them to appear under the project UI descriptors 654A in the UI object browser pane 650, which displays them as UI descriptors 658A. Figure 6L .exist Figure 6MIn the UI activity view of the UI object browser pane 650 shown, project 654B and UI activity 656B are shown. Although the RPA workflow may have various activities of different types, the UI activity view lists the UI activities 656B so that the developer can easily view them. The UI activity 656B can be extracted and has a similar structure as shown in the excerpt pane 620. In some embodiments, all UI descriptors that have been developed and shared in the library can be displayed in the excerpt pane 620, while only the UI descriptors used by the UI activities appear in the UI activity view of the UI object browser pane 650.

[0166] Once the selector is in the object browser pane 650, the user can drag and drop a given selector to the desired activity, such as Figure 6N As shown. This provides project-level reusability. In some embodiments, in addition to or instead of the selector, other UI descriptors such as CV descriptors, unified target descriptors, etc. can be dragged and dropped into the desired activity. Once the selector is dropped into the corresponding activity, the RPA workflow 612 is ready to be executed by the RPA robot to interact with the corresponding UI element.

[0167] Consider what developers want to 10Calculator (see Fig.6O of 10 The same RPA process is executed by the calculator 680. For activities that interact with various UI elements, the operation is 7. Calculator-designed RPA processes may fail because 10 Calculator UI elements and The UI elements of the 7 calculators are quite different. Figure 6B If it is 10 The selector is developed by the calculator and is available in the published library of UI descriptors in the snippet pane 620, as in this example, the user can delete the 7Calculator selector, and add 10 calculator selectors, which then appear in the UI active view of the UI object browser pane 650. 10Figure of calculator selector.

[0168] When the target application is set to 10, the screenshot portion of each activity and the corresponding selector are updated in the RPA workflow 612. Figure 6Q . RPA workflow 612 will not work in 10. This provides project-wide reusability. However, the selectors are specific to this project and are not yet available globally as part of the UI object library.

[0169] To "upgrade" the calculator app, change the UI descriptors of the UI elements to the UI descriptors of the new version of the app. For example, the NuGet that includes these UI descriptors TM Packages can be created, uploaded, and then retrieved. Therefore, when the dependencies of a UI element change, the UI descriptor will be updated.

[0170] In order to provide global reusability so that all processes can use UI descriptors, a UI object library 656A can be used. In this embodiment, by clicking the publish button 658 (see Figure 6P ), a publish UI object library window 660 appears. Figure 6R This allows selectors and / or other UI descriptors from a project to be published into a package (e.g., to NuGet TM The package can be accessed by other developers and used in other RPA processes after being published via the manage packages window 662. Figure 6S Installing the calculator descriptor package as a dependency of the project causes the selector / UI descriptor to appear under Libraries 656A in the UI object browser pane 650. See Figure 6T The UI Object Library can then be used in a given project. As a best practice, selectors and / or other UI descriptors used to interact with UI elements should only be imported as dependencies from the UI Object Library if they are needed for reuse. However, if a given selector or other UI descriptor is specific to a given project and is unlikely to be reused, some selectors / UI descriptors can be created local to that project, while others can be imported from the UI Object Library.

[0171] Fig. 7A is a screenshot illustrating an RPA designer application 700 according to an embodiment of the present invention. The RPA designer application 700 includes a main RPA development pane 702, which includes an RPA workflow 710 (in this example, a sequence of activities including different UI elements identifying activities). The project pane 704 includes the dependencies of the current project. The UI descriptor pane 706 currently does not include a selection of project UI descriptors and UI object libraries.

[0172] Figure 7B is an enlarged view illustrating an RPA workflow 710. Activity 720 will use selector-based identification, activity 730 will use CV-based identification, and activity 740 will use unified target-based identification. The configuration of activities 720, 730, 740 of this example is described below.

[0173] Figure 7C Diagram showing the TM Modified 10 Calculator 750. When the user selects "Indicate on screen" 721 for the selector-based identification activity 720 (see Figure 7B ), the identified element of the UI over which the developer moves the mouse is highlighted. For example, in Figure 7C , button 752 is highlighted. When the user clicks button 752, a screenshot 722 including the selected UI element appears in the selector-based identification activity 720. Fig.7D .

[0174] By clicking on button 724 of the selected activity 720, a menu 725 appears. Fig. 7E . Select the "Edit Selector" option to open the Selector Editor 726. See Figure 7F The selector editor 726 allows the developer to edit the properties of a selector and verify that the selector is in 10 Find the corresponding UI element (i.e., 9 button 752) in the calculator 750. Select "Highlight" to make the corresponding UI element 10 is highlighted on the calculator 750. After creating the selector, the developer can click on the button 724 of the selector-based identification activity 720 and select the "Add to UI Object" option on the menu 725, which causes the Add to UI Object Library window 728 to appear. Figure 7G The developer can then add the selector created for the 9 button to the project's UI object library. The selector now appears in the UI descriptor pane 706. See Figure 7H .

[0175] To configure the CV-based identification activity 730, in this embodiment, the user should first indicate the screen on which the CV will be executed. When the user selects "Indicate on Screen" 731 (see Figure 7B ), the user clicks 10 Calculator 750 UI, which enables UiPath Studio TM Detect elements (using CV) and labels (using the selected OCR engine 732 - in this case OCR), causing message 733 to appear. Fig.7I Once detected 10 Elements and labels in the UI of the calculator 750, a screenshot of the detected UI element 734 appears in the CV-based identification activity 730. Figure 7K. A CV click activity 735 may be used, which may be configured to identify a 9 button using a CV. This activity may be added as a nested activity within a CV-based identification activity 730 using a search interface 708, which may appear when a developer clicks the plus icon in a given activity. See Figure 7J .

[0176] When the developer clicks on the indication on the scope 736 for the CV click activity 735, 10 Calculator 750 appears with CV click interface 754. Figure 7L However, when the user clicks 9 button 752, the button is not uniquely identified using the CV. Figure 7M CV click interface 754 displays a message to the developer that duplicate matches were found and asks the developer to add anchors. The developer then adds anchors for button 752 (e.g., using selected anchors 756, 757 and candidate anchors 758 of the anchor selection functionality) until button 752 is matched based on its match with the 10 The relationship of the other anchor UI elements in calculator 750 is uniquely identified as the target UI element. After 9 the CV descriptor of button 752 uniquely identifies it using the target and (multiple) anchors, the screenshot 737 appears in the CV click activity 735. See Fig.7O .exist Fig.7O , screen shot 737 shows a version without the target and anchor points selected correctly, but this is shown only as an example. In some embodiments, the CV descriptor may be saved without being configured to uniquely identify the target UI element.

[0177] Developers can use the properties tab 760 to view CV hits with descriptor properties. Figure 7P . This provides various variations of the CV descriptor. However, unlike the selector, the underlying characteristics of the selector for the CV descriptor may be complex and related to the application of the AI / ML model to the image. Therefore, although a CV selector is provided, in some embodiments it may not be easily accessible or modifiable by the developer. Instead, the developer can re-indicate the target UI element in order to edit the CV descriptor / selector. This also means that in some embodiments, even non-technical users can indicate the target / anchor point. The set of UI elements on the screen and the UI elements that the developer wishes to click on can be saved as part of the CV-based identification activity 730.

[0178] Regarding the unified target based identification activity 740, when the developer selects "indicate application" 741 (see Figure 7B ), the user clicks 10 Calculator 750 UI, which enables UiPath Studio TMA screenshot 743 of the UI is captured and displayed in the unified target based identification activity 740. Figure 7Q . Also shown is the application path 744 of the application associated with the screenshot 743, and the developer can add application parameters 745 if desired. The do activity 742 is nested within the identify activity 740 based on the unified goal. When the developer clicks the plus icon, the search interface 708 appears. See Figure 7R The developer can then search for the desired click functionality.

[0179] The selected click option causes a nested click activity 746 to appear within the do activity 742. Figure 7S The click activity 746 includes an on-screen indication button 747 that enables the developer to indicate the target to be selected and clicked on the screen. The developer can also specify the type of click and the mouse button to click. In this case, a left click is specified.

[0180] Clicking on the on-screen indication button 747 causes a unified target selection option window 770 to appear. Figure 7T . Unified Targeting adds targets and / or anchors based on images and uses a selector-based framework behind the scenes. Hovering the mouse over the 9 button causes the target UI element outline 772 to appear. When the user clicks the 9 button, it is selected as the target and this state is indicated on the target UI element outline 772. See Figure 7U Various option icons also appear, including an anchor designation icon 774. In this embodiment, UiPath Studio TM Tried to automatically find a suitable anchor point but was unable to do so.

[0181] The developer specifies an anchor 776 (in this case, the 8 button), and the combination of these elements allows the unified target logic to uniquely identify the target 9 button using the combination of the target and the anchor 8 button. See Figure 7V This causes the unified target selection options window 770 to confirm that the target and anchor are OK, and the designer can validate the target and anchor selectors, confirm the selection, or cancel. Here, the developer selects to validate the target and anchor selectors, and the unified target selection options window 770 confirms that the validation was successful with a validation accuracy score of 100%, as shown in the validation score notification 778. See Figure 7W Screen shot 748 of the target and anchor appearing after indicating that the target and anchor selectors are normal. Figure 7X In certain embodiments, CV techniques may be used within a unified objective framework.

[0182] In some embodiments, image matching can be used to find UI elements. Figure 7YA click on image activity 780 has been added. When the developer selects "indicate image on screen" 782, the developer can define an image for image search. Figure 7Z , the developer drags a box 784 around the 9 button, and a screenshot 786 of the 9 button appears in the click image activity 780. The properties of the click image activity appear in the properties tab 760. See Figure 7a . It also generates image selectors, such as Figure 7b The selector editor 726 is shown in FIG. 726. The selector captures the application from which the developer obtains the snapshot.

[0183] In some embodiments, a combination of selectors and OCR may be used. Figure 7c A click OCR text activity 790 is shown, which includes a field 794 where a developer can manually enter text to search and an OCR engine 796. In this embodiment, when a user clicks "Indicate on Screen" 792, the user selects a UI element in a manner similar to the selector-based method (see Figure 7C ). After selecting a UI element, a screenshot 798 of the selected UI element appears in the Click to OCR Text activity 790. Figure 7d However, unlike the selector-based approach, the selector and OCR approach also uses OCR to identify UI elements. For example, although many calculator buttons have the same shape, there is only one button with each number.

[0184] Figure 8 8 is a flow chart illustrating a process 800 for generating an activity based on a selector according to an embodiment of the present invention. The process begins by generating an activity based on a selector at 810. The activity may click a button, enter text, open a menu, select an option, etc. A command is received on the screen at 820, and an application is detected at 830. In some embodiments, this may include automatically detecting the application, detecting the version of the application, and / or detecting a screen within the application. If the application is not automatically detected, detecting the application may include requesting the user to provide the application name and version. For example, if multiple applications are detected and / or the user can manually enter this information, a drop-down list may be provided to the user to provide the application name and application version.

[0185] A range of activities may be provided, which allow developers to create the desired RPA process. Some activities may allow the user to indicate a screen / application using. When a user wishes to perform an automation, the user may indicate the screen, UI element on the screen, etc. that the activity should interact with. Some activities may open applications. This may include opening a desktop application, navigating using a URL, etc.

[0186] At 840, UI elements on the application screen are detected. In some embodiments, the detected elements can be highlighted for the user to generate a UI model. The detected UI elements can be organized into categories or trees as a model of the screen. However, in some embodiments, a non-taxonomic approach such as an ontology can be used. An ontology can define certain structures and logical relationships between screens. For example, if a "Submit" button is clicked on one screen, this may cause at least some different UI elements to appear on another screen. In addition, some UI elements and / or portions of a screen may be common to multiple screens. For example, in Microsoft In the example, the left column navigation sub-screen traditionally does not change and can be reused between screens. Knowing which sub-screens and / or UI elements do not change can reduce processing time.

[0187] At 850, an indication of an active UI element is received from the user. In some embodiments, this may include clicking the desired UI element. At 860, an out-of-the-box selector is generated for the indicated UI element using a driver (e.g., driver 340) based on the best match for the UI element, and the activity is configured accordingly.

[0188] Fig. 9 900 is a flow chart illustrating a process for generating an activity based on a CV descriptor according to an embodiment of the present invention. The process begins by generating an activity based on a CV descriptor at 910. At 920, a command is indicated on the screen and a user's selection of a screen to be automated is received. For example, in some embodiments, the user may select an entire application interface. At 930, the application is detected, and at 940, the UI elements on the application screen are detected using the CV.

[0189] At 950, a command is received to specify a UI element for interaction, and an interface with selectable UI elements detected by CV is displayed. Then, at 960, a selection of a target UI element is received from the user. If the target can be uniquely identified at 970, a CV descriptor is generated at 980 and the activity is configured accordingly. However, if the target cannot be uniquely identified at 970, an anchor point designation is required at 990 and is received from the user until the target can be uniquely identified. Then, the process proceeds to step 980 to generate a CV descriptor and configure the activity.

[0190] Fig.10 1 is a flow chart illustrating a process 1000 for generating unified goal-based activities according to an embodiment of the present invention. At 1020, an application command and a user's selection of a screen to be automated are received. For example, in some embodiments, the user may select the entire application interface. For example, at 1030, the application is detected, and at 1040, a driver is used to detect UI elements on the application screen.

[0191] At 1050, a command to specify a UI element for interaction is received, and an interface with selectable UI elements detected by the driver is displayed. Then at 1060, a selection of a target UI element is received from the user. If the target can be uniquely identified using only the selector of the element at 1070, a unified target descriptor (e.g., a selector) is generated at 1080 and the activity is configured accordingly. However, if the target cannot be uniquely identified at 1070, an anchor point designation is required and received from the user until the target can be uniquely identified at 1090. In some embodiments, this may include verifying the target and (multiple) anchor points by determining a verification score. The process then proceeds to step 1080 to generate a unified target descriptor (e.g., a selector for the target and (multiple) anchor points and the position / geometric relationship between them) and configure the activity.

[0192] To determine whether a UI element is uniquely identified, a strength calculation (e.g., a validation score) may be used to determine how well the target UI element is believed to match based on the unified target UI descriptor. If the strength calculation is below a threshold (e.g., below 97%, below 95%, etc.), it is possible that the UI element may not be determined as accurately as desired at runtime. The acceptable accuracy may vary depending on the task(s) completed by the RPA workflow. UI element targets that are below the threshold may require anchor points to be uniquely identified. These anchor points may be automatically determined and displayed to the user, or the user may hover over the UI element to select an anchor point. In some embodiments, the unified target descriptor strength resulting from adding a given anchor point may be displayed to the user. If the threshold is still not reached after adding the anchor point, the user may need to continue adding additional anchor points until the target UI element reaches the threshold.

[0193] Anchors are other UI elements on the screen that can be used to help uniquely identify a target UI element on the screen. For example, if multiple text fields are included on a screen, searching for one text field alone is not sufficient to uniquely identify a given text field. Therefore, some embodiments look for additional information to uniquely identify a given UI element. Using the text field example, a text field for entering a name may appear to the right of a label "name." This name label can be set as an "anchor" to help uniquely identify the text field as a selector for "target."

[0194] In some embodiments, various positions and / or geometric associations between a target and an anchor point can potentially be used within a tolerance to uniquely identify a target. For example, the center of the bounding box of the anchor point and the target can be used to define a line segment. This line segment can then be required to have a certain length within a tolerance and / or a slope within a tolerance to uniquely identify a target using a target / anchor point pair.

[0195] In some cases, a single anchor point may not be sufficient to uniquely identify a target element on the screen. For example, consider the case where two text fields for entering a name appear to the right of a corresponding label "Name" at different locations on the screen. In such cases, one or more additional anchor points may be useful for uniquely identifying a given target. Geometric properties between the anchor point and the target (e.g., line segment length, angle, and / or relative position with tolerance) can be used to uniquely identify the target. The user may be asked to continue adding anchor points until the match strength of the target exceeds a threshold.

[0196] In some embodiments, instead of marking and displaying UI elements and anchor candidates to the user, one or more anchors may be automatically assigned until the strength calculation exceeds a threshold. For example, if the threshold is 97% and a given UI element has a 90% match strength, the system may continue to add anchors until the match strength reaches or exceeds the threshold. In some embodiments, this may be accomplished by determining the anchor that increases the match strength the most for the selector, adding that anchor if it is still below the threshold, determining the increase in match strength for other elements after adding the anchor, adding the anchor that increases the match strength the most, and repeating until the threshold is met.

[0197] Fig.11 1 is a flow chart illustrating a process 1100 for generating an image matching activity according to an embodiment of the present invention. The process begins by generating an image matching activity at 1110. The activity may click a button, enter text, open a menu, select an option, etc. based on image matching technology. At 1120, a command is received on the screen, and an application is detected at 1130.

[0198] At 1140, a selection of an application area to perform image matching is received. This may include the user drawing a box around the UI element of interest, using a lasso tool, drawing a free-form selection, or any other selection mechanism without departing from the scope of the present invention. Then, at 1150, an image matching selector is generated for the selected image, and the activity is configured accordingly.

[0199] Fig.12 1 is a flow chart illustrating a process 1200 for generating an activity based on a selector and OCR according to an embodiment of the present invention. The process begins by generating an activity based on a selector and OCR at 1210. The activity may click a button, enter text, open a menu, select an option, etc. At 1220, a command is received to indicate on the screen, and an application is detected at 1230. At 1240, UI elements on the application screen are also detected.

[0200] At 1250, an indication of an active UI element is received from a user. In some embodiments, this may include clicking the desired UI element. At 1260, an out-of-the-box selector is generated for the indicated UI element using a driver based on the best match for the UI element, and the activity is configured accordingly. At 1270, a selection of an OCR engine is received, and the activity is configured accordingly. When an activity based on a selector and OCR is executed as part of an RPA process, it checks whether a selector for the UI element is found and verifies whether the text of the UI element matches the text specified in the activity using OCR, rather than searching for the selector separately.

[0201] Fig.13 1 is a flow chart illustrating a process 1300 for generating an RPA workflow using multiple identification technologies according to an embodiment of the present invention. The process begins by generating UI element identification and interaction activities at 1310. These activities can be one or more selector-based identification activities, CV-based identification activities, unified target identification activities, image matching identification activities, selector-based and OCR-based identification activities, any other suitable identification activities, any combination thereof, etc.

[0202] Then, at 1320, the UI element identification and the interactive activities are configured as a logical relationship. The logical relationship can be sequential, conditional (e.g., if the first activity does not successfully uniquely detect the UI element, then use the second, then the third, and so on, until the UI element is uniquely identified or all activities have been attempted), parallel, etc. At 1330, an RPA robot that implements the workflow including the UI element identification and the interactive activities is generated, and at 1340, the RPA robot is executed to detect and interact with the UI element.

[0203] Fig.14 14 is a flow chart illustrating a process 1400 for creating a UI object library and configuring an item to work with a new version of an application according to an embodiment of the present invention. If one or more UI object libraries have been previously created at 1410, the process may begin by loading one or more UI object libraries from a UI object repository. Then, at 1420, UI descriptors for interacting with corresponding UI elements of an activity are created and / or added from the one or more UI object libraries to interact with the new version of the application with a changed UI.

[0204] Once the appropriate UI descriptors are updated, the UI object library including the UI descriptors is published in the UI object repository at 1430. The UI object library may also include applications, screens, and at least some UI elements from the applications with which the activity interacts. However, in some embodiments, other UI elements and / or UI descriptors that are not specifically used in the RPA workflow may be included. The UI object library is then published or republished in the UI object repository at 1440 for use in other RPA workflows and / or by other developers.

[0205] If UI objects are missing from the UI object library, or if the active UI descriptor no longer applies to the new version of the application, the user can add these objects. The created UI object library and possible other UI object libraries in the UI object repository can be displayed to the user in the designer application. In some embodiments, the publication of object libraries can be performed as part of a global object library database (OLDB) or a local OLDB. For example, the global OLDB UI object repository can be a globally shared database (Center of Excellence (CoE) level). The local OLDB UI object repository can be a locally shared database (local machine level). In some embodiments, the UI object library can be pushed and published to a service, such as UiPath TM Service or Go! Service. This allows sharing the UI object repository collection with the world if desired. The storage format of the UI object repository (e.g., OLDB) should be decided on an architectural / technology level.

[0206] According to an embodiment of the present invention, Figures 8 to 14 The process steps performed in the embodiment may be performed by a computer program, which is coded for a processor (or processors) to execute Figure 8-14 The computer program may be embodied on a non-transitory computer readable medium. The computer readable medium may be, but is not limited to, a hard drive, a flash memory device, a RAM, a magnetic tape, and / or any other such medium or combination of media for storing data. The computer program may include instructions for controlling the processor(s) of the computing system (e.g., Figure 5 The processor(s) 510 of the computing system 500 are used to implement Figures 8 to 14 The coded instructions for all or part of the process steps described in the present invention may also be stored on a computer-readable medium.

[0207] The computer program may be implemented in hardware, software or a hybrid implementation. The computer program may consist of modules that are operable to communicate with each other and are designed to pass information or instructions to a display. The computer program may be configured to run on a general purpose computer, an ASIC or any other suitable device.

[0208] It is readily understood that the components of the various embodiments of the present invention, as generally described and illustrated in the drawings herein, may be arranged and designed in a variety of different configurations. Therefore, the detailed description of the embodiments of the present invention, as shown in the drawings, is not intended to limit the scope of the claimed invention, but is merely a representation of selected embodiments of the present invention.

[0209] In one or more embodiments, the features, structures or characteristics of the present invention described throughout this specification may be combined in any suitable manner. For example, references to "certain embodiments," "some embodiments," or similar language throughout the specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in at least one embodiment of the present invention. Therefore, phrases "in certain embodiments," "in some embodiments," "in other embodiments," or similar language appearing throughout the specification do not necessarily all refer to the same set of embodiments, and in one or more embodiments, the described features, structures, or characteristics may be combined in any suitable manner.

[0210] It should be noted that references to features, advantages, or similar language throughout this specification do not imply that all features and advantages that can be realized with the present invention should be in any single embodiment of the present invention or in any single embodiment of the present invention. Rather, language referring to features and advantages is understood to mean that a particular feature, advantage, or characteristic described in conjunction with an embodiment is included in at least one embodiment of the present invention. Therefore, discussion of features and advantages and similar language throughout this specification may, but does not necessarily, refer to the same embodiment.

[0211] In addition, in one or more embodiments, the features, advantages and characteristics of the present invention may be combined in any suitable manner. Those skilled in the relevant art will recognize that the present invention may be implemented without one or more of the specific features or advantages of a particular embodiment. In other cases, additional features and advantages that may not be present in all embodiments of the present invention may be recognized in certain embodiments.

[0212] Those skilled in the art will readily appreciate that the present invention as discussed above may be implemented by steps in different orders and / or hardware elements different from the disclosed configurations. Therefore, although the present invention has been described based on these preferred embodiments, certain modifications, variations and alternative configurations will be apparent to those skilled in the art while remaining within the spirit and scope of the present invention. Therefore, in order to determine the dimensions and limits of the present invention, reference should be made to the appended claims.

Claims

1. A computer program product for creating a user interface UI object library for robotic process automation (RPA), the computer program product comprising a computer program configured to cause at least one processor to: Detecting applications associated with screens of the UI; Detecting UI elements on the screen; Receive selection of target UI element; Generate a UI descriptor for the selected UI element, wherein the generated UI descriptor includes an instruction set for finding the target UI element on the screen; adding the generated UI descriptor for the selected UI element to a corresponding RPA workflow activity; as well as creating and publishing a UI object library including the UI descriptor in a UI object repository, The UI object library includes UI descriptors corresponding to one or more screens from the detected version of the application. 2 . The computer program product of claim 1 , wherein the UI descriptor is or includes a selector generated based on a best match using a driver.

3. The computer program product according to claim 1, wherein the RPA workflow activity is or includes a selector-based identification activity, a computer vision (CV)-based identification activity, a unified target identification activity, an image matching identification activity, or an identification activity based on a selector and optical character recognition (OCR).

4. The computer program product of claim 1, wherein The detecting of the application associated with the screen of the UI comprises automatically detecting the application and a version of the application, For the RPA workflow activity, the detected application and the version of the application are added, and The generated UI descriptor is a UI descriptor for the detected version of the application.

5. The computer program product of claim 1 , wherein the computer program is further configured to cause the at least one processor to: extracting a UI descriptor from an activity in an RPA workflow including the RPA workflow activity; and The extracted UI descriptors are added to a structural schema that groups the UI descriptors by UI application, version, screen, and UI element.

6. The computer program product of claim 5, wherein the structured patterns are arranged in an ontology, the ontology comprising relationships between UI elements in different screens.

7. The computer program product of claim 1, wherein the detected application is a web application and the UI descriptor includes a Hypertext Markup Language (HTML) path to a corresponding UI element for a corresponding selector.

8. The computer program product of claim 1, wherein the computer program is or comprises an RPA developer application.

9. The computer program product of claim 1 , wherein the computer program is further configured to cause the at least one processor to: Retrieving one or more other object repositories from the UI object repository; and The created UI object library and the one or more other object libraries are displayed.

10. The computer program product of claim 1, wherein the computer program is or includes a UI object browser that displays UI descriptors in one or more UI object libraries of a UI object repository, the one or more UI object libraries being organized by application and version.

11. The computer program product according to claim 1, wherein the RPA workflow activity is or includes a computer vision (CV) based activity, and the computer program is further configured to cause the at least one processor to: performing CV on the screen of the UI to identify UI elements on the screen; displaying the screen of the UI having the selectable component identified by the CV; Receiving a selection of one or more anchor UI elements to help uniquely identify the target UI element; as well as The one or more anchor points are added to the RPA workflow activity to assist in identifying the target UI element.

12. The computer program product of claim 1 , wherein the RPA workflow activity is or includes a unified target-based activity, and when the target UI element cannot be uniquely identified, the computer program is further configured to cause the at least one processor to: receiving a selection of one or more anchor UI elements to help uniquely identify the target UI element; verifying that the selected one or more anchor UI elements uniquely identify the target UI element having at least a minimum verification score; and When the selected one or more anchor UI elements uniquely identify the target UI element having at least the minimum validation score, The one or more anchor points are added to the RPA workflow activity to assist in identifying the target UI element.

13. The computer program product of claim 1, wherein the RPA workflow activity is or includes an image matching based activity that uses image matching technology to search for an image on the screen specified by a user.

14. A computer program product according to claim 1, wherein the RPA workflow activity is or includes a selector and optical character recognition (OCR) based activity, wherein the selector and OCR based activity uses a combination of a selector and OCR of text associated with the target UI element to identify the target UI element, and the selector is generated based on the best match using a driver.

15. The computer program product of claim 1 , wherein the computer program is further configured to cause the at least one processor to: A plurality of RPA workflow activities including the RPA workflow activity are configured into a logical relationship.

16. The computer program product of claim 1, wherein the computer program is further configured to cause the at least one processor to: An RPA robot is generated to implement the RPA workflow including the RPA workflow activities.

17. The computer program product of claim 1, wherein the generated UI descriptor is a UI descriptor for a detected new version of the application.

18. A computer program product, comprising a computer program configured to cause at least one processor to: Receiving a selection of a target user interface (Ul) element on a screen of the application; Generate a UI descriptor for the selected UI element, wherein the generated UI descriptor includes an instruction set for finding the target UI element on the screen; as well as adding the generated UI descriptor for the selected UI element to a corresponding Robotic Process Automation (RPA) workflow activity, wherein The RPA workflow activity is or includes a computer vision (CV)-based identification activity or a unified target identification activity.

19. The computer program product of claim 18, wherein the computer program is further configured to cause the at least one processor to: extracting a UI descriptor from an activity in an RPA workflow including the RPA workflow activity; and The extracted UI descriptors are added to a structural schema that groups the UI descriptors by UI application, version, screen, and UI element.

20. The computer program product of claim 19, wherein the structured patterns are arranged in an ontology, the ontology comprising relationships between UI elements in different screens.

21. The computer program product of claim 18, wherein the computer program is further configured to cause the at least one processor to: creating and publishing a UI object library including the UI descriptor in a UI object repository, The UI object library includes UI descriptors corresponding to one or more screens from the detected version of the application.

22. The computer program product of claim 18, wherein the computer program is or comprises a UI object browser that displays UI descriptors in one or more UI object libraries of a UI object repository, the one or more UI object libraries being organized by application and version.

23. The computer program product of claim 18, wherein the RPA workflow activity is or includes the CV-based identification activity, and the computer program is further configured to cause the at least one processor to: performing CV on the screen of the UI to identify UI elements on the screen; displaying the screen of the UI having the selectable component identified by the CV; receiving a selection of one or more anchor UI elements to help uniquely identify the target UI element; as well as The one or more anchor points are added to the RPA workflow activity to assist in identifying the target UI element.

24. The computer program product of claim 18, wherein the RPA workflow activity is or includes a unified target-based activity, and when the target UI element cannot be uniquely identified, the computer program is further configured to cause the at least one processor to: receiving a selection of one or more anchor UI elements to help uniquely identify the target UI element; verifying that the selected one or more anchor UI elements uniquely identify the target UI element having at least a minimum verification score; and When the selected one or more anchor UI elements uniquely identify the target UI element having at least a minimum validation score, The one or more anchor points are added to the RPA workflow activity to assist in identifying the target UI element.

25. A method of Robotic Process Automation (RPA), applied to RPA developer applications, the method comprising: Detecting an application associated with a screen of a user interface UI; Detecting UI elements on the screen; Receive selection of target UI element; Generate a UI descriptor for the selected UI element, wherein the generated UI descriptor includes an instruction set for finding the target UI element on the screen; as well as The generated UI descriptor for the selected UI element is added to a corresponding RPA workflow activity, wherein The RPA workflow activity is or includes a selector-based identification activity, a computer vision CV-based identification activity, a unified target identification activity, an image matching identification activity, or a selector-based and optical character recognition OCR-based identification activity, and The RPA developer application includes a UI object browser that displays UI descriptors in one or more UI object libraries of a UI object repository, the one or more UI object libraries being organized by application and version.

26. The method according to claim 25, further comprising: Creating and publishing a UI object library including the UI descriptor in the UI object repository, wherein The generated UI descriptor is a UI descriptor for the detected new version of the application.

27. The method of claim 25, further comprising: extracting a UI descriptor from an activity in an RPA workflow including the RPA workflow activity; as well as adding the extracted UI descriptors to a structured schema that groups the UI descriptors by UI application, version, screen, and UI element, wherein The structural patterns are arranged in an ontology that includes relationships between UI elements in different screens.

Images