Computer-implemented method, system, computer program, and computer implementation method (user interface automation using natural language)

Abstract

To provide a method and system for automating a user interface using a natural language instruction and a large language model.SOLUTION: A method includes the steps of: receiving a demonstration of automation to be performed on an application; detecting one or more objects and one or more corresponding labels associated with the demonstration on the application; transforming the demonstration into one or more natural language instructions: and semantically selecting an object during a runtime action of a computing device based on a large language model (LLM). The semantic selection is reflective of an intention captured in the one or more natural language instructions despite any change in a corresponding label of the object.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates generally to user interface automation, and more particularly to methods and systems for automating user interfaces using natural language instructions and large-scale language models. [Background technology]

[0002] User Interface (UI) automation is an aspect of modern software development and testing processes. UI automation may involve creating and running scripts or code that simulate user interactions with a software application's graphical user interface (GUI). The automation process may aim to improve the efficiency, accuracy, and reliability of a variety of tasks, such as software testing, performing repetitive tasks, and workflow automation.

[0003] The complexity and diversity of modern software applications can lead to the use of UI automation tools. With the proliferation of web, desktop, mobile applications, and other digital platforms, manually testing and performing repetitive tasks can become increasingly time-consuming and error-prone, especially when the UI changes.

[0004] UI automation may typically involve interacting with UI elements such as buttons, input fields, drop-down menus, and dialog boxes. This interaction may involve actions that are typically performed using a mouse and keyboard. At the heart of most UI automation tools is programmatic interaction with UI elements. This interaction is typically facilitated through automation frameworks and tools that enable a wide variety of actions to be performed.

[0005] UI automation tools are typically used by operators with a deep understanding of UI technologies. Experienced programmers may define stable selector anchors for UI elements in such automation tools, which may limit scalability and generalization across various applications in some cases. Some low-code and no-code (LCNC) platforms may offer visual development environments, but these may also have limited flexibility. In essence, UI automation is a critical part of the software development process, but the availability of robust automation tools capable of withstanding numerous evolving software scenarios remains a challenge. Summary of the Invention

[0006] According to one embodiment of the present disclosure, a method includes receiving a demonstration of automation to be performed on an application. One or more objects and one or more corresponding labels associated with the demonstration may be detected. The demonstration may be converted into natural language instructions. During runtime action, objects are semantically selected based on a large-scale language model (LLM), and the semantic selection reflects the intent captured in the natural language instructions, regardless of any changes to the objects' corresponding labels. According to one embodiment, a system includes a processor that receives a demonstration of automation to be performed on an application. The processor may detect one or more objects and one or more corresponding labels associated with the demonstration on the application. The processor then converts the demonstration into one or more natural language instructions and, during runtime action, semantically selects the objects based on the large-scale language model to reflect the intent of the one or more natural language instructions, regardless of any changes to the objects' corresponding labels.

[0007] According to one embodiment of the present disclosure, a non-transitory computer-readable storage medium stores computer-executable instructions that, when executed by a computer system, can cause the computer system to receive a demonstration of automation to be performed on an application and detect, on the application, one or more objects and one or more corresponding labels associated with the demonstration. The computer is further caused to convert the demonstration into one or more natural language instructions and, at runtime action, semantically select objects based on a large-scale language model. The semantic selection reflects the intent captured in the one or more natural language instructions, regardless of any changes to the corresponding labels of the objects.

[0008] According to one embodiment of the present disclosure, a method is provided. In the method, a demonstration of an automation to be performed on an application is received. The demonstration is converted into one or more natural language instructions. Then, during runtime action, objects are semantically selected based on a large-scale language model. The semantic selection reflects the intent captured in the one or more natural language instructions, regardless of any changes to the objects' corresponding labels. Furthermore, the automation can be modified based on new natural language user input. According to one embodiment of the present disclosure, a system is provided. The system includes a knowledge representation dataset including one or more natural language instructions for an automation, and a processor. The processor receives the knowledge representation dataset; and, during runtime action of the automation, semantically selects the objects based on a large-scale language model (LLM) to reflect the intent of the one or more natural language instructions, regardless of any changes to the objects' corresponding labels.

[0009] In one embodiment, the system receives a demonstration of automation to be performed on an application, and detects on the application one or more objects and one or more corresponding labels associated with the demonstration; and converts the demonstration into one or more natural language instructions representing actions to be performed on the one or more objects. [Brief explanation of the drawings]

[0010] The drawings are of exemplary embodiments. The drawings do not depict every embodiment. Other embodiments may be used in addition or instead. Details that may be obvious or unnecessary may be omitted to save space or for a more efficient illustration. Some embodiments may be practiced using additional components or steps and / or without using all of the components or steps shown. When the same numeral appears in different drawings, it refers to the same or similar components or steps.

[0011] [Figure 1] 1 is a block diagram of a network of data processing systems in accordance with an illustrative embodiment;

[0012] [Figure 2] FIG. 1 is a block diagram of a computing environment in accordance with an illustrative embodiment.

[0013] [Figure 3] FIG. 1 is an illustration of a block diagram of an application architecture in accordance with an illustrative embodiment;

[0014] [Figure 4A] 1 is a block diagram of a page of an application in accordance with an illustrative embodiment;

[0015] [Figure 4B] 1 is a block diagram illustrating a visualization of page understanding of an application in accordance with an illustrative embodiment;

[0016] [Figure 5] FIG. 1 is a block diagram of a system architecture in accordance with an exemplary embodiment;

[0017] [Figure 6] FIG. 1 is an illustration of a block diagram for visualizing natural language instructions in accordance with an illustrative embodiment;

[0018] [Figure 7] FIG. 1 illustrates a routine according to one embodiment.

[0019] [Figure 8] FIG. 1 illustrates a method according to an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] According to one aspect of the present disclosure, a method is provided in which a demonstration of automation to be performed on an application is received. One or more objects and one or more corresponding labels associated with the demonstration may be detected. The demonstration may be converted into one or more natural language instructions using a computing device. During runtime action, objects are semantically selected based on a large-scale language model (LLM), and the semantic selection reflects the intent captured in the natural language instructions, regardless of any changes to the objects' corresponding labels. This enables an operator to both teach and execute UI automation without any coding or programming skills, thereby bringing a unique combination of accessibility, efficiency, scalability, and customization to the field of automation targeted to the specific needs of non-technical business users. Thus, the method may incorporate a semantic programming model that focuses on the meaning or intent behind the code rather than its syntax. This abstract approach makes automation more intuitive for users who may not be familiar with technical details. While low-code and no-code platforms offer customization through visual programming, the method allows end users to both teach and execute UI automation even when UI elements change, thereby providing greater flexibility without sacrificing ease of use.

[0021] In an embodiment, automation is performed through PbD (Programming by Demonstration). The use of PbD can simplify the programming process by allowing an operator to effectively demonstrate tasks in a non-technical manner.

[0022] In embodiments, one or more objects and one or more corresponding labels may be detected by performing page understanding to generate a dataset representing the original state of the available objects on a page of an application. Page understanding enables understanding the overall state of a page, including the topology of UI elements, thereby analyzing the current interactable elements on the page.

[0023] In an embodiment, one or more natural language instructions are stored in a knowledge representation dataset that includes an automation workflow that can be re-executed during runtime to make automation faster and more accessible.

[0024] In an embodiment, semantic selection of an object is achieved by detecting a change in a corresponding label associated with an identified natural language instruction of one or more natural language instructions. The semantic selection further includes obtaining a plurality of possible objects corresponding to the change and providing the identified natural language instruction and the plurality of possible objects to an LLM. An indication of the object that most likely corresponds to the identified natural language instruction is received from the LLM, and that object is selected. Due to the ability of the LLM to decipher which object corresponds to the change, this advantageously accounts for the change in the object's label.

[0025] In embodiments, page understanding is performed to generate a dataset representing the current state of objects available on a page of an application, and changes are detected in response to the execution of page understanding. Advantageously, by detecting the current state of objects on the page, corresponding label changes can be determined based on a comparison with the original label. This can trigger the use of the LLM to detect and select objects corresponding to the changes, thereby providing the effect of a robust automation tool that can tolerate object changes and thus can be used on a wide variety of applications. By employing the concept of page understanding in automation workflows and allowing users to demonstrate tasks that are then performed by the LLM based at least in part on page understanding, resiliency in UI automation can be achieved.

[0026] In an embodiment, object selection is performed during runtime, which may allow automation to be re-executed on demand.

[0027] In an embodiment, the demonstration is received during the teaching action, so that the automated workflow can be directed by the operator in a desired manner.

[0028] In an embodiment, the application is a plug-in module that can be plugged into a web, desktop, or mobile application. The modular nature of the plug-in module may enable automation workflows to be implemented on various types of applications. According to one aspect of the present disclosure, a system is provided that includes a processor that receives a demonstration of automation to be implemented on an application. The processor may detect one or more objects and one or more corresponding labels associated with the demonstration on the application. The processor then converts the demonstration into one or more natural language instructions and semantically selects objects based on a large-scale language model to reflect the intent of the one or more natural language instructions during runtime action, regardless of any changes to the objects' corresponding labels. This may enable the use of a dedicated system to teach and execute UI automation without any coding or programming skills. Because the system may now be able to create more robust interaction logic that captures the operator's intent, traditional programmatic interactions manually created on the system for interacting with UI elements are therefore unnecessary.

[0029] In an embodiment, one or more objects are user interface elements, which may be collected along with their attributes and metadata and assigned a unique name for easy identification.

[0030] In an embodiment, automation is performed by PbD, which may make automation accessible to non-technical operators so that the automation systems described herein may be utilized.

[0031] In an embodiment, one or more natural language instructions of the system are stored in a knowledge representation dataset from which the necessary stored natural language instructions for a given task can be quickly retrieved.

[0032] In an embodiment, the processor may detect a change in a corresponding label associated with an identified natural language instruction of the one or more natural language instructions; obtain a number of possible objects corresponding to this change; provide the identified natural language instruction and the number of possible objects to the LLM; receive an indication from the LLM of the object that most likely corresponds to the identified natural language instruction; and select that object. Thus, the processor may be an orchestrator that semantically selects desired objects to implement automation.

[0033] In an embodiment, the demonstration is received during a teaching action. The demonstration may be received via a user-friendly interface to teach UI automation, thereby minimizing the need for extensive technical training.

[0034] In an embodiment of the system, the application is a plug-in module that can be plugged into a web, desktop, or mobile application. A plug-in module may be a modular software component (e.g., a plug-in, extension, add-on, or other module that plugs in or is otherwise configured to be plugged in) for another application and can be controlled to perform actions on that application.

[0035] According to one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a computer system, can cause the computer system to receive a demonstration of automation to be performed on an application; detect one or more objects and one or more corresponding labels associated with the demonstration on the application; and convert the demonstration into one or more natural language instructions. The non-transitory computer-readable storage medium further stores computer-executable instructions that, when executed by the computer system, can cause the computer system to semantically select objects based on a large-scale language model (LLM) during runtime action. The semantic selection reflects the intent captured in the one or more natural language instructions, regardless of any changes to the objects' corresponding labels. Storing computer-executable instructions can package intuitive automation modules and make them more readily available to users who may not be familiar with the technical know-how of automation workflows.

[0036] In an embodiment, automation is performed by PbD. The use of PbD can simplify the programming process by allowing operators to effectively demonstrate tasks in a non-technical manner.

[0037] In an embodiment, the one or more natural language instructions are stored in a knowledge representation dataset, which may include a specification of the program to be executed, providing, for example, the flow, application name, external parameters, and instructions to be performed as part of the automation.

[0038] In embodiments, the application is a plug-in module that can be plugged into a web, desktop, or mobile application, and instructional and runtime actions can be performed under the control of the plug-in module.

[0039] In an embodiment, to semantically select an object, the computer-executable instructions are configured to: detect a change in a corresponding label associated with an identified natural language instruction of the one or more natural language instructions; obtain a plurality of possible objects corresponding to the change; and provide the identified natural language instruction and the plurality of possible objects to an LLM. The computer-executable instructions may receive an indication from the LLM of the object that most likely corresponds to the identified natural language instruction; and select that object. The ability of the LLM to decipher which object corresponds to which change allows the change in the object's label to be explained through the use of the computer-executable instructions.

[0040] According to one aspect of the present disclosure, a method is provided in which a demonstration of an automation to be performed on an application is received. The demonstration is converted into one or more natural language instructions. Then, during runtime action, objects are semantically selected based on a large-scale language model. The semantic selection reflects the intent captured in the one or more natural language instructions, regardless of any changes to the objects' corresponding labels. The automation is modified based on new natural language user input. The ability to modify the demonstration based on new natural language input enables flexibility in the automation as well as the ability for non-technical operators to update the automation flow in an efficient and timely manner.

[0041] According to one aspect of the present disclosure, a system is provided. The system includes: a knowledge representation dataset including one or more natural language instructions for automation; and a processor. The processor receives the knowledge representation dataset; and during runtime action of the automation, semantically selects objects based on a large-scale language model (LLM) to reflect the intent of the one or more natural language instructions, regardless of any changes to the objects' corresponding labels. This allows the automation flow to be transferred to and executed on multiple systems, even if the original automation was performed on a different system.

[0042] In the following detailed description, numerous specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. However, it should be apparent that the present teachings may be practiced without such details. In other instances, well-known methods, procedures, and components have been described in relatively general terms, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

[0043] The present disclosure generally relates to methods and systems for automating user interfaces using natural language instructions and large-scale language models.

[0044] The illustrative embodiments provide a method for automating a user interface using natural language instructions and large-scale language models.

[0045] In one embodiment, some operations are described as occurring in a certain component or location. Such locality of operations is not intended to be a limitation to the exemplary embodiment. Any operation described herein as occurring in or performed by a particular component may be implemented in a manner such that the operation occurs or is performed in another component, e.g., a local or remote engine, depending on the component's specific functionality. In one aspect, the methods described herein are implemented to run on a specially configured computing device or data processing system, providing a significant increase in the capabilities of that computing device or data processing system by enabling the use of large-scale language models and natural language input. Thus, embodiments have the ability to improve the technical field of UI automation by generalizing the process of UI automation to target the specific needs of non-technical users.

[0046] Importantly, while the operational / functional descriptions described herein may be understandable by the human intellect, they are not abstract ideas of operations / functions separated from the computational implementation of those operations / functions. Rather, the operations / functions represent specifications for an appropriately configured computing device. As discussed in detail below, operational / functional language should be read in its proper technical context, i.e., as a concrete specification of a physical implementation.

[0047] It should be understood that aspects of the teachings herein are beyond the capabilities of the human mind. It should also be understood that various embodiments of the present disclosure described herein may involve information that is impossible for an entity, such as a human user, to manually obtain. For example, the type, amount, and / or variety of information involved in performing the processes discussed herein may be more complex than information that a human user can reasonably process manually.

[0048] Thus, one or more of the methodologies discussed herein may eliminate the need for advanced coding skills, efficiency, and speed for non-technical users, which may have the technical effect of reducing computing resources used by one or more devices in a system.

[0049] The exemplary embodiments are described in terms of several types of machines. The exemplary embodiments are also described in terms of other scenes, objects, measurements, devices, data processing systems, environments, components, and applications, by way of example only. Any particular manifestation of these and other similar artifacts is not intended as a limitation on the present disclosure. Any suitable manifestation of these and other similar artifacts may be selected within the scope of the exemplary embodiments.

[0050] Furthermore, exemplary embodiments may be implemented with any type of data, data source, or access to a data source via a data network. Any type of data storage device may provide data to an embodiment of the present disclosure within the scope of the present disclosure, either locally at a data processing system or via a data network. Where an embodiment is described using a mobile device, any type of data storage device suitable for use with a mobile device may provide data to such an embodiment within the scope of exemplary embodiments, either locally at the mobile device or via a data network.

[0051] The exemplary embodiments are described using specific studies, code, hardware, algorithms, designs, architectures, protocols, layouts, schematics, and tools as examples only, and are not limited to the exemplary embodiments. Furthermore, in some instances, the exemplary embodiments are described using specific software, tools, and data processing environments as examples only, for clarity of explanation. The exemplary embodiments may be used in conjunction with other structures, systems, applications, or architectures for equivalent or similar purposes. For example, other equivalent devices, structures, systems, applications, or architectures therefor may be used in conjunction with such embodiments of the present disclosure within the scope of the present disclosure. An exemplary embodiment may be implemented in hardware, software, or a combination thereof.

[0052] The examples in this disclosure are used merely for clarity of explanation and are not intended to be limiting of the exemplary embodiments. Additional data, operations, actions, tasks, acts, and manipulations are contemplated from this disclosure and are within the scope of the exemplary embodiments.

[0053] Any advantages listed herein are merely examples and are not intended to limit the exemplary embodiments. Additional or different advantages may be realized by certain exemplary embodiments. Furthermore, certain exemplary embodiments may have some, all, or none of the above-listed advantages. [Example Data Processing Environment]

[0054] 1 illustrates a block diagram of a network of data processing systems in which exemplary embodiments may be implemented. Data processing environment 100 is a network of computers in which exemplary embodiments may be implemented. Data processing environment 100 includes network 102. Network 102 is the medium used to provide communications links between the various devices and computers connected to each other within data processing environment 100. Network 102 may include connections such as wired, wireless communication links, or fiber optic cables.

[0055] Client or server are merely exemplary roles for some data processing systems connected to network 102 and are not intended to exclude other configurations or roles of these data processing systems. Server 104 and server 106 are coupled to network 102 along with storage unit 108. Software applications may run on any computer in data processing environment 100. Client 110, client 112, and client 114 are also coupled to network 102. Data processing systems such as clients (client 110, client 112, client 114), UI automation engine 126, and device 122 may contain data and have software applications or software tools running thereon. Server 104 and server 106 may include one or more graphics processing units (GPUs) for statistical analysis or machine learning.

[0056] By way of example only, and without implying any limitation to such an architecture, Figure 1 illustrates several components that may be used in an exemplary implementation of one embodiment. For example, the servers and clients are merely examples and are not intended to imply limitation to a client-server architecture. As another example, one embodiment may be distributed across several data processing systems and data networks as shown, while another embodiment may be implemented on a single data processing system, all within the scope of the exemplary embodiment.

[0057] The data processing system (UI automation engine 126, server 104, server 106, client 110, client 112, client 114, and device 122) also represents exemplary nodes within a cluster, partitions, and other configurations suitable for implementing an embodiment.

[0058] Server 104, server 106, storage unit 108, client 110, client 112, client 114, device 122, and UI automation engine 126 may be coupled to network 102 using wired connections, wireless communication protocols, or other suitable data connection capabilities. Client 110, client 112, and client 114 may be, for example, personal computers or network computers.

[0059] In the illustrated example, a server may provide data such as boot files, operating system images, and applications to clients 110, 112, and 114. Clients 110, 112, and 114 may be clients to the server in this example. Clients 110, 112, and 114, or some combination thereof, may include their own data, boot files, operating system images, and applications. Data processing environment 100 may include additional servers, clients, and other devices not shown. Server 104 may include a server application 116 that may be configured to implement one or more of the functions described herein, according to one or more embodiments. Server application 116, client application 124, and / or UI automation engine 126 may include UI automation code 118 configured for UI automation. In some embodiments, UI automation engine 126 may be part of or form a part of a server or client described herein.

[0060] Device 122 is one example of a device as described herein. For example, device 122 may take the form of a smartphone, a tablet computer, a laptop computer, a client 110 in stationary or portable form, or any other suitable device. Any software application described as executing in another data processing system of FIG. 1 may be configured to execute in a similar manner in device 122. Any data or information stored or generated in another data processing system of FIG. 1 may be configured to be stored or generated in a similar manner in device 122. Database 120 of storage unit 108 may store one or more data samples for computation, as used herein.

[0061] Data processing environment 100 may be the Internet. Network 102 may represent a collection of networks and gateways that use Transmission Control Protocol / Internet Protocol (TCP / IP) and other protocols to communicate with each other. At the core of the Internet is a backbone of data communication links between major nodes or host computers, including thousands of commercial, government, educational, and other computer systems, that route data and messages. Of course, data processing environment 100 may be implemented as many different types of networks, such as, for example, an intranet, a local area network (LAN), or a wide area network (WAN). Figure 1 is intended as an example, and not as an architectural limitation for various illustrative embodiments.

[0062] Among other uses, data processing environment 100 may be used to implement a client-server environment in which exemplary embodiments may be implemented. A client-server environment allows software applications and data to be distributed across a network so that applications function by using interactivity between client and server data processing systems. Data processing environment 100 may employ a service-oriented architecture, in which interoperable software components distributed across a network may be packaged together as a coherent business application. Data processing environment 100 may take the form of a cloud and employ a cloud computing model of service delivery to enable convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the provider of the service.

[0063] Various aspects of the present disclosure are described through text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). With respect to any flowchart, operations may be performed in an order different from that shown in a given flowchart, depending on the technology involved. For example, two operations shown in successive flowchart blocks may be performed in the reverse order, as a single integrated step, simultaneously, or in an at least partially overlapping manner, also depending on the technology involved.

[0064] A computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also called "mediums") collectively contained in one or more storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. Without limitation, the computer-readable storage medium may be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded device (such as a punch card or pits / lands formed on a major surface of a disk), or any suitable combination of the foregoing. Computer-readable storage media, as the term is used in this disclosure, is not to be construed as storage in the form of a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through fiber optic cables, electrical signals transmitted through wires, and / or other transmission media.As will be appreciated by those skilled in the art, data is typically moved at some infrequent time during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, but the above does not make a storage device transient, as data is not transient while it is stored.

[0065] Computing environment 200 includes an example of an environment for executing at least some of the computer code involved in implementing the methods of the present invention, such as UI automation code 118. In addition to UI automation code 118, computing environment 200 includes, for example, a computer 202, a wide area network 228 (WAN), an end user device 230 (EUD), a remote server 232, a public cloud 240, and a private cloud 236. In this embodiment, computer 202 includes a set of processors 204 (including processing circuitry 206 and cache 208), a communications fabric 210, volatile memory 212, persistent storage 214 (including an operating system 216 and the above-identified UI automation code 118), a set of peripheral devices 218 (including a set of user interface (UI) devices 220, storage 222, and a set of Internet of Things (IoT) sensors 224), and a network module 226. Remote server 232 includes a remote database 234. The public cloud 240 includes a gateway 238 , a cloud orchestration module 242 , a set of host physical machines 246 , a set of virtual machines 244 , and a set of containers 248 .

[0066] Computer 202 may take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or later developed that is capable of executing programs, accessing a network, or querying a database, such as remote database 234. As is well understood in the art of computer technology, and depending on the technology, the performance of a computer-implemented method may be distributed among multiple computers and / or multiple locations. However, in this description of computing environment 200, to keep the explanation as simple as possible, the detailed discussion focuses on a single computer, specifically computer 202. Although computer 202 is not depicted in FIG. 2 within a cloud, it may be located within a cloud. However, computer 202 need not reside within a cloud except to any extent that may be indicated in the affirmative.

[0067] The processor set 204 includes one or more computer processors of any type now known or later developed. The processing circuitry 206 may be distributed across multiple packages, e.g., multiple linked integrated circuit chips. The processing circuitry 206 may implement multiple processor threads and / or multiple processor cores. The cache 208 is memory located within the processor chip package and is typically used for data or code that should be available for fast access by threads or cores executing on the processor set 204. Cache memory is typically organized into multiple levels depending on relative proximity to the processing circuitry. Alternatively, some or all of the cache for a processor set may be located “off-chip.” In some computing environments, the processor set 204 may be designed to operate with qubits and perform quantum computing.

[0068] Computer-readable program instructions are typically loaded into computer 202 and cause processor set 204 of computer 202 to perform a series of operational steps, thereby performing a computer-implemented method; thus, the instructions so executed will instantiate the method specified in the computer-implemented method flowcharts and / or descriptions contained in this document (collectively, the "methods of the present invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 208 and other storage media discussed below. The program instructions and associated data are accessed by processor set 204 to control and direct the performance of the methods of the present invention. In computing environment 200, at least some of the instructions for performing the methods of the present invention may be stored in UI automation code 118 in persistent storage 214.

[0069] Communications fabric 210 is the signal-conducting pathway that allows various components of computer 202 to communicate with one another. Typically, this fabric is made up of switches and conductive pathways, such as those that make up buses, bridges, physical input / output ports, and the like. Other types of signal communication pathways may be used, such as fiber optic and / or wireless communication pathways.

[0070] Volatile memory 212 may be any type of volatile memory now known or later developed. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory 212 is characterized by random access, although this is not required unless affirmatively indicated. In computer 202, volatile memory 212 is located in a single package and is internal to computer 202, although alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 202.

[0071] Persistent storage 214 is any form of non-volatile storage for a computer, now known or later developed. The non-volatility of this storage means that stored data is maintained regardless of whether power is supplied to computer 202 and / or directly to persistent storage 214. While persistent storage 214 may be read-only memory (ROM), typically at least a portion of persistent storage allows data to be written, data to be deleted, and data to be rewritten. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 216 may take several forms, such as various known proprietary operating systems employing a kernel or open-source Portable Operating System Interface-type operating systems. The code included in UI automation code 118 typically includes at least some of the computer code involved in implementing the methods of the present invention.

[0072] Peripheral device set 218 includes a set of peripheral devices of computer 202. Data communication connections between peripheral devices and other components of computer 202 may be implemented in various ways, such as Bluetooth® connections, near field communication (NFC) connections, connections made by cable (such as a Universal Serial Bus (USB)-type cable), insertion-type connections (e.g., Secure Digital (SD) cards), connections made through a local area communication network, and even connections made through a wide area network such as the Internet. In various embodiments, UI device set 220 may include components such as a display screen, speakers, microphones, wearable devices (such as goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage 222 may be external storage, such as an external hard drive, or insertable storage, such as an SD card. Storage 222 may be persistent and / or volatile. In some embodiments, storage 222 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 202 is required to have large amounts of storage (e.g., computer 202 stores and manages large databases locally), then this storage may be provided by a peripheral storage device designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. IoT sensor set 224 consists of sensors that may be used in Internet of Things applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

[0073] Network module 226 is a collection of computer software, hardware, and firmware that enables computer 202 to communicate with other computers over WAN 228. Network module 226 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of network module 226 are implemented on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN)), the control and forwarding functions of network module 226 are implemented on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for implementing the methods of the present invention may be downloaded to computer 202, typically from an external computer or external storage device, through a network adapter card or network interface included in network module 226.

[0074] WAN 228 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances by any technology for transmitting computer data now known or later developed. In some embodiments, WAN 228 may be replaced and / or supplemented by a local area network (LAN) designed to transmit data between devices located in a local area, such as a Wi-Fi® network. WANs and / or LANs typically include copper transmission cables, optical fiber transmissions, wireless transmissions, and computer hardware such as routers, firewalls, switches, gateway computers, and edge servers.

[0075] End-user device (EUD) 230 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating computer 202) and may take any of the forms discussed above in connection with computer 202. EUD 230 typically receives useful and useful data from the operation of computer 202. For example, in the hypothetical case where computer 202 is designed to provide recommendations to the end user, the recommendations would typically be communicated from computer 202's network module 226 over WAN 228 to EUD 230. In this manner, EUD 230 can display or otherwise present the recommendations to the end user. In some embodiments, EUD 230 may be a client device such as a thin client, a heavy client, a mainframe computer, a desktop computer, or the like.

[0076] Remote server 232 is any computer system that provides at least some data and / or functionality to computer 202. Remote server 232 may be controlled and used by the same entity that operates computer 202. Remote server 232 represents a machine that collects and stores useful and useful data for use by other computers, such as computer 202. For example, in the hypothetical case where computer 202 is designed and programmed to provide recommendations based on historical data, then this historical data may be provided to computer 202 from remote database 234 of remote server 232.

[0077] Public cloud 240 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing power, without direct active management by users. Cloud computing typically leverages resource sharing to achieve consistency and economies of scale. Direct active management of public cloud 240's computing resources is performed by computer hardware and / or software in cloud orchestration module 242. The computing resources provided by public cloud 240 are typically implemented by virtual computing environments running on various computers comprising host physical machine set 246, a collection of physical computers within and / or available to public cloud 240. Virtual computing environments (VCEs) typically take the form of virtual machines from virtual machine set 244 and / or containers from container set 248. It is understood that these VCEs may be stored as images and can be transferred among and between various physical machine hosts, either as images or after instantiation of the VCEs. Cloud orchestration module 242 manages the transfer and storage of images, deploys new instantiations of VCE, and manages active instantiations of VCE deployments. Gateway 238 is a collection of computer software, hardware, and firmware that enables public cloud 240 to communicate over WAN 228.

[0078] We now provide some further information about virtualized computing environments (VCEs). A VCE can be stored as an "image." From this image, a new, active instance of the VCE can be instantiated. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to a feature of an operating system in which the kernel allows multiple isolated user space instances, called containers, to exist. These isolated user space instances typically behave as actual computers from the perspective of programs running within them. A computer program running on a typical operating system can utilize all of the computer's resources, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running inside a container can only use the contents of the container and the devices assigned to that container; this feature is known as containerization.

[0079] Private cloud 236 is similar to public cloud 240, except that the computing resources are available only for use by a single enterprise. While private cloud 236 is shown as being in communication with WAN 228, in other embodiments, the private cloud may be completely disconnected from the Internet and accessible only through a local / private network. A hybrid cloud is a composite of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. While each of the multiple clouds remains a separate, discrete entity, the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the constituent clouds. In this embodiment, both public cloud 240 and private cloud 236 are part of a larger hybrid cloud. [Example Architecture]

[0080] Reference is now made to FIG. 3 , which illustrates the architecture of first and second applications configured as an application 314 and a plug-in module 302, respectively, in which UI automation may be implemented. The plug-in module 302 and the application 314 may operate based on the UI automation code 118 or the UI automation engine 126 to implement UI automation. The plug-in module may further include or operate an object detection module 304, a demonstration transformer 306, an object selection module 308, and an automation corrector 310, as discussed herein. The plug-in module 302 may perform, or be controlled by a controller to perform, one or more operations described herein. In one embodiment, the plug-in module 302 may be deployed as a modular software component (e.g., a plug-in, extension, add-on, or other module configured to be plugged in) for a web browser. However, the plug-in module 302 may alternatively be associated with other applications, such as desktop applications, mobile applications, or other applications, in which UI automation may be implemented. Thus, although a web application may be specified, this is not meant to be limiting, as other applications 314 may be implemented in light of this description.

[0081] During the teaching phase, in which teaching actions are performed on the application 314 associated with the plug-in module 302, an automation workflow demonstration 312 is performed to generate an automation workflow that may include actions to be performed upon re-execution of the automation. The object detection module 304 may perform page understanding of the page of the application 314, in which an analysis of the objects 316 (i.e., interactable elements or UI elements, e.g., inputs, outputs, buttons, forms, etc.) on the page and their corresponding labels is performed to capture the original state of the page, including the topology of the objects 316. For example, as can be seen in the exemplary embodiment of FIGS. 4A-4B, a page 402 may include the objects 316 shown in FIG. 4A, which may be classified as, for example, an input 406, an output (not shown), a form 404, or a button 408 in the visualized page understanding of FIG. 4B. The corresponding labels 318 of the objects 316 may also be shown. Page understanding helps analyze the interactable elements on the page by understanding the overall state of the page, including the topology of the UI elements. Page understanding can help identify the functional role of each element of the page 402, which helps the automated system described herein to correctly select elements. A page understanding algorithm may be used for page understanding, which may be rule-based, where each rule may represent either a topology group or an interactable element. Elements from the page may be collected based on the current rules and organized or grouped together based on their relationships in a model that describes the structure of objects 316 on the application 314, for example, based on their relationships in an HTML DOM (HyperText Markup Language Document Object Model) file.

[0082] According to an exemplary embodiment, heuristics for the current page may be used to generate a list of all objects 316 on the current page. The objects 316 and their corresponding labels 318 may then be combined based on reference attributes and location information. In one embodiment, page understanding is performed using not only DOM-based techniques but also pixel proximity logic. An element hierarchy, such as lists, groups, and forms, may then be constructed, and a complete textual (or visual) representation of the page 402 is displayed, with unique names for each of the multiple objects 316.

[0083] 3 , the object detection module 304 may detect, on the application 314, one or more objects 316 and one or more corresponding labels 318 associated with the demonstration 312. In other aspects, the object detection module 304 may alternatively or additionally detect the one or more objects 316 and one or more corresponding labels 318 through natural language input. Furthermore, the object detection module 304 may verify the one or more objects 316 and one or more corresponding labels 318 with a user to establish the accuracy of the detection.

[0084] Additionally, the demonstration transformer 306 may convert the demonstration 312 into one or more natural language instructions. In one embodiment, the natural language instructions may be further stored in a knowledge representation database within a knowledge representation dataset that includes one or more workflows for automation. Thus, the knowledge representation database may comprise multiple automation workflows or knowledge representation datasets that may be re-executed during runtime for automation purposes.

[0085] During a runtime phase in which runtime actions are performed on an application 314 associated with a plug-in module 302, an object selection module 308 may semantically select an object 316 based on the LLM. More specifically, semantic selection may be performed that reflects the intent captured in the stored natural language instructions to select the correct object during runtime despite any changes in the object's corresponding label.

[0086] To semantically select objects, the object selection module 308 can implement a novel page understanding that detects the current state of the objects 316 within the page. The object selection module 308 then obtains a plurality of possible objects 316 that correspond to the identified natural language instruction for automation. Further, the object selection module 308 can provide the identified natural language instruction and the plurality of possible objects 316 to the LLM. In response, the object selection module 308 receives from the LLM the object 316 that corresponds or is most likely to correspond to the identified flow. The object selection module 308 can select that object 316 during runtime.

[0087] In another aspect, if an operator needs to modify a stored automated workflow by a user, the automated modifier 310 may receive natural language input for use in modifying the stored natural language instructions. The modifications may be performed to update, replace, or add one or more instructions to the automated workflow.

[0088] 5 illustrates a block diagram of a UI automation system 502 according to an example embodiment. The UI automation system 502 includes a web browser 504, a server 532, a knowledge store 534, and an LLM 530 communicatively coupled together via an application programming interface (API 520).

[0089] Web browser 504 may include a web browser plug-in 506 and a web application 518. Web browser plug-in 506 may include an instruction module 508, a runtime module 510, a UI interactor 512, a page understanding module 514, and a player 516. In one embodiment, UI interactor 512 is a tool or library configured to simulate user interactions with a graphical user interface. In one embodiment, page understanding module 514 analyzes and understands the structure, elements, and behavior of the user interface of the web application.

[0090] Additionally, server 532 is an example of server 104 or server 106 and may include a controller, referred to herein as orchestrator 522, that controls or implements one or more actions described herein. Server 532 further includes semantic page understanding module 524, model pipeline 526, and LLM manager 528. In one embodiment, orchestrator 522 is designed to automate and execute various teaching and runtime actions. In one aspect, orchestrator 522 is communicatively coupled to knowledge store 534, such as database 120 of FIG. 1, which serves as a repository for knowledge representation datasets. This enables orchestrator 522 to retrieve the necessary stored natural language instructions for a given task. Orchestrator 522 may also be communicatively linked to semantic page understanding module 524 and LLM manager 528. During runtime execution of the stored automated workflow, the orchestrator may follow a systematic approach using the semantic page understanding module 524 and the runtime module 510. Meanwhile, during the teaching phase, the orchestrator 522 may have the teaching module 508 and the page understanding module 514 generate a page understanding of the page 402.

[0091] When an operator intends to execute an existing automation, the operator may begin by sending a natural language utterance (e.g., log a call) that can trigger a workflow. The orchestrator 522 then retrieves the associated workflow (knowledge representation dataset) containing one or more natural language instructions for that task from the knowledge store 534. In one aspect, the orchestrator 522 employs a model pipeline 526 to perform Named Entity Recognition (NER), a natural language processing (NLP) method that extracts information from text and maps the operator's utterance to the parameters of the workflow. For each identified natural language instruction in the workflow, a state analysis from the semantic page understanding module 524 can be computed. This process involves the collection of all matching objects 316 on the page 402. Further, a search of these objects 316 to determine which ones correspond to the identified natural language instructions is computed by providing the objects 316 and the natural language instructions to the LLM 530 via the LLM manager 528, which assists in selecting the most appropriate element from the screen based on the context and the instruction. Finally, the orchestrator 522 allows the player 516 to select the object 316 identified by the LLM 530. This comprehensive process ensures that each flow step is performed accurately, in close coordination with the user's needs and the specifics of the web page being interacted with.

[0092] In some cases, the process in the teaching phase may be similar. For example, if an operator types "click home," the orchestrator 522 may perform a task such as NER, called instruction generalization, to decompose the natural language instruction into several parts, such as the action "click," the element name "home," and any parameters. The orchestrator 522 may trigger semantic page understanding to select the correct object 316, perform a similarity search, and provide the output to the LLM pipeline. The action is then performed and feedback is received from the operator about the action's accuracy.

[0093] Turning now to FIG. 6 , a visualized automation workflow 602 including natural language instructions 604 is disclosed according to an exemplary embodiment. In one embodiment, the automation workflow is a knowledge representation dataset that includes the specification of a program to be executed. It may include sample utterances that can trigger the flow, application names, external parameters, and instructions to be performed as part of the automation (e.g., click, type, type-submit, navigate, copy, paste, smart Q&A, conditional instructions, etc.). For instructions to interact with on-screen UI elements, the information may include the UI action type (e.g., click, type, type-submit, etc.), UI element name (e.g., advanced search), parameters / variables / references / constants, required or optional, where optional means only if the user provides and mentions it in the utterance.

[0094] In an exemplary embodiment, a workflow may include a knowledge document that may be serialized as a YAML file. Additionally, sample utterances (sentences) may be converted into embeddings and stored in a vector store along with the YAML file and additional metadata, such as the application name. Before storing specific sample utterances, they may be processed to represent more generic utterances, for example, by replacing specific entities with their generic types (e.g., John Doe may be converted to Person Name).

[0095] 7, a routine for UI automation is disclosed according to an example embodiment. At block 702, the UI automation engine 126 receives a demonstration of an automation to be performed on an application 314. At block 704, the UI automation engine 126 detects one or more objects 316 and one or more corresponding labels 318 on the application 314 that are associated with the demonstration. At block 706, the UI automation engine 126 converts the demonstration into one or more natural language instructions.

[0096] In block 708, during runtime action, the UI automation engine 126 semantically selects an object based on a large-scale language model (LLM), where the semantic selection reflects the intent captured in one or more natural language instructions, regardless of any changes in the corresponding labels of the object 316.

[0097] Turning now to FIG. 8 , another routine 800 for UI automation is disclosed, according to an example embodiment. At block 804, the UI automation engine 126 may receive a demonstration of automation to be performed on an application. In one embodiment, the application may be a web application. In one embodiment, the automation may be performed by programming by demonstration (PbD). In one embodiment, the demonstration may be received during a teaching phase. At block 806, the UI automation engine 126 may detect one or more objects 316 and one or more corresponding labels 318 on the application associated with the demonstration. At block 808, the UI automation engine 126 may convert the demonstration into one or more natural language instructions 604. In one embodiment, the natural language instructions 604 may be stored in a knowledge representation dataset or workflow. At block 810, the UI automation engine 126 may detect a change in the corresponding label 318 associated with the identified natural language instruction 604. At block 812, the UI automation engine 126 may obtain multiple possible objects corresponding to the change. At block 814, the UI automation engine 126 may provide the identified natural language instruction and multiple possible objects to the LLM. At block 816, the UI automation engine 126 may receive an indication of the object 316 that corresponds or is most likely to correspond to the identified natural language instruction from the LLM. At block 818, the UI automation engine 126 may select an object during a runtime action based on the LLM. [Conclusion]

[0098] The description of various embodiments of the present teachings has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0099] While the foregoing has set forth what is believed to be the best mode and / or alternative examples, it should be understood that various modifications may be made thereto, that the subject matter disclosed herein may be implemented in a variety of forms and examples, and that the teachings may be applied to numerous applications, only a few of which are described herein. It is intended in the following claims to claim any and all applications, modifications, and variations that fall within the true scope of the present teachings.

[0100] The components, steps, features, objects, benefits, and advantages discussed herein are merely exemplary. None of them, nor the discussions associated therewith, are intended to limit the scope of protection. While various advantages have been discussed herein, it will be understood that not all embodiments necessarily include all advantages. Unless otherwise specified, all measurements, values, ratings, positions, dimensions, sizes, and other specifications set forth herein, including in the following claims, are approximate and not exact. They are intended to have a reasonable range consistent with the functions to which they relate and those customary in the technical field to which they pertain.

[0101] Numerous other embodiments are contemplated, including embodiments having fewer, additional, and / or different components, steps, features, objects, benefits, and advantages, including embodiments in which components and / or steps are arranged and / or ordered differently.

[0102] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0103] These computer-readable program instructions may be provided to a processor of a computer, special purpose computer, or other programmable data processing apparatus to cause a machine to create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in a certain manner; thus, the computer-readable storage medium having instructions stored thereon constitutes an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0104] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device, thereby creating a computer-implemented process; thus, the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0105] The flowcharts and block diagrams in the figures herein illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or, depending on the functionality involved, the blocks may sometimes be executed in the reverse order. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0106] While the foregoing has been described in conjunction with exemplary embodiments, it should be understood that the term "exemplary" is intended to be merely an example, not best or optimal. Except as noted immediately above, nothing described or illustrated is intended to, or should be construed to, provide the public with any component, step, feature, object, benefit, advantage, or equivalent, whether claimed or not.

[0107] It will be understood that the terms and expressions used herein have the ordinary meanings ascribed to such terms and expressions with respect to their corresponding fields of inquiry and study, unless a specific meaning is otherwise stated herein. Relative terms such as first and second, and the like, may be used solely to distinguish one entity or action from another, without necessarily requiring or suggesting any actual relationship or order between such entities or actions. The terms "comprises," "comprising," or any other variation thereof, are intended to cover non-exclusive inclusions, and thus a process, method, article, or apparatus that includes a list of elements does not include only those elements, but may include other elements not expressly listed or inherent in such process, method, article, or apparatus. An element preceded by "a" or "an" does not, in the absence of further constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that includes that element.

[0108] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of this technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments have more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

Claims

1. receiving, by the computing device, a demonstration of automation to be performed on the application; detecting, on the application, one or more objects and one or more corresponding labels associated with the demonstration; the computing device converting the demonstration into one or more natural language instructions; and semantically selecting objects based on a large-scale language model (LLM) during runtime action of the computing device, the semantically selecting reflecting the intent captured in the one or more natural language instructions despite any changes in corresponding labels of the objects; A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the automation is performed by programming by demonstration (PbD).

3. detecting the one or more objects and the one or more corresponding labels by performing page understanding to generate a dataset representing an original state of the objects available on a page of the application; The computer-implemented method of claim 1 further comprising:

4. The computer-implemented method of claim 1 , wherein the one or more natural language instructions are stored in a knowledge representation dataset.

5. detecting a change in a corresponding label associated with an identified natural language instruction of the one or more natural language instructions; obtaining a plurality of possible objects corresponding to said change; providing the identified natural language instruction and the plurality of possible objects to the LLM; receiving from the LLM an indication of the object that most likely corresponds to the identified natural language instruction; and Selecting the object The computer-implemented method of claim 4 , further comprising semantically selecting the object by:

6. 6. The computer-implemented method of claim 5, further comprising: performing page understanding to generate a data set representing a current state of objects available on a page of the application; and detecting the changes in response to performing the page understanding.

7. The computer-implemented method of claim 5 , wherein the step of semantically selecting the object is performed during the runtime action.

8. The computer-implemented method of claim 1 , wherein the demonstration is received during a teaching action.

9. The computer-implemented method of claim 1 , wherein the application is a plug-in module configured to be plugged into a web, desktop, or mobile application.

10. Receive a demonstration of the automation to be implemented on the application; Detecting, on the application, one or more objects and one or more corresponding labels associated with the demonstration; converting the demonstration into one or more natural language instructions; and semantically selecting the object based on a large-scale language model (LLM) to reflect the intent of the one or more natural language instructions despite any changes to the object's corresponding label during runtime action. A processor configured to A system comprising:

11. The system of claim 10 , wherein the one or more objects are user interface elements.

12. The system of claim 10 , wherein the automation is performed by PbD (Programming by Demonstration).

13. The system of claim 10 , wherein the one or more natural language instructions are stored in a knowledge representation dataset.

14. The processor: detecting a change in a corresponding label associated with an identified natural language instruction of the one or more natural language instructions; obtaining a plurality of possible objects corresponding to said change; providing the identified natural language instruction and the plurality of possible objects to the LLM; receiving from the LLM an indication of the object that most likely corresponds to the identified natural language instruction; and Select the object The system of claim 13 configured to:

15. The system of claim 10 , wherein the demonstration is received during a teaching action.

16. 16. The system of claim 10, wherein the application is a plug-in module configured to be plugged into a web, desktop, or mobile application.

17. When executed by a computer system, the computer system: Receive a demonstration of the automation to be implemented on the application; Detecting, on the application, one or more objects and one or more corresponding labels associated with the demonstration; converting the demonstration into one or more natural language instructions; and semantically selecting an object based on a large-scale language model (LLM) during runtime action, the semantic selection reflecting the intent captured in the one or more natural language instructions despite any changes in the corresponding labels of the object; A computer program comprising computer-executable instructions for causing a computer to:

18. 18. The computer program of claim 17, wherein the automation is performed by PbD (Programming by Demonstration).

19. 19. A computer program product according to claim 17 or 18, wherein the one or more natural language instructions are stored in a knowledge representation dataset.

20. 20. The computer program product of claim 17, wherein the application is a plug-in module configured to be plugged into a web, desktop, or mobile application.

21. To semantically select the object, the computer-executable instructions include: detecting a change in a corresponding label associated with an identified natural language instruction of the one or more natural language instructions; obtaining a plurality of possible objects corresponding to said change; providing the identified natural language instruction and the plurality of possible objects to the LLM; receiving from the LLM an indication of the object that most likely corresponds to the identified natural language instruction; and Select the object 20. A computer program as claimed in claim 19, configured to:

22. receiving, by the computing device, a demonstration of automation to be performed on the application; converting the demonstration into one or more natural language instructions; semantically selecting objects based on a large-scale language model during runtime action of the computing device, the semantically selecting reflecting the intent captured in the one or more natural language instructions despite any changes in corresponding labels of the objects; and modifying the automation based on new natural language user input A computer-implemented method comprising:

23. 23. The computer-implemented method of claim 22, wherein the modifying is performed to update, replace, or add one or more natural language instructions to the automation.

24. a knowledge representation dataset including one or more natural language instructions for automation; and receiving the knowledge representation dataset; and semantically selecting the object based on a large-scale language model (LLM) to reflect the intent of the one or more natural language instructions, regardless of any changes to the object's corresponding label during runtime actions of the automation. A processor configured to A system comprising:

25. The processor: Receive a demonstration of the automation to be implemented on the application; Detecting, on the application, one or more objects and one or more corresponding labels associated with the demonstration; and converting the demonstration into one or more natural language instructions representing actions to be performed on the one or more objects; 25. The system of claim 24, further configured to:

Images