Computer implementation methods, computer systems, and computer programs (identification of user interface elements within application pages using heuristic rules and large-scale language models)

Abstract

The challenges in maintaining high accuracy stem from limitations in interpreting complex data (e.g., complex UI elements), adapting to changes in dynamic environments, and handling unexpected situations. High accuracy is achieved by experts who can navigate complex coding and design requirements. [Solution] The computer implementation method comprises the steps of: applying a set of heuristic rules to an application page to identify a plurality of user interface (UI) elements in the page; displaying the page, which has the step of visually highlighting each of the plurality of UI elements identified by the set of rules; receiving user feedback to identify each of the plurality of UI elements that are not visually highlighted; and creating new rules to identify each of the UI elements that are not visually highlighted using a first pipeline of a large language model.

Description

Technical Field

[0001] The present disclosure generally relates to user interface (UI) automation, and more specifically, to tools for providing information about UI elements.

Background Art

[0002] UI automation refers to a framework that enables an application to provide and obtain information about UI elements. A UI automation client can register for specific event notifications, and they can require that specific UI automation properties and control pattern information be passed within their event handlers. UI automation also provides tools for automating tasks across multiple applications.

[0003] UI automation tools may include an automation agent for selecting the correct UI elements for interaction. Selecting elements with high accuracy ensures reliable results and reduces the risk of errors. As a result, computational efficiency is improved.

[0004] The challenges in maintaining high accuracy arise from the interpretation of complex data (e.g., complex UI elements), adaptation to changes in a dynamic environment, and limitations in handling unexpected situations. High accuracy is achieved by experts who manipulate complex coding and design requirements.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The challenges in maintaining high accuracy arise from the interpretation of complex data (e.g., complex UI elements), adaptation to changes in a dynamic environment, and limitations in handling unexpected situations. High accuracy is achieved by experts who manipulate complex coding and design requirements.

Means for Solving the Problems

[0006] According to one embodiment of the present disclosure, a computer implementation method comprises the steps of: applying a set of heuristic rules to a page of an application to identify a plurality of UI elements in the page; displaying the page, including visually highlighting each of the plurality of UI elements identified by the set of rules; receiving user feedback to identify each of the plurality of UI elements that are not visually highlighted; and creating new rules to identify each of the UI elements that are not visually highlighted using a first pipeline of the LLM.

[0007] In some embodiments, the set of heuristic rules includes a first subset for identifying individual UI elements; a second subset for detecting groups of UI elements; and a third subset for detecting UI element contexts.

[0008] In some embodiments, the method further comprises a step of iteratively testing and refining each new rule until its corresponding UI element is accurately identified.

[0009] In some embodiments, the step of using the first pipeline includes the steps of creating a prompt for a given new rule and sending the prompt to the LLM. The prompt is iteratively tested and refined with user feedback until the given new rule accurately identifies its corresponding UI element.

[0010] In some embodiments, the prompt for the given new rule comprises an instruction for creating a selector and the attributes of the corresponding UI element.

[0011] In some embodiments, the method further comprises a step of proposing a name for a given UI element that is not visually highlighted, using a second pipeline of the LLM.

[0012] In some embodiments, the step of using the second pipeline includes the steps of creating a prompt for a given UI element and sending the prompt to the LLM. The prompt has instructions for selecting a number of names and a one-shot input. The one-shot input has the following format: [style / state modifier][name][type][anchoring reference], where the style, state, name, and type are attributes of the given UI element, and the anchoring reference describes the context of the given UI element.

[0013] In some embodiments, the method further comprises the steps of: identifying the state of a stateful UI element; displaying the identified state; and obtaining additional user feedback to update any displayed state.

[0014] According to one embodiment of the present disclosure, the computer system comprises: a memory having computer-readable instructions; and one or more processors for executing the computer-readable instructions, configured to perform the following steps: a procedure for running an application for displaying a page including a UI; a procedure for applying a set of heuristic rules to the page to identify a plurality of UI elements; a procedure for displaying the page including the plurality of UI elements; a procedure for visually highlighting each of the plurality of UI elements identified by the set of rules; a procedure for receiving user feedback to identify each of the plurality of UI elements that are not visually highlighted; and a procedure for creating a new rule to identify each of the UI elements that are not visually highlighted using a first pipeline of the LLM.

[0015] In some embodiments, the computer-readable instructions, when executed, further configure the one or more processors to iteratively test and adjust each new rule until its corresponding UI element is accurately identified.

[0016] In some embodiments, a prompt for a given new rule is created and sent to the LLM. The prompt is iteratively refined with user feedback until the given new rule accurately identifies its corresponding UI element.

[0017] In some embodiments, the prompt for the given new rule comprises an instruction for creating a selector and the attributes of the corresponding UI element.

[0018] In some embodiments, the computer-readable instructions, when executed, further configure the one or more processors to propose names for a given UI element that is not visually highlighted using the second pipeline of the LLM.

[0019] According to one embodiment of the present disclosure, a computer program product comprises one or more computer-readable memory devices encoded with data containing computer-readable instructions that cause a set of processors to perform a method for identifying UI elements within a page of the application. The method includes the steps of: applying a set of heuristic rules to the page to identify the UI elements; displaying the page; visually highlighting each of the plurality of UI elements identified by the set of rules; receiving user feedback to identify each of the UI elements that are not visually highlighted; and using a first pipeline of the LLM to create new rules to identify each of the UI elements that are not visually highlighted.

[0020] In some embodiments, the computer-readable instructions, once executed, further configure the set of processors to iteratively test and adjust each new rule until its corresponding UI element is accurately identified.

[0021] In some embodiments, a prompt for a given new rule is created and sent to the LLM. The prompt is iteratively adjusted with the user feedback until the given new rule accurately identifies its corresponding UI element.

[0022] In some embodiments, the prompt for the given new rule has instructions for creating a selector and attributes of the corresponding UI element.

[0023] In some embodiments, when the computer-readable instructions are executed, they further configure the processor set to propose a name for each UI element that is not visually highlighted using a second pipeline of the LLM.

Brief Description of the Drawings

[0024] The drawings are of exemplary embodiments. These drawings do not illustrate all embodiments. Other embodiments may be used additionally or instead. Details that may be apparent or unnecessary may be omitted for space savings or more effective illustration. Some embodiments may be practiced using additional components or steps or without using all of the illustrated components or steps and / or both. When the same numeral appears in different drawings, it refers to the same or similar components or steps.

[0025] [Figure 1] A computing environment that conforms to an exemplary embodiment.

[0026] [Figure 2] A computer-implemented method that conforms to an exemplary embodiment.

[0027] [Figure 3] A computer-implemented method that conforms to an exemplary embodiment.

[0028] [Figure 4] This is a file of heuristic rules consistent with an exemplary embodiment.

[0029] [Figure 5] A screenshot of a webpage in which specific UI elements are visually highlighted, matching an exemplary embodiment.

[0030] [Figure 6] This is a prompt to generate proposed names for UI elements that match the exemplary embodiments.

[0031] [Figure 7] This is a prompt for generating a new heuristic rule consistent with the exemplary embodiment. [Modes for carrying out the invention]

[0032] [Overview and Support] The following detailed explanation includes many specific details as examples to provide a complete understanding of the relevant teachings. However, it is clear that these teachings can be practiced without such details. In other cases, to avoid unnecessarily ambiguous aspects of these teachings, well-known methods, procedures, components, and / or circuits or combinations thereof are described relatively broadly without detail.

[0033] This disclosure generally relates to the identification of UI elements within application pages. The concepts considered in this disclosure utilize large language models (LLMs), but not for the direct identification of UI elements. LLMs have probabilistic outputs, and the resulting uncertainty can complicate their use in critical automation processes. Instead, heuristic rules are used to identify UI elements. If a heuristic rule fails to identify a UI element, an LLM is used to add a new rule or to refine an existing rule to properly identify that UI element.

[0034] According to one embodiment of the present disclosure, a computer implementation method comprises the steps of: applying a set of heuristic rules to a page of an application to identify a plurality of UI elements in the page; displaying the page, including visually highlighting each of the plurality of UI elements identified by the set of rules; receiving user feedback to identify each of the plurality of UI elements that are not visually highlighted; and creating new rules to identify each of the UI elements that are not visually highlighted using a first pipeline of the LLM.

[0035] The accuracy and predictability of identifying UI elements within an application's pages can be improved through a combination of heuristics, user feedback, and LLM (Limited Licensing Method). Furthermore, accuracy and predictability can be improved by non-technical users who do not possess the skills or training to navigate complex coding and design requirements.

[0036] This method allows UI automation tools to select UI elements with high precision. High-precision element selection ensures reliable results and reduces the risk of errors.

[0037] In some embodiments that may be combined with the previous embodiments, the set of heuristic rules includes a first subset for identifying individual UI elements; a second subset for detecting groups of UI elements; and a third subset for detecting UI element contexts. Group and context detection further improves accuracy and predictability.

[0038] In some embodiments, which may be combined with one or more previous embodiments, the highlighting step includes the steps of creating a bounding box around each UI element identified by the set of rules; and displaying text describing each UI element within the bounding box.

[0039] In some embodiments, which may be combined with one or more previous embodiments, the method further comprises a step of iteratively testing and refining each new rule until its corresponding UI element is accurately identified. Iterative testing and refining, combined with user feedback, reduces any uncertainty introduced by the LLM.

[0040] In some embodiments, which may be combined with one or more previous embodiments, the step of using the first pipeline includes the steps of creating a prompt for a given new rule, and sending the prompt to the LLM. The prompt is iteratively tested and refined with user feedback until the given new rule accurately identifies its corresponding UI element. Iterative testing and refinement, in combination with user feedback, reduces any uncertainties introduced by prompt engineering and any uncertainties introduced by the LLM.

[0041] In some embodiments, which may be combined with one or more previous embodiments, the prompt for the given new rule comprises instructions for creating a selector and attributes of the corresponding UI element. The prompt may be generated by a non-technical user who does not have the skills or training to navigate complex coding and design requirements.

[0042] In some embodiments, which may be combined with one or more previous embodiments, the method further comprises a step of proposing a name for a given UI element that is not visually highlighted using a second pipeline of the LLM. This leads to more predictable naming.

[0043] In some embodiments, which may be combined with one or more previous embodiments, the step of using the second pipeline comprises the steps of creating a prompt for a given UI element and sending the prompt to the LLM. The prompt has instructions for selecting a number of names and a one-shot input. The one-shot input has the following format: [style / state modifier][name][type][anchoring reference], where the style, state, name, and type are attributes of the given UI element, and the anchoring reference describes the context of the given UI element. The prompt may be generated by a non-technical user who does not have the skills or training to navigate complex coding and design requirements.

[0044] In some embodiments, which may be combined with one or more previous embodiments, the method further comprises the steps of: identifying the state of a stateful UI element; displaying the identified state; and obtaining additional user feedback to update any displayed state. Accuracy is further improved.

[0045] The method further includes a step of updating the set with each new rule created by the LLM. This makes the rules available to other users and other applications.

[0046] According to one embodiment of the present disclosure, the computer system comprises: a memory having computer-readable instructions; and one or more processors for executing the computer-readable instructions, configured to perform the following steps: a procedure for running an application for displaying a page including a UI; a procedure for applying a set of heuristic rules to the page to identify a plurality of UI elements; a procedure for displaying the page including the plurality of UI elements; a procedure for visually highlighting each of the plurality of UI elements identified by the set of rules; a procedure for receiving user feedback to identify each of the plurality of UI elements that are not visually highlighted; and a procedure for creating a new rule to identify each of the UI elements that are not visually highlighted using a first pipeline of the LLM.

[0047] The accuracy and predictability of identifying UI elements within an application's pages can be improved through a combination of heuristics, user feedback, and LLM (Limited Licensing Method). Furthermore, accuracy and predictability can be improved by non-technical users who do not possess the skills or training to navigate complex coding and design requirements.

[0048] This method allows UI automation tools to select UI elements with high precision. High-precision element selection ensures reliable results and reduces the risk of errors.

[0049] In some embodiments that may be combined with the previous embodiments, the computer-readable instructions, when executed, further configure the one or more processors to iteratively test and adjust each new rule until its corresponding UI element is accurately identified.

[0050] In some embodiments, which may be combined with one or more previous embodiments, a prompt for the given new rule is created and sent to the LLM. The prompt is iteratively refined with user feedback until the given new rule accurately identifies its corresponding UI element.

[0051] In some embodiments, which may be combined with one or more previous embodiments, the prompt for the given new rule comprises an instruction for creating a selector and attributes of the corresponding UI element.

[0052] In some embodiments, which may be combined with one or more previous embodiments, the computer-readable instructions, when executed, further configure the one or more processors to suggest names for a given UI element that is not visually highlighted using a second pipeline of the LLM.

[0053] According to one embodiment of the present disclosure, a computer program product comprises one or more computer-readable memory devices encoded with data containing computer-readable instructions that cause a set of processors to perform a method for identifying UI elements within a page of the application. The method includes the steps of: applying a set of heuristic rules to the page to identify the UI elements; displaying the page; visually highlighting each of the plurality of UI elements identified by the set of rules; receiving user feedback to identify each of the UI elements that are not visually highlighted; and using a first pipeline of a large-scale language learning model (LLM) to create new rules to identify each of the UI elements that are not visually highlighted.

[0054] The accuracy and predictability of identifying UI elements within an application's pages can be improved through a combination of heuristics, user feedback, and LLM (Limited Licensing Method). Furthermore, accuracy and predictability can be improved by non-technical users who do not possess the skills or training to navigate complex coding and design requirements.

[0055] This method allows UI automation tools to select UI elements with high precision. High-precision element selection ensures reliable results and reduces the risk of errors.

[0056] In some embodiments that can be combined with the previous embodiments, the computer-readable instructions, when executed, further configure the set of processors to iteratively test and adjust each new rule until its corresponding UI element is accurately identified.

[0057] In some embodiments, which may be combined with one or more previous embodiments, a prompt for a given new rule is created and sent to the LLM. The prompt is iteratively refined with user feedback until the given new rule accurately identifies its corresponding UI element.

[0058] In some embodiments, which may be combined with one or more previous embodiments, the prompt for the given new rule comprises an instruction for creating a selector and attributes of the corresponding UI element.

[0059] In some embodiments, which may be combined with one or more previous embodiments, the computer-readable instructions, when executed, further configure the set of processors to suggest names for each UI element that is not visually highlighted, using a second pipeline of the LLM. [Example computing environment]

[0060] Various aspects of this disclosure are described by explanatory text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of computer program products (CPPs). With respect to any flowchart, depending on the technology involved, operations may be performed in a different order than those shown in a given flowchart. For example, again depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated stage, simultaneously, or with at least partial time overlap.

[0061] Computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also called "mediums") that collectively comprise a set of one or more storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. Computer-readable storage media may be, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any preferred combination thereof. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (such as pits / lands formed on the main surface of a punch card or disk), or any suitable combination of the foregoing. When the term "computer-readable storage medium" is used in this disclosure, it shall not be interpreted as storage in the form of a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses passing through optical fiber cables, electrical signals communicated through wires, and / or other transmission media.As those skilled in the art will understand, data is typically moved at several intermittent points during the normal operation of a storage device, such as during access, defragmentation, or garbage collection; however, data is not transient while it is stored, and therefore the storage device is not transient.

[0062] Refer to Figure 1. The computing environment 100 includes an example of an environment for the execution of at least some computer code 200 that is involved in performing the methods of the present invention, such as identifying UI elements within an application page. In addition to the code 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, end user devices (EUDs) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including processing circuits 120 and a cache 121), a communication fabric 111, volatile memory 112, persistent storage 113 (including the operating system 122 and code 200 identified above), a peripheral device set 114, a user interface (UI), a device set 123, storage 124, an Internet of Things (IoT) sensor set 125, and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a host physical machine set 142, a virtual machine set 143, and a container set 144.

[0063] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other known or future-developed form of computer or mobile device capable of executing programs, accessing networks, or querying databases such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, the execution of a computer implementation may be distributed among multiple computers and / or multiple locations. On the other hand, in this description of the computing environment 100, in order to make the explanation as concise as possible, the detailed considerations focus on a single computer, specifically computer 101. Although computer 101 is not shown in the cloud in Figure 1, it may be located in the cloud. On the other hand, computer 101 is not required to be located in the cloud, except to any extent that can be definitively shown.

[0064] The processor set 110 includes one or more computer processors of any type currently known or to be developed in the future. The processing circuitry 120 may be distributed across multiple packages, for example, multiple interconnected integrated circuit chips. The processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for high-speed access by threads or cores running on the processor set 110. The cache memory is typically organized into multiple levels depending on its relative proximity to the processing circuitry. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 110 may operate using qubits and be designed to perform quantum computing.

[0065] Computer-readable program instructions are typically loaded onto computer 101 to implement a computer implementation method by having the processor set 110 of computer 101 execute a series of operational steps, and as a result, the instructions thus executed instantiate the method specified in the flowchart and / or description of the computer implementation method contained herein (collectively referred to as the “Method of the Invention”). These computer-readable program instructions are stored in various types of computer-readable storage media, such as the cache 121 and other storage media considered below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the Method of the Invention. In the computing environment 100, at least some of the instructions for executing the Method of the Invention may be stored in code 200 in persistent storage 113.

[0066] The communication fabric 111 is a signal conduction path that enables various components of the computer 101 to communicate with one another. Typically, this fabric is made up of switches and conduction paths, such as buses, bridges, physical input / output ports, and similar components. Other types of signal communication paths may be used, such as fiber optic communication paths and / or wireless communication paths.

[0067] The volatile memory 112 is any type of volatile memory currently known or to be developed in the future. Examples include dynamic random-access memory (RAM) or static RAM. Typically, the volatile memory 112 is characterized by random access, but this is not required unless explicitly stated. In computer 101, the volatile memory 112 is located in a single package and resides inside computer 101, but alternatively or additionally, the volatile memory may be distributed across multiple packages and / or located externally to computer 101.

[0068] Persistent storage 113 is any form of non-volatile storage for a computer, currently known or to be developed in the future. Non-volatility of this storage means that stored data is maintained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage 113. Persistent storage 113 may be read-only memory (ROM), but typically at least a portion of the persistent storage allows for writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 122 may take several forms, such as various known proprietary operating systems employing a kernel or open-source portable operating system interface type operating systems. Code 200 typically includes at least some computer code involved in performing the method of the present invention.

[0069] The peripheral device set 114 includes a set of peripheral devices for the computer 101. Data communication connections between the computer 101's peripheral devices and other components may be implemented in various ways, such as Bluetooth® connections, Near-Field Communication (NFC) connections, connections made by cables (such as universal serial bus (USB) type cables), insert-type connections (e.g., secure digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the internet. In various embodiments, the UI device set 123 may include components such as a display screen, speaker, microphone, wearable devices (such as goggles and smartwatches), keyboard, mouse, printer, touchpad, game controller, and haptic devices. Storage 124 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing memory device for storing data in the form of qubits. In embodiments where computer 101 requires a large amount of storage (for example, when computer 101 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and the other may be a motion detector.

[0070] The network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for transmitting data over the internet. In some embodiments, the network control and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN)), the control and forwarding functions of the network module 115 are performed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for carrying out the method of the present invention can typically be downloaded from an external computer or external storage device to computer 101 via a network adapter card or network interface included in the network module 115.

[0071] WAN102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances by any currently known or future-developed technology for transmitting computer data. In some embodiments, WAN102 may be replaced and / or complemented by a local area network (LAN), such as a Wi-Fi network, designed to transmit data between devices located in a local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers.

[0072] An end-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 101) and can take any of the forms considered above in relation to computer 101. Typically, EUD 103 receives useful and valuable data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to an end-user, these recommendations would typically be transmitted from computer 101's network module 115 to EUD 103 via WAN 102. Thus, EUD 103 can display or otherwise present recommendations to the end-user. In some embodiments, EUD 103 may be a client device such as a thin client, heavy client, mainframe computer, and desktop computer.

[0073] The remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. The remote server 104 may be controlled and used by the same entity that operates computer 101. The remote server 104 represents a machine that collects and stores useful and valuable data for use by other computers, such as computer 101. For example, in a hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from the remote database 130 of the remote server 104.

[0074] The public cloud 105 is any computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computer functions, particularly data storage (cloud storage) and computing power, without direct active management by the user. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct and active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that make up the host physical machine set 142, which is a universe of physical computers that reside within and / or are available to the public cloud 105. Virtual computing environments (VCEs) typically take the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. These VCEs can be stored as images and transferred between and between various physical machine hosts, either as images or after VCE instantiation. The cloud orchestration module 141 manages the transfer and storage of images, deploys new VCE instantiations, and manages the active instantiation of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate over the WAN 102.

[0075] Here, some further explanation of virtualized computing environments (VCEs) is provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel allows for the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave like actual computers in terms of the programs running within them. Computer programs running on a normal operating system can utilize all of that computer's resources, including connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and the devices allocated to the container; this functionality is known as containerization.

[0076] The private cloud 106 is similar to the public cloud 105, except that its computing resources are available only for use by a single enterprise. While the private cloud 106 is shown as being in communication with the WAN 102, in other embodiments, the private cloud may be completely isolated from the internet and accessible only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. Each of the multiple clouds remains a separate discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple configured clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud. [Examples of methods for identifying and naming UI elements]

[0077] Refer to Figure 2, which illustrates a computer implementation method for identifying UI elements within an application page. Generally speaking, UI refers to the space in which users and computer systems interact, specifically through the use of input devices and software. UI can take various forms, including graphical, voice-controlled, and gesture-based.

[0078] An application page refers to a document accessed by an application. A document can be written in a language that represents the user interface. Hypertext Markup Language (HTML) is a standard markup language or document designed to be displayed within a web browser. Extensible Markup Language (XML) is a markup language and file format for storing, transmitting, and reconstructing arbitrary data.

[0079] A page contains UI elements. Standard examples of graphical user interface (GUI) UI elements include buttons, charts, checkboxes, dropdowns, feeds, forms, icons, input fields, loaders, and modals. Other examples include custom UI elements.

[0080] In block 210, the application's page is accessed. For example, an application such as a web browser running on computer 101 may download an HTML file from a remote location (e.g., a web server running on remote server 104) or from local storage (e.g., persistent storage 113 or private cloud 106). The UI elements of the page are displayed by the application.

[0081] It may be desirable to obtain information about the UI within the displayed page. For example, this information may be used by UI automation tools, such as automation agents, to select the correct UI elements for UI automation.

[0082] In block 215, computer 101 executes code 200 in Figure 1 to identify and name UI elements within the page. Code 200 may be executed from a web browser as a plug-in or extension, and it may be executed from another type of application via an application programming interface (API).

[0083] In block 220, computer 101 loads a set of heuristic rules. The heuristic rules may include global rules and application-specific rules. Global rules are configured to identify UI elements across different applications. The set of rules may be stored in a database, such as remote database 130. The set of rules may include a first subset for identifying UI elements, a second subset for detecting groups of UI elements, and a third subset for detecting modal UI elements.

[0084] In block 225, computer 101 applies a set of heuristic rules to the page to identify UI elements within the page. As used herein, applying rules to a page includes applying a set of rules directly to the page and applying a set of rules to the page's document object model (DOM). When a web page is loaded, the browser creates the page's DOM, which is a hierarchical, tree-like structure that organizes the page's elements as objects. A set of rules can be applied to each object in the DOM. Examples of rules are provided below.

[0085] In block 230, computer 101 displays a page containing the UI. Computer 101 visually highlights each UI element identified by the set of rules. For example, a bounding box of a specific color may be formed around an identified UI element. UI elements not identified by the set of rules are not highlighted.

[0086] Computer 101 also provides information about identified UI elements on the displayed page. For example, a label and other text describing each identified UI element may be displayed next to the bounding box of that UI element.

[0087] When computer 101 displays a page, it may also provide a tool that allows users to perform a quality assessment. The tool allows one or more users to review the displayed page to ensure that all UI elements are correctly identified and described.

[0088] In block 235, one or more users generate feedback on any unidentified UI elements and inaccurate text. If a UI element is not highlighted, the tool allows the user to mark the unidentified element. If the text is incorrect (for example, a UI element is mislabeled), the tool allows the text to be corrected.

[0089] In some cases, feedback may be generated by another computer, such as an end-user device 103. A screenshot is sent to the end-user device 103, and the end-user device 103 sends user feedback to computer 101.

[0090] In block 240, the computer 101 receives user feedback. As used herein, the term “receive user feedback” includes the computer 101 receiving feedback from one or more users. The term also includes receiving feedback from external sources, such as an end-user device 103.

[0091] Feedback can be used to update the set of heuristic rules. If a UI element is not identified, the heuristic rule is updated to identify that UI element. The following paragraphs describe no-code techniques for updating the set of heuristic rules.

[0092] No-code methods involve the use of pipelines to large-scale language models (LLMs). LLMs learn statistical relationships from vast amounts of text during computationally intensive self-supervised and semi-supervised training. LLMs can be used for text generation by taking input text and iteratively predicting the next token or word.

[0093] As used herein, an LLM pipeline refers to a series of steps or processes that use an LLM to complete a task or a set of tasks. This pipeline often involves multiple stages, such as data preprocessing, model inference, and postprocessing. For example, in a text summarization pipeline, the steps may include cleaning and tokenizing the input text, feeding it to the LLM to generate a summary, and then refining the output to ensure coherence and readability.

[0094] In block 245, computer 101 uses the first pipeline of the LLM to create new rules for each UI element that is not visually highlighted. That is, the LLM creates new rules for identifying UI elements that were not identified in block 225. The LLM may be accessible via a public cloud 105 or a private cloud 106.

[0095] User feedback may also include information that allows computer 101 to find unidentified UI elements on the page. For example, the feedback may describe the function of a UI element, its style, and its context (e.g., its position in a list, neighboring UI elements). Once a UI element is found, its attributes may be added to the first pipeline. An example is provided below.

[0096] In block 250, each new rule is tested to see if it identifies a UI element that was not previously identified. If a new rule does not accurately identify a UI element that was not previously identified, control is returned to block 235, and the new rule is iteratively tested and adjusted until its corresponding UI element is accurately identified. If a new rule identifies a UI element that has also been identified by other rules in the set of heuristic rules, a conflict occurs, and the new rule is iteratively tested and adjusted until the conflict is avoided. If a new rule is tested as part of UI automation and does not work correctly, the new rule is iteratively tested and adjusted until it works correctly.

[0097] In block 255, computer 101 uses the second pipeline of the LLM to propose a name for each UI element identified by the new rules. The attributes and context of the newly identified UI elements may be provided on the second pipeline along with another set of naming rules. The naming is described in detail below.

[0098] In block 260, when the scope and precision of a given application page become acceptable, computer 101 updates the set of heuristic rules with any new rules. The updated set of heuristic rules may be stored so that it can be used by other applications and other users.

[0099] The method in Figure 2 offers several advantages compared to relying solely on LLM to identify UI elements within an application's page. Heuristic rules are deterministic. Unlike LLM, there is no randomness involved in identifying UI elements. Heuristic rules always produce the same output from a given starting condition or initial state. They ensure consistent, reliable, and predictable behavior in automated tasks and reduce the risk of errors and inconsistencies that can arise from the more flexible but non-deterministic nature of language models.

[0100] Because the rules are heuristic, they are more flexible and can encompass a wide range of UI elements. Heuristic rules can also be adapted to new UI elements.

[0101] Even if LLMs are used to generate new rules, iterative testing and updating of new rules overcomes the problems inherent in the probabilistic nature of LLMs. Even if LLMs occasionally "hallucinate," iterative processes and user feedback can identify any hallucination.

[0102] The no-code approach to updating a set of heuristic rules has its own advantages. Users don't need to know how to manipulate pages or the DOM. Furthermore, users don't need to know the syntax of the rule files. The method allows non-technical business users to generate accurate and secure heuristic rules for UI automation. [Another illustrative method for identifying UI elements and updating rules]

[0103] Next, refer to Figure 3, which shows a specific example of a computer implementation method for identifying and naming UI elements within a web browser page. In block 310, the set of heuristic rules is initialized by loading a file containing the set of heuristic rules. The file may conform to the YAML format, a format commonly used for configuration files where data is stored or transmitted. The file may contain a list of objects, where each object represents a heuristic rule.

[0104] Each rule can have a type. Examples of types include "normal" rules, "group" rules, "context" rules, and "other" rules. Normal rules can target UI elements such as buttons and inputs. Group rules can target group UI elements such as formats. Context rules can target the context being detected. Other rules can include custom rules, extensions to normal and group rules, etc.

[0105] Each rule has a selector. For example, a selector may be a Cascading Style Sheets (CSS) selector. CSS is a stylesheet language that allows for the separation of presentation style and content. CSS selectors declare which parts of the markup the style applies to by matching tags and attributes in the markup itself. For the purposes of this specification, CSS selectors declare which UI elements the rule applies to by matching tags and attributes in the page itself.

[0106] Each rule has a highlighting function. For example, the highlighting function may be an indication of a specific color for the user to see on the screen.

[0107] See Figure 4 for additional reference, which provides an example of the structure of rule file 410. Four rules are shown, each defined by name, selector, color, and type. The first rule, the Other rule, defines the area of ​​interest (dialog) in HTML. The second rule, the Group rule, defines the hierarchy (format) of elements. The third rule, the Normal rule, selects a text field according to its type and its CSS selector. The fourth rule, also a Normal rule, selects a checkbox according to its type and its CSS selector.

[0108] Returning to Figure 3, in block 320, the page is parsed by applying rule selectors to the web page's DOM. Context rules for identifying modals can be applied to identify any modal. Different normal rules can identify different UI elements. Other context rules can be applied to determine different contexts for each identified UI element. Group rules can be applied to determine whether any of the identified UI elements are part of a group. For example, a group element may identify a grandchild in a nested topology.

[0109] Other rules may be applied to determine whether any of the identified UI elements are stateful. For example, a button may have two states: on and off. An attachment may have two states: loaded and unloaded. Other rules may determine the state, or user feedback may indicate the state.

[0110] In block 330, the web page and the analysis results are displayed. The results include bounding boxes with colors specified by rules drawn around each identified UI element. Each bounding box may be accompanied by a unique identifier and nested text output representing the understanding of the bounded UI element.

[0111] See Figure 5 for further reference, which provides an example of a screen 500 displaying the analysis results. Screen 500 displays a title bar 505, the first, second, third, and fourth input fields 510, 515, 520, and 525, and buttons 530 and 535. The analysis results identify the title bar 505, the four input fields 510, 515, 520, and 525, and buttons 530 and 535. Bounding boxes 506, 511, 521, 526, 531, and 536 are drawn around their respective UI elements 505, 510, 520, 525, 530, and 535. Text indicating the type of their UI elements ("Title", "Input", and "Button") is added. Labels associated with the first, third, and fourth input fields 510, 520, and 525 ("Name", "City", and "State") are also identified and text ("Label") is added.

[0112] The second input field 515 is not identified by the analysis. The bounding box is not drawn around the second input field 515.

[0113] Returning to Figure 3, in block 340, the analysis evaluation is performed. By viewing the display, the user can evaluate the quality of the analysis and identify issues with scope, missing names, duplicates, and conflicting rules. For example, the user can evaluate whether the bounding boxes, their colors, and their tags are accurate. The user can also select a specific UI element, highlight it, and view its detailed metadata.

[0114] In block 350, unidentified UI elements are marked, and inaccurate text is corrected. The user can select an unidentified UI element. For example, the user can click on the selector tool. When the user moves the mouse cursor over a UI element, a new bounding box is drawn over the unidentified UI element.

[0115] UI elements within the new bounding box are named. In some embodiments, the display may allow the user to determine and input the name of an unidentified UI element. However, in the example in Figure 3, the second pipeline of the LLM is used to provide the name.

[0116] In block 360, a prompt is sent to the LLM. A one-shot prompt may contain a command after a one-shot input. The input may have the following canonical form: [Style / State Modifier][Element Name][Element Type][Anchoring Reference]. Anchoring references may include one or more of the following: relative position (e.g., next to a search field); context-based reference (e.g., another UI element on the same line); positional reference (e.g., the second item in a list); nesting / hierarchy (e.g., within a form); or internal text (e.g., "submit").

[0117] In block 370, LLM responds to prompts by discovering the semantic meanings of one or more UI elements. These meanings are treated as suggestions. In some cases, only a single suggestion is displayed, and the user may validate the single suggestion or enter a different name. In other cases, multiple different suggestions are displayed, for example, in the form of a prioritized list. The user may select the best suggestion or enter a different name.

[0118] See Figure 6 for further reference, which shows a simple example of a prompt 610 for suggesting a name for a blue login button. Prompt 610 includes an instruction 620 and a one-shot input 630 for suggesting a name for a blue login button. LLM provides an output 640 that provides three suggestions. All three suggestions include the name of the UI element ("button"). The first suggestion modifies the name by adding a style ("blue"). The second suggestion modifies the name by adding internal text ("login"). The third suggestion modifies the name by adding both a style and internal text.

[0119] In block 375, the state of a stateful UI element can be specified. For example, the UI element "Attachment" can have a loaded state or an unloaded state. Display may allow the user to specify the state of the attachment.

[0120] In block 380, computer 101 uses a first pipeline to create CSS selectors for each UI element within a new bounding box. Once the new bounding box is created, the attributes of the UI elements are accessed and used to form a prompt. A one-shot prompt may include a one-shot input containing attributes or code that define the UI elements, following an instruction specifying the creation of CSS selectors.

[0121] See Figure 7 for an example of a prompt 710 for creating a CSS selector for the "Submit" button. The prompt 710 includes an instruction 720 specifying the creation of a CSS selector, and a one-shot input 730 containing HTML language that defines the UI element.

[0122] LLM responds to the prompt by providing a one-shot output 740. The one-shot output specifies a new rule containing three CSS selectors.

[0123] In block 390, each new rule is tested and validated. Testing may include determining whether the new rule conflicts with any existing rule. A conflict occurs if the new rule captures any UI element that was identified by an existing rule. If a conflict occurs, the second pipeline can update the new rule to make it more specific. Control is returned to block 340, where the updated rule is then tested again.

[0124] The testing may further include using the results of the classification in the UI automation and ensuring that the desired outcome is achieved. If the desired outcome is not achieved, control is returned to block 340.

[0125] In block 395, rule management is performed. When a new rule is validated, the validated rule is saved. The user may decide whether to store the validated rule in the global rule file or in a local rule file. [Conclusion]

[0126] While various embodiments of this teaching have been presented for illustrative purposes, they are not intended to be exhaustive or to limit the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best describe the principles of the embodiments, their practical applications, or technical improvements to the art available on the market, or to enable other those skilled in the art to understand the embodiments disclosed herein.

[0127] While the above describes what is considered to be the best-case and / or other examples, it should be understood that various modifications are possible therein, that the subject matter disclosed herein can be implemented in various forms and examples, and that the teachings can be applied to numerous uses, with only a portion of those uses described herein. Any use, modification, and variation that falls within the true scope of these teachings is intended to be claimed by the following claims.

[0128] The components, stages, features, objects, benefits, and advantages discussed herein are for illustrative purposes only. None of them, nor any related discussions, are intended to limit the scope of protection. While various advantages have been discussed herein, it should be understood that not all embodiments necessarily include all of these advantages. Unless otherwise specified, all measurements, values, ratings, locations, sizes, dimensions, and other specifications described herein, including the following claims, are approximate and not precise. They are intended to be within a reasonable range consistent with the functions to which they relate and those that are customary in the art to which they relate.

[0129] Many other embodiments are also contemplated. These include embodiments having fewer, additional, and / or different components, stages, features, objects, benefits, and advantages. These also include embodiments in which the components and / or stages are configured and / or ordered differently.

[0130] While the above has been explained in conjunction with exemplary embodiments, it should be understood that the term “exemplary” means merely an example, not the best or optimal. Except as stated immediately above, nothing described or illustrated, whether or not it is included in the claims, is intended to provide to the public any component, stage, feature, subject matter, benefit, advantage, or equivalent.

[0131] The terms and expressions used herein shall be understood to have the ordinary meanings given to such terms and expressions in their respective fields of study and research, unless otherwise stated herein for a specific meaning. Relative terms such as 1, 2, and similar terms may be used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual relationship or order between those entities or actions. The terms “comprises,” “comprising,” or any other variation thereof are intended to be of non-exclusive inclusion, such that a process, method, article, or apparatus containing a list of elements may contain not only those elements, but other elements not expressly described or specific to such process, method, article, or apparatus. The element preceded by “a” or “an” does not, unless further constraints apply, exclude the presence of additional identical elements in a process, method, article, or apparatus containing that element.

[0132] The descriptions of behaviors / functions presented herein, while understandable to the human mind, are not abstract ideas of behaviors / functions separate from their computational implementation. Rather, behaviors / functions represent specifications for a properly configured computing device. As will be discussed in detail below, the language of behaviors / functions should be interpreted in their appropriate technical context, namely as concrete specifications for their physical implementation.

[0133] It should be understood that the teachings in this specification may exceed the capabilities of the human mind. It should also be understood that various embodiments of the disclosure described herein may include information that is impossible for entities such as human users to obtain manually. For example, the type, amount, and / or nature of information involved in performing the processes considered herein may be more complex than information that a human user can easily process manually.

[0134] Exemplary embodiments are described in relation to a particular type of machine. Exemplary embodiments are also described, merely as examples, in relation to other scenes, objects, measurements, devices, data processing systems, environments, components, and applications. Any specific representation of these and other similar works is not intended to limit this disclosure. Any preferred representation of these and other similar works may be selected within the scope of the exemplary embodiments.

[0135] Furthermore, exemplary embodiments may be implemented with respect to any type of data, data source, or access to a data source via a data network. Any type of data storage device may, within the scope of the disclosure, provide data to one embodiment of the disclosure, either locally in a data processing system or via a data network. If an embodiment is described using a mobile device, any type of data storage device suitable for use with a mobile device may, within the scope of the exemplary embodiment, provide data to such embodiment, either locally in the mobile device or via a data network.

[0136] An abstract of this disclosure is provided to enable readers to quickly confirm the nature of the technical disclosure. The abstract is submitted with the understanding that it is not to be used to interpret or limit the scope or meaning of the claims. In addition, it is found that in the modes for carrying out the invention described herein, various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments have more features than those explicitly described in each claim. Rather, as the following claims represent, the subject matter of the invention lies in fewer features than all the features of the single embodiment disclosed. Accordingly, the following claims are incorporated herein into the modes for carrying out the invention, and each claim stands alone as separately claimed subject matter.

Claims

1. The step of applying a set of heuristic rules to the pages of an application to identify multiple user interface (UI) elements within the pages; A step of displaying the page, which includes a step of visually highlighting each of the plurality of UI elements identified by the set of heuristic rules; The step of receiving user feedback identifying each of the plurality of UI elements that are not visually highlighted; and The first step involves using the Large-Scale Language Learning Model (LLM) pipeline to create new rules for identifying each UI element that is not visually highlighted. A computer implementation method comprising the above.

2. The aforementioned set of heuristic rules is: A first subset for identifying individual UI elements; A second subset for detecting UI element groups; and A third subset for detecting UI element context A computer implementation method according to claim 1, comprising:

3. The aforementioned highlighting step is, The step of creating a bounding box around each UI element identified by the set of heuristic rules; and The stage where text describing each UI element within the bounding box is displayed. A computer implementation method according to claim 1 or 2, comprising:

4. The computer implementation method according to claim 1 or 2, further comprising the step of iteratively testing and adjusting each new rule until its corresponding UI element is accurately identified.

5. Regarding the given new rules, The step of using the first pipeline includes the steps of creating a prompt for the given new rule, and sending the prompt to the LLM, The iterative testing and adjustment step includes adjusting the prompts with user feedback until the given new rule accurately identifies its corresponding UI element. The computer implementation method according to claim 4.

6. The computer implementation method according to claim 5, wherein the prompt for the given new rule comprises an instruction for creating a selector and attributes of the corresponding UI element.

7. A step of using the second pipeline of the LLM to propose a name for a given UI element that is not visually highlighted; and In order to select from the aforementioned names, we will proceed to the next step of obtaining further user feedback. The computer implementation method according to claim 1 or 2, further comprising:

8. Using the second pipeline comprises the steps of creating a prompt for the given UI element and sending the prompt to the LLM; The prompt includes instructions for selecting multiple names and one-shot input; The aforementioned one-shot input has format [style / state modifier] [name] [type] [anchoring reference], Here, the style, state, name, and type are attributes of the given UI element, and the anchoring reference describes the context of the given UI element. The computer implementation method according to claim 7.

9. The stage of identifying the state of a stateful UI element; A step of displaying the identified state; and A stage to obtain additional user feedback to update any displayed state. The computer implementation method according to claim 1 or 2, further comprising:

10. The computer implementation method according to claim 1 or 2, further comprising the step of updating the set with new rules created by the LLM.

11. A computer system having a memory with computer-readable instructions; and the computer system, Procedure for running an application that displays a page including a user interface (UI); A procedure for identifying multiple UI elements by applying a set of heuristic rules to the aforementioned page; A procedure for displaying the page containing the aforementioned plurality of UI elements; A procedure for visually highlighting each of the plurality of UI elements identified by the set of heuristic rules; A procedure for receiving user feedback identifying each of the aforementioned UI elements that are not visually highlighted; and A procedure for creating new rules to identify each UI element that is not visually highlighted, using the first pipeline of a Large-Scale Language Learning Model (LLM); One or more processors for executing the computer-readable instructions configured to do so A computer system equipped with the following features.

12. The computer system according to claim 11, wherein, when the computer-readable instruction is executed, the one or more processors are further configured to iteratively test and adjust each new rule until its corresponding UI element is accurately identified.

13. Regarding the given new rules, A prompt for the given new rule is created and sent to the LLM. The prompt is iteratively adjusted with user feedback until the given new rule accurately identifies its corresponding UI element. The computer system according to claim 12.

14. The computer system according to claim 13, wherein the prompt for the given new rule comprises an instruction for creating a selector and attributes of the corresponding UI element.

15. The computer system according to any one of claims 11 to 14, wherein, when the computer-readable instruction is executed, the one or more processors are further configured to propose names for a given UI element that is not visually highlighted using a second pipeline of the LLM.

16. The processor set includes, A procedure for applying a set of heuristic rules to pages within an application to identify multiple user interface (UI) elements within said pages; Steps to display the aforementioned page; A procedure for visually highlighting each of the plurality of UI elements identified by the set of heuristic rules; A procedure for receiving user feedback identifying each of the aforementioned UI elements that are not visually highlighted; and A procedure for creating new rules to identify each UI element that is not visually highlighted, using the first pipeline of a Large-Scale Language Learning Model (LLM); A computer program designed to execute something.

17. The computer program according to claim 16, which causes the processor set to further perform a procedure for iteratively testing and adjusting each new rule until its corresponding UI element is accurately identified.

18. A prompt for a given new rule is created and sent to the LLM. The prompt is iteratively adjusted with user feedback until the given new rule accurately identifies its corresponding UI element. The computer program according to claim 17.

19. The computer program according to claim 18, wherein the prompt for the given new rule comprises an instruction for creating a selector and attributes of the corresponding UI element.

20. The computer program according to any one of claims 16 to 19, wherein the set of processors is further configured to perform a procedure of proposing a name for each UI element that is not visually highlighted, using the second pipeline of the LLM.

Images