Automatic execution method, device and system based on interface context representation information

Abstract

The present disclosure relates to a method, device and system for automatic execution of an interactive interface based on interface context representation information. The method comprises: receiving an execution instruction for a target interactive interface; constructing and maintaining interface context representation information of the target interactive interface, the interface context representation information comprising page fingerprint information for representing interface state and operation history graph information for modeling inter-interface operation flow dependency relationship; determining a control strategy required for executing the execution instruction based on the interface context representation information and current interface state, the control strategy at least comprising a target interface element positioning strategy; and executing an automatic operation on the target interactive interface using the control strategy, and updating the operation history graph information and the page fingerprint information based on the operation result. The present disclosure can utilize context awareness capability and multi-level fault tolerance mechanism to realize adaptive repair of automatic operation when interface structure changes, thereby improving the robustness and maintenance efficiency of automatic execution.

Description

Technical Field

[0001] This disclosure relates to the field of computer software automation technology, specifically to an interactive interface automation execution method, apparatus, and system based on interface context representation information. Background Technology

[0002] With the accelerating pace of software development iterations, user interfaces (UIs) are being updated more and more frequently. In the fields of software testing and process automation, traditional automation scripts typically rely on fixed methods for locating UI elements. For example, existing mainstream automated testing frameworks (such as Selenium and Appium) mainly rely on the Document Object Model (DOM) structure or control tree structure, using XPath or CSS selectors to locate target elements.

[0003] However, this traditional positioning method has significant technical drawbacks. First, with frequently changing UI interfaces, fixed-path-based positioning strategies become ineffective once the front-end code is restructured or element attributes are changed, causing automated scripts to fail to execute. Second, existing tools lack awareness of the operational flow context, typically treating each operation as an independent event and failing to understand the dependencies between pages, making them prone to errors when handling complex cross-page business processes. Furthermore, when script execution fails, existing technologies often lack effective fault tolerance and self-healing mechanisms, requiring manual intervention to repair the script, resulting in high maintenance costs. Summary of the Invention

[0004] In view of this, this disclosure provides an automated execution method, apparatus and system for interactive interfaces based on interface context representation information, in order to solve the problems of existing automated scripts being prone to failure due to interface changes, lacking context awareness capabilities and having high maintenance costs.

[0005] The first aspect of this disclosure provides an automated execution method for an interactive interface based on interface context representation information, comprising the following steps: receiving an execution instruction for a target interactive interface; constructing and maintaining interface context representation information of the target interactive interface, the interface context representation information including page fingerprint information for representing the interface state and operation history graph information for modeling the operation flow dependencies between interfaces; determining a control strategy required to execute the execution instruction based on the interface context representation information and the current interface state, the control strategy including at least a target interface element positioning strategy; and performing automated operations on the target interactive interface using the control strategy, and updating the operation history graph information and page fingerprint information based on the operation results.

[0006] A second aspect of this disclosure provides an automated execution device for an interactive interface based on interface context representation information, comprising: a memory for storing a computer program; and a processor coupled to the memory for executing the computer program to perform the following steps: receiving execution instructions for a target interactive interface; constructing and maintaining interface context representation information of the target interactive interface, the interface context representation information including page fingerprint information for representing the interface state and operation history graph information for modeling the operation flow dependencies between interfaces; determining a control strategy required to execute the execution instructions based on the interface context representation information and the current interface state, the control strategy including at least a target interface element positioning strategy; and performing automated operations on the target interactive interface using the control strategy, and updating the operation history graph information and page fingerprint information based on the operation results.

[0007] A third aspect of this disclosure provides an automated execution system for an interactive interface based on interface context representation information, including a user terminal and a server, with the server communicatively connected to the user terminal. The user terminal is used to: collect multimodal input data for a target interactive interface and send it to the server; and receive control commands from the server and execute automated operations on the target interactive interface. The server is used to: receive multimodal input data from the user terminal and parse it using a multimodal fusion algorithm to generate intent commands; construct and maintain interface context representation information for the target interactive interface, including page fingerprint information representing the interface state and operation history graph information for modeling the operational flow dependencies between interfaces; determine the control strategy required to execute the intent commands based on the interface context representation information and the current interface state uploaded by the user terminal, the control strategy including at least a target interface element location strategy; and generate control commands based on the control strategy and send them to the user terminal, and update the operation history graph information and page fingerprint information according to the operation results fed back by the user terminal.

[0008] Based on the above technical solutions, the interactive interface automated execution method, apparatus, and system provided in this disclosure, which are based on interface context representation information, enable the automated execution system to possess a deep perception capability of interface state and operation flow by constructing and maintaining interface context representation information containing page fingerprint information and operation history graph information. Specifically, page fingerprint information can uniquely identify the interface state, while operation history graph information can model the jump dependencies between interfaces. This allows the system to not only know the current interface environment but also understand the logical connections between operations. On this basis, this disclosure achieves closed-loop control and adaptive evolution of automated operations by determining a control strategy that includes a target interface element positioning strategy and updating the interface context representation information in real time based on the operation results. Especially when facing interface redesign or element attribute changes, the system can dynamically adjust the positioning strategy using context information and automatically update the history record after the operation is successful, thereby achieving script self-healing without manual intervention, effectively reducing the maintenance cost of automated scripts, and improving the robustness and accuracy of the execution process. Attached Figure Description

[0009] Figure 1A This is a schematic diagram of the architecture of an automated execution device for an interactive interface based on interface context representation information according to an embodiment of the present disclosure; Figure 1B This is a schematic diagram of the architecture of an automated execution system for interactive interfaces based on interface context representation information according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating an embodiment of the automated execution method for interactive interfaces based on interface context representation information according to this disclosure. Figure 3 This is a simplified flowchart illustrating the multimodal input parsing process according to an embodiment of the present disclosure; Figure 4 This is a detailed flowchart illustrating the multimodal input parsing process according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram illustrating the construction of interface context representation information according to an embodiment of the present disclosure; Figure 6 This is a flowchart of a session recovery operation according to an embodiment of the present disclosure; Figure 7 This is a flowchart illustrating a multi-level fault-tolerant positioning method according to an embodiment of the present disclosure; Figure 8A This is a schematic diagram illustrating an application example of a cross-page execution path according to an embodiment of the present disclosure; Figure 8B This is a detailed flowchart illustrating the process of determining a cross-page execution path according to an embodiment of the present disclosure; Figure 9 This is a schematic diagram of the dynamic adjustment process of positioning stability score according to an embodiment of the present disclosure; Figure 10 This is a flowchart illustrating real-time environmental monitoring according to an embodiment of the present disclosure; Figure 11 This is a timing diagram of an automated execution device for an interactive interface according to an embodiment of the present disclosure; Figure 12 This is a timing diagram of an automated execution system for an interactive interface according to an embodiment of the present disclosure. Detailed Implementation

[0010] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same component symbols are used in the drawings and description to denote the same or similar parts.

[0011] Reference Figure 1A In one embodiment, this disclosure provides an automated execution device 10 for interactive interfaces based on interface context representation information. The automated execution device 10 includes a processor 110, a storage device 120, an input / output interface 130, and a memory 140. The processor 110 is coupled to the memory 140, the storage device 120, and the input / output interface 130, and is used to execute a computer program stored in the memory 140 to implement an automated execution method for interactive interfaces based on interface context representation information. The memory 140 stores the computer program and interface context representation information, wherein the interface context representation information includes page fingerprint information for representing the interface state and operation history graph information for modeling the operational flow dependencies between interfaces. The storage device 120 persistently stores the interface context representation information to achieve cross-session state persistence. The input / output interface 130 communicates with a target interactive interface, including collecting interface state information of the target interactive interface, receiving execution instructions for the target interactive interface, and sending control instructions to the target interactive interface.

[0012] In another embodiment, the interactive interface automation execution device 10 is an independent computing device, and the processor 110 executes all the steps of the interactive interface automation execution method based on the interface context representation information.

[0013] Specifically, the processor first receives execution instructions for the target interactive interface through an input / output interface. These instructions can originate from various input methods and triggering mechanisms to adapt to different application scenarios and usage requirements. In one embodiment, the execution instructions come from direct user input. Users can input text descriptions of operations via a keyboard or touch interface, such as "click the login button and enter keywords in the search box"; users can also input voice descriptions via a microphone, which the system will convert into structured execution instructions after speech recognition; users can also indicate the operation to be performed by uploading screenshots of the interface or recording operation videos, which the system will extract the operation intent through visual analysis. These various input methods can be used individually or in combination to form multimodal input, thereby providing a more natural and accurate human-computer interaction experience. In another embodiment, the execution instructions come from a preset automated script. This automated script can be a pre-written test case script stored in a storage device, containing a series of operation steps executed in a predetermined order and their parameter configurations. The automated script can also be a batch script used to define a sequence of operations that need to be repeatedly executed. Furthermore, the automated script can be written in a domain-specific language or a general scripting language, which the processor will parse to generate a corresponding sequence of execution instructions.

[0014] In another embodiment, the execution instructions originate from a call request from an external system. The processor receives remote call requests from a continuous integration system, test management platform, or other business systems via an application programming interface (API) and converts the request into an internal execution instruction format. This implementation allows the interactive interface automation execution device to be integrated as a service component into larger-scale software development or business processes. In another embodiment, the execution instructions originate from a task queue or scheduling system. The processor sequentially retrieves task descriptions to be executed from a preset task queue, or initiates specific execution instructions based on timed trigger signals from the scheduling system. This implementation is suitable for automation scenarios that require periodic execution or triggering under specific conditions, such as daily scheduled regression testing or event-triggered monitoring tasks.

[0015] In another embodiment, the execution instructions are generated through a record-playback mechanism. In record mode, the processor captures the user's actual actions on the target interface, including click locations, input content, scroll distances, and operation timing, and converts this information into a repeatable sequence of execution instructions. In playback mode, the processor drives the target interface to reproduce the same operation flow based on the previously recorded sequence of execution instructions.

[0016] In another embodiment, execution instructions are generated based on predefined template instantiation. The processor maintains a set of parameterized operation templates, each defining an operation framework and configurable parameters for a specific business scenario. When an instantiation request containing specific parameter values ​​is received, the processor fills the parameter values ​​into the corresponding template, generating complete execution instructions. For example, a template for a "user registration" scenario can receive parameters such as username, password, and email address, and after instantiation, generates a sequence of execution instructions containing the specific content to be filled in. In another embodiment, execution instructions are extracted and generated by a natural language understanding module from unstructured descriptions. Users can provide broad or colloquial task descriptions, such as "help me complete the purchase process for this order." The processor analyzes the semantics of the description using natural language understanding technology, combines it with interface contextual representation information to infer the specific operation steps to be performed, and automatically generates the corresponding sequence of execution instructions. This implementation lowers the technical threshold for users to write precise operation instructions. The various sources of execution instructions mentioned above are not mutually exclusive; the processor can support the mixed reception of execution instructions from different sources in the same automated execution session and process and execute them according to a unified internal format.

[0017] After receiving the execution instruction, processor 110 reads the stored interface context representation information from memory 140 or storage device 120. Processor 110 constructs and maintains this interface context representation information based on the current interface state, where page fingerprint information is used to uniquely identify the state of each page in the target interactive interface, and operation history graph information is used to record and model the operation flow dependencies between interfaces. Processor 110 then determines the control strategy required to execute the execution instruction based on the interface context representation information and the current interface state. This control strategy includes at least a target interface element positioning strategy. Processor 110 uses this control strategy to perform automated operations on the target interactive interface through input / output interface 130, and updates the operation history graph information and page fingerprint information based on the operation results. The updated interface context representation information is then stored in storage device 120 for persistence.

[0018] In another embodiment, memory 140 and storage device 120 are functionally distinct. Memory 140 is implemented using a volatile storage medium, such as dynamic random access memory or static random access memory, and is used to store working copies of runtime data and interface context representation information required by processor 110 when executing computer programs. Storage device 120 is implemented using a non-volatile storage medium, such as flash memory, solid-state drive, or disk storage device, and is used to persistently store interface context representation information. By using a non-volatile storage medium, storage device 120 can retain interface context representation information after the interactive interface automation execution device 10 is powered off or restarted, thereby achieving cross-session state persistence. When the automation execution session is interrupted due to an unexpected situation, processor 110 can read the previously stored interface context representation information from storage device 120 upon restart and determine the session recovery point based on this information, without having to re-execute the completed operation steps.

[0019] In another embodiment, the input / output interface 130 includes various types of interface modules to support different types of target interactive interfaces and input methods. The input / output interface 130 may include a network interface module for communicating with a web page interface, sending control commands and receiving interface status information via network protocols. The input / output interface 130 may also include an application programming interface (API) module for communicating with an application programming interface and executing automated operations through interface calls provided by the operating system. The input / output interface 130 may also include a multimodal input interface module for receiving input data in multiple modalities, such as voice, text, images, and video, to support the execution of multimodal input methods.

[0020] In another embodiment, the components of the interactive interface automation execution device 10 can be distributed across multiple physical devices. Specifically, the processor 110 and memory 140 can be deployed on a first physical device, the storage device 120 on a second physical device, and the input / output interface 130 on a third physical device. The first, second, and third physical devices communicate with each other via a network connection. In this configuration, when the processor 110 executes a computer program, it reads interface context representation information from the storage device 120 deployed on the second physical device via the network connection and sends control commands to the input / output interface 130 deployed on the third physical device via the network connection. This distributed architecture allows the interactive interface automation execution device 10 to be flexibly expanded, separating computationally intensive tasks from storage-intensive tasks, thereby improving the system's scalability.

[0021] In another embodiment, the processor 110 can be implemented as a combination of multiple processing units, each responsible for different processing tasks. For example, a first processing unit is responsible for parsing and executing instructions and extracting operational intent, a second processing unit is responsible for constructing and maintaining interface context representation information, and a third processing unit is responsible for determining control strategies and generating control instructions. The processing units exchange data via an internal bus or shared memory 140. This multi-processor unit architecture enables the processor 110 to process multiple tasks in parallel, thereby improving the execution efficiency of automated operations.

[0022] In one embodiment, the interactive interface automation execution device 10 may further include a graphics processing unit coupled to the processor 110, used to accelerate visual feature extraction and graph neural network inference computation. The processor 110 is responsible for overall flow control and logical operations, while the graphics processing unit is responsible for computationally intensive visual processing and neural network inference tasks. When constructing page fingerprint information, the graphics processing unit performs structural feature extraction of the interface screenshot to generate a visual hash value, while the processor 110 is responsible for generating semantic vectors and combining and storing page fingerprint information.

[0023] In one embodiment, the automated execution device 10 for the interactive interface can be implemented as an embedded testing device, integrated into the hardware environment of the system under test. In this configuration, the input / output interface 130 is directly connected to the display interface and input interface of the system under test. The processor 110 directly acquires the interface screen of the system under test and injects operation events through the input / output interface 130, without the need for software-level interface calls, thereby enabling the testing of lower-level interface interaction behaviors.

[0024] In another embodiment, the input / output interface 130 includes various types of interface modules to support different types of target interactive interfaces and input methods. The input / output interface 130 may include a network interface module for communicating with a web page interface, sending control commands and receiving interface status information via network protocols. The input / output interface 130 may also include an application programming interface (API) module for communicating with an application programming interface and executing automated operations through interface calls provided by the operating system. The input / output interface 130 may also include a multimodal input interface module for receiving input data in multiple modalities, such as voice, text, images, and video, to support the execution of multimodal input methods.

[0025] In another embodiment, the input / output interface 130 can be implemented in various ways in practice. When the target interactive interface is a web page, the input / output interface 130 can communicate with the browser via the WebDriver protocol. This protocol defines a standardized set of commands for controlling the browser to perform operations such as page navigation, element positioning, and user interaction simulation. When the target interactive interface is a mobile application interface, the input / output interface 130 can communicate with the mobile device to perform touch event injection and interface state acquisition. When the target interactive interface is a desktop application interface, the input / output interface 130 can interact with the application via the Accessibility API or UI Automation API provided by the operating system. The input / output interface 130 may also include a screen capture module for capturing screenshots of the target interactive interface for generating page fingerprint information. This screen capture module can acquire interface image data through a graphics device interface or frame buffer reading.

[0026] In another embodiment, the processor 110 can be implemented in various ways in practice. The processor 110 can be implemented as a Central Processing Unit (CPU), such as a general-purpose processor based on x86 or ARM architecture, used to execute computer programs in memory 140 and coordinate the overall operation of the interactive interface automation execution device 10. When the interactive interface automation execution device 10 needs to perform computationally intensive visual feature extraction or neural network inference tasks, the processor 110 can work in conjunction with a Graphics Processing Unit (GPU), with the processor 110 responsible for flow control and logical operations, and the GPU responsible for parallel computing tasks. The processor 110 can also be implemented as a System on Chip (SoC), integrating the CPU, GPU, neural network acceleration unit, and memory controller onto a single chip to reduce power consumption and improve data transfer efficiency. In cloud deployment scenarios, the processor 110 can be implemented as a virtualized processing resource, dynamically allocating computing resources through a cloud computing platform to handle automation execution tasks of different scales.

[0027] Reference Figure 1BIn one embodiment, this disclosure provides an automated execution system 11 for interactive interfaces based on interface context representation information. The automated execution system 11 includes a server 100 and a user terminal 200, with the server 100 communicatively connected to the user terminal 200. The automated execution system 11 adopts an edge-cloud separation architecture, where the server 100 is responsible for computationally intensive multimodal fusion parsing and context modeling, while the user terminal 200 is responsible for lightweight data acquisition and operation execution. This architecture enables centralized management and dynamic evolution of interface context representation information.

[0028] In another embodiment, the user terminal 200 includes a multimodal data acquisition B210, a target interactive interface B220, and an automated execution module B230. The multimodal data acquisition B210 is used to acquire multimodal input data for the target interactive interface B220, such as... Figure 1B As shown, the input data includes voice, text, images, and video. The target interactive interface B220 refers to the web page or app application to which the automated operation is to be performed, whose underlying structure includes a Web Document Object Model (DOM) or an app control tree. The automation execution module B230 receives control commands and drives the target interactive interface B220 to perform specific operations. In terms of hardware implementation, the user terminal 200 can be a personal computer, mobile device, or embedded test machine, and connects to the server 100 via protocols such as TCP / IP or WebSocket.

[0029] In another embodiment, server 100 includes a multimodal fusion parsing module B110, a context-aware model B120, a control policy generation module B130, and an information update module B140. The multimodal fusion parsing module B110 parses input data to generate intent commands. The context-aware model B120 is the core storage and inference unit of the system, internally maintaining a page fingerprint database and an operation history graph to characterize interface states and model operation flow dependencies. The control policy generation module B130 generates specific control commands based on context information. The information update module B140 processes feedback results and executes adaptive or self-healing logic to update the model.

[0030] In another embodiment, the operation flow of the interactive interface automated execution system 11 is shown by arrows A11 to A17: As shown by arrow A11, the multimodal data acquisition B210 of the user terminal 200 transmits the acquired input data to the multimodal fusion analysis B110 of the server 100.

[0031] As shown by arrow A12, after the multimodal fusion parser B110 parses and generates the intent instruction, it passes it to the context-aware model B120 so that the intent can be understood in the current context.

[0032] As shown by arrow A13, the context-aware model B120 provides the matched interface context representation information to the control strategy generator B130.

[0033] As shown by arrow A14, the control strategy generator B130 generates control commands based on the current state and sends them to the automation execution module B230 of the user terminal 200.

[0034] As shown by arrow A15, the automated execution module B230 performs operations (such as clicking or inputting) on ​​the target interactive interface B220 according to instructions.

[0035] As shown by arrow A16, the results and feedback after the operation (including the new interface status) are sent back to the information update module B140 of the server 100.

[0036] As shown by arrow A17, the information update module B140 updates the page fingerprint database and operation history graph in the context-aware model B120 based on the feedback results, thereby achieving a closed loop for updating representation information.

[0037] In one embodiment, server 100 supports a multi-terminal collaborative learning mode. When multiple user terminals 200 perform automated execution on the same target interactive interface B220 (e.g., different test devices of the same app), the information update module B140 of server 100 can aggregate the operation history and page fingerprints from different terminals to construct a globally shared context-aware model B120. This means that a new page or path explored by one terminal can be immediately reused by other terminals, thereby improving the overall automation coverage efficiency and path planning capability.

[0038] In another embodiment, Figure 1B The multimodal data acquisition B210, target interaction interface B220, and automated execution module B230 in the user terminal 200, and the multimodal fusion parsing B110, context-aware model B120, control strategy generation B130, and information update module B140 in the server 100, are not limited to the two-end separate configuration shown in the figure in terms of physical deployment. The above functional modules can be distributed in multiple independent devices according to actual application needs, or they can be centrally deployed in a single computing device. Regardless of the deployment configuration, the data interaction and processing logic between the modules follow the overall process shown by arrows A11 to A17, that is, sequentially completing the closed-loop processing of input data acquisition, multimodal fusion parsing, context-aware matching, control strategy generation, automated operation execution, and representation information update.

[0039] Reference Figure 2 In one embodiment, this disclosure provides a method for automated execution of an interactive interface based on interface context representation information. This method comprises... Figure 1B The method is executed by the server 100 shown, specifically by the processor 110 in the server 100 calling a computer program stored in the memory 140. The method includes steps S210 to S250.

[0040] In step S210, the processor 110 receives execution instructions for the target interactive interface. Specifically, the processor 110 receives multimodal input data from the user terminal 200 through the input / output interface 130, and uses a multimodal fusion algorithm to parse the multimodal input data to generate execution instructions containing action types and descriptions of target interface elements. These execution instructions specify the specific operation the user expects to perform on the target interactive interface, such as "clicking the login button" or "entering keywords in the search box." In some cases, the execution instructions may also come directly from a preset automated test script or task queue; the processor 110 reads the script from the storage device 120 and parses out the corresponding execution instructions.

[0041] In step S220, the processor 110 constructs and maintains interface context representation information for the target interactive interface. This interface context representation information includes page fingerprint information representing the interface state and operation history graph information for modeling the operational flow dependencies between interfaces. Specifically, the processor 110 generates a visual hash value based on a screenshot of the target interactive interface and generates a semantic vector based on the interface's Uniform Resource Locator (URL) and title. The visual hash value and semantic vector are combined to generate a page fingerprint information that uniquely identifies the current interface state. Simultaneously, the processor 110 constructs graph nodes based on the page fingerprint information and builds directed connections between different graph nodes based on user operations, thereby generating operation history graph information. This operation history graph information is modeled using a graph neural network, enabling dynamic recording and deduction of the jump logic and business process dependencies between interfaces.

[0042] In step S230, the processor 110 determines the control strategy required to execute the execution instruction based on the interface context representation information and the current interface state. This control strategy includes at least a target interface element location strategy. Specifically, the processor 110 first obtains the current interface state uploaded by the user terminal 200, calculates the page fingerprint information of the current interface, and locates the current node position in the operation history graph information. Subsequently, the processor 110 generates a multi-level fault-tolerant location strategy for the target interface element based on the target interface element description in the execution instruction, combined with the historical location data recorded in the operation history graph information and the structural information of the current interface. This control strategy may also include cross-page execution path planning and session recovery point determination strategies to cope with complex cross-page business processes or interruption recovery scenarios.

[0043] In step S240, the processor 110 uses the control strategy to perform automated operations on the target interactive interface. Specifically, the processor 110 converts the generated control strategy into specific control instructions and sends them to the user terminal 200 through the input / output interface 130. After receiving the control instructions, the user terminal 200 drives the automated execution module B230 to perform corresponding click, input, or swipe operations on the target interactive interface. During this process, the processor 110 continuously monitors the execution status of the operation. If a positioning failure or execution abnormality is encountered, a retry or rollback process is triggered according to the fault tolerance mechanism in the control strategy.

[0044] Specifically, in the step of generating control instructions, processor 110 performs a mapping transformation from abstract business logic to the underlying driver protocol. Specifically, processor 110 calls the corresponding automation driver interface or underlying input event library based on the type of the target interactive interface (such as a webpage, native application, or hybrid application). Processor 110 converts the "action type" in the execution instructions into standard operation commands recognizable by the target system (such as DOM event triggering, control API calls, or hardware input event simulation), and converts the "target interface element description" into specific positioning parameters (such as element selector paths, absolute screen coordinates, or relative percentage coordinates). This conversion mechanism ensures that the system can shield the underlying technical differences of different interactive interfaces, driving web browser kernels, mobile operating system frameworks, or desktop window managers to perform automated operations through a unified control instruction format.

[0045] In step S250, the processor 110 updates the operation history graph information and page fingerprint information based on the operation result. Specifically, after the automated operation is successfully executed, the processor 110 receives the operation result and the new interface state after the operation from the user terminal 200. The processor 110 calculates new page fingerprint information based on the new interface state and determines whether a page jump has occurred. If a jump occurs, the processor 110 adds new graph nodes or updates the connection relationships of existing nodes in the operation history graph information, and stores the attribute features of this operation (such as click coordinates and element paths) in the corresponding directed connection relationships. Through this closed-loop update mechanism, the interface context representation information can continuously evolve and improve itself during the automated execution process, thereby improving the accuracy and robustness of subsequent executions.

[0046] Reference Figure 3 In one embodiment, the processor 110 executes a multimodal input parsing process, which transforms multimodal input data into machine-understandable execution instructions through parallel feature extraction and subsequent fusion processing.

[0047] Specifically, as described in block B321, processor 110 performs intent and entity extraction operations, processing the input data from the speech or text modality of block B311. In this process, processor 110 first converts the speech signal into text, then uses natural language processing techniques to perform dependency parsing and named entity recognition on the text, extracting the operational intent representing the action type and the target interface element description information representing interface features. As shown by arrow A32, the extracted intent and entity information are passed to the subsequent verification stage.

[0048] Simultaneously, as described in block B322, processor 110 performs visual feature extraction operations, analyzing the input data from the image or video modality of block B312. Processor 110 identifies the visual features and region coordinates of each candidate element in the image, and performs temporal change detection on the video data to identify potential page jump points. Furthermore, processor 110 parses the mouse cursor movement trajectory and hover events in the video to capture the user's attention behavior towards specific interface elements at specific points in time. As shown by arrow A34, the extracted visual features and trajectory information are transmitted synchronously.

[0049] Next, as described in block B330, processor 110 performs a spatiotemporal consistency verification operation, aligning the feature information from blocks B321 and B322. Processor 110 acquires the input timestamp of the voice or text, retrieves a video frame or image matching that timestamp within a preset time window, and spatially maps the mouse trajectory coordinates or hover events in the video with the target interface element description in the input data. Through this verification operation, processor 110 can confirm the correlation between the user's voice command and the mouse pointing action, thereby eliminating ambiguity. For example, when the user issues the voice command "click this," processor 110 clarifies the specific interface element referred to by "this" by aligning the mouse hover position.

[0050] Subsequently, as described in block B340, processor 110 performs a multimodal fusion operation, using an attention mechanism to weightedly fuse the features of each modality that have been verified for spatiotemporal consistency. Processor 110 uses the operation intent as the query vector and the candidate element features as the key vector, calculates the matching score between the two, and generates attention weights for each modality accordingly. Processor 110 uses these attention weights to perform weighted summation and nonlinear mapping on the feature vectors, ultimately generating the execution instruction as described in block B350. This execution instruction contains a defined action type and a description of the target interface elements, for use by the subsequent control strategy generation module.

[0051] Reference Figure 4 In one embodiment, the processor 110 executes a detailed process for multimodal input parsing. This process performs corresponding feature extraction processing on input data of four modalities: speech, text, image, and video. After spatiotemporal consistency verification and multimodal feature fusion, an execution instruction B460 is generated. The core of spatiotemporal consistency verification is to establish the correspondence between different modal input data in the time and spatial dimensions.

[0052] In terms of time dimension, "time" refers to the moment when each modal input occurs, such as the timestamp of the user issuing a voice command, the timestamp of the user submitting text input, and the timestamps corresponding to each frame in the video stream. For example, when the user says "click this" at a certain moment, the processor 110 needs to determine which frame in the video stream corresponds to the voice command in order to know the interface state at the time the user speaks.

[0053] In terms of spatial dimension, "empty" refers to the coordinates of the interface elements and user pointing actions on the screen, such as the hover coordinates of the mouse cursor, the bounding box coordinates of candidate interface elements, and the coordinates of a specific area in a screenshot uploaded by the user. When the user's voice command contains pronouns such as "this," the processor 110 needs to determine which element on the screen the user is referring to.

[0054] By jointly constraining the temporal and spatial dimensions, processor 110 can answer the question, "At the moment the user speaks, which element on the screen is the mouse pointing to?" For example, when processor 110 detects that the user issues a voice command at timestamp T, and the mouse hovers at coordinates (X,Y) for a preset duration within a preset time window before and after timestamp T, and coordinates (X,Y) fall exactly within the bounding box of a candidate interface element, processor 110 can determine that the object referred to in the user's voice command is that candidate interface element. This dual temporal and spatial alignment mechanism enables processor 110 to eliminate ambiguity caused by unclear referencing or vague expression in multimodal input, thereby generating accurate execution instructions.

[0055] Specifically, in one embodiment, as described in block B411, processor 110 receives voice input as input data for a first modality. As shown by arrow A41, the voice input is passed to block B421 for processing. As described in block B421, processor 110 performs speech recognition and natural language processing operations. First, it performs automatic speech recognition processing on the voice input, converting the voice signal into a text sequence. Then, it performs natural language processing on the text sequence to extract the operation intent and target interface element description. Processor 110 also records the input timestamp of the voice input, which is used for subsequent spatiotemporal consistency verification. For example, if a user says "click the login button" into a microphone, processor 110 converts the speech into text and extracts the action intent of "click" and the target element description of the "login button".

[0056] In one embodiment, as described in block B412, processor 110 receives text input as input data for a second modality. As shown by arrow A42, the text input is passed to block B422 for processing. As described in block B422, processor 110 performs dependency parsing and named entity recognition operations, performing dependency parsing (or semantic analysis) on the text input to resolve the grammatical dependencies between words in the text, thereby identifying action predicates and their objects. Processor 110 also performs named entity recognition on the text input, extracting entity information with specific semantics from the text, such as interface element names, input parameter values, or page identifiers. Processor 110 also records the input timestamp of the text input. For example, if a user enters "enter headphones in the search box and click search" in a dialog box, processor 110 parses the two actions "enter" and "click", and identifies "search box" and "search" as target elements and "headphones" as input parameters.

[0057] In one embodiment, as described in block B413, processor 110 receives image input as input data for a third modality. As shown by arrow A43, this image input is passed to block B423 for processing. As described in block B423, processor 110 performs image element recognition and visual feature extraction operations, detecting interface elements in the image input, identifying interactive elements such as buttons, input boxes, links, and icons, and extracting the visual feature vectors of each element. Processor 110 simultaneously calculates the region coordinates of each candidate element in the image, which are represented in the form of bounding boxes, including the coordinates of the top-left corner and the width and height information of the element. For example, if a user uploads a screenshot of an interface containing a shopping cart icon, processor 110 detects the visual features of the shopping cart icon and the region information located at the top-right corner of the interface with coordinates (850, 50) and a size of (40, 40) pixels. In another embodiment, processor 110 can also directly detect the user's image to identify various elements or objects in the image, thereby extracting the corresponding visual features and coordinate / size information.

[0058] In one embodiment, as described in block B414, processor 110 receives video input as input data for a fourth modality. As shown by arrow A44, this video input is passed to block B424 for processing. As described in block B424, processor 110 performs timing change detection and mouse trajectory parsing operations, analyzes the video input frame by frame, detects key interface changes in the video frame sequence, such as changes in Uniform Resource Locators (URLs), abrupt changes in page layout, or the appearance of pop-ups, and marks these change points as potential page jump points. Simultaneously, processor 110 parses the movement trajectory of the mouse cursor in the video, extracts the mouse coordinates at each time point, and identifies hover events where the mouse stays at a specific position for more than a preset time threshold. For example, processor 110 captures the user's operation video stream in real time and detects that at 3.5 seconds of the video stream, the mouse hovers at coordinates (200, 300) for 800 milliseconds, and then a page jump occurs at 5.2 seconds.

[0059] In one embodiment, as shown by arrow A45, feature information from blocks B421, B422, B423, and B424 is converged to block B430. As described in block B430, processor 110 performs a spatiotemporal consistency verification operation, retrieves matching video frames or images within a preset time window, and aligns the mouse trajectory or hover event with the target element description. Specifically, processor 110 obtains the input timestamp of voice or text input and retrieves video frames within the preset time window that match the input timestamp in the video input frame sequence. Processor 110 spatially matches the mouse coordinates in the retrieved video frames with the candidate element region in the image input to determine whether the mouse is pointing to a candidate element. When there is ambiguity in the target interface element description in voice or text, processor 110 determines the interface element actually pointed to by the user based on the mouse hover position, thereby eliminating ambiguity. For example, when a user says "add this to cart" and the mouse hovers over the second item, the processor 110 determines through spatiotemporal alignment that "this" refers to the second item at the mouse hover position, rather than other items.

[0060] Specifically, in one embodiment, when the processor 110 performs the spatiotemporal consistency verification operation of block B430, it calculates the alignment matching degree to quantitatively evaluate the degree of spatiotemporal correspondence between the modal input data. This alignment matching degree comprehensively considers the matching degree of both temporal consistency and spatial consistency.

[0061] Regarding time consistency, processor 110 calculates the time difference between the input timestamp of the voice or text input and the frame timestamp of the retrieved video frame. When this time difference is less than a preset time threshold (e.g., 500 milliseconds), the time consistency condition is satisfied; when the time difference exceeds the time threshold, the time consistency score decreases as the time difference increases. The time consistency score can be expressed as the ratio of the time difference to the time threshold mapped by a decay function, such as using exponential decay or linear decay methods.

[0062] Regarding spatial consistency, processor 110 calculates the degree of spatial overlap between the mouse hover position or trajectory coordinates in the video frame and the bounding boxes of candidate interface elements. The degree of spatial overlap can be calculated based on the intersection-union ratio (IU) or the point-fall-in rate (PFR). The IU is the ratio of the intersection area to the union area of ​​the mouse hover region and the bounding boxes of candidate elements; the PFR is the determination result of whether the mouse coordinates fall within the bounding box of a candidate element. A higher spatial consistency score is achieved when the mouse hover position is completely within the bounding box of a candidate element; a lower spatial consistency score is achieved when the mouse hover position deviates from the bounding boxes of all candidate elements.

[0063] Processor 110 weights and combines the temporal consistency score and spatial consistency score to generate a comprehensive alignment match score. This weighting can be linear, for example, the alignment match score equals the sum of the temporal consistency score multiplied by a first weight coefficient and the spatial consistency score multiplied by a second weight coefficient; or it can be product-based, requiring both temporal and spatial consistency to meet certain conditions to achieve a higher alignment match score. The weight coefficients can be configured according to specific application scenarios. For example, in scenarios with fast-paced user operations, the weight of temporal consistency can be appropriately increased, while in scenarios with densely packed interface elements, the weight of spatial consistency can be appropriately increased.

[0064] More specifically, in one embodiment, the spatiotemporal consistency verification operation of block B430 can be implemented as an alignment mechanism of "query vector-key vector matching". Specifically, processor 110 constructs a query vector Q from the core operation intent extracted from the speech / text modality. The query vector Q can be obtained by concatenating or weighting action type vectors (such as click, input, scroll) and operation object semantic vectors (such as "login button" and "search box"); wherein, the semantic vector can be obtained by encoding the instruction text (or ASR transcribed text) by a pre-trained language model, and can be further combined with dependency parsing and named entity recognition results to explicitly model the "predicate-object" structure to improve the determinism of intent representation.

[0065] In one embodiment, the processor 110 performs interactive element detection on the current screenshot or video keyframe to obtain multiple candidate element regions, and constructs a corresponding key vector for each candidate element. The key vector It may include at least: image feature embeddings of candidate elements (e.g., generated by ViT, YOLO feature heads, or CLIP image encoders), text label features of candidate elements (e.g., generated by OCR-recognized text, control text, accessibility attributes aria-label, or placeholder and encoded by a language model), spatial coordinate features of candidate elements (e.g., coordinates of the top-left corner of the bounding box, width and height, center point coordinates, and relative scale coordinates), and contextual features of candidate elements (e.g., text of neighboring elements, semantics of parent / sibling nodes, DOM path fragments, or encoding of parent-child relationships in the control tree). In some implementations, spatial coordinate features may be mapped to the same dimension as semantic / visual features using positional encoding (e.g., sine positional encoding or learnable embedding vectors) for unified subsequent calculation.

[0066] In one embodiment, processor 110 calculates a matching function. To characterize the query vector Q and each key vector The degree of matching between them. Matching function. The scaled dot product form can be used: ,in The key vector dimension can be used; alternatively, a fusion scoring method can be employed, combining semantic similarity, visual similarity, and spatiotemporal consistency terms in a weighted manner. For example, semantic similarity can be given by text vector cosine similarity or CLIP text-image similarity; visual similarity can be given by candidate element image embedding similarity; and the spatiotemporal consistency term can be determined by the difference between the input timestamp and the candidate frame timestamp, as well as the spatial overlap between the mouse trajectory / hover point and the candidate element bounding box (e.g., IoU or point-fall rate). This can be achieved through... By introducing a spatiotemporal consistency term, the processor 110 can jointly constrain the "time when the user speaks" with the "location of the element the user points to / focuses on", thereby reducing ambiguity in reference.

[0067] In one embodiment, the processor 110 sets a preset threshold condition for spatiotemporal consistency. Specifically, when the difference between the voice / text input timestamp and the current frame timestamp is greater than a time threshold... When the alignment is unreliable (e.g., 500ms by default, configurable) or when the spatial consistency between the mouse hover point and any candidate element is lower than the spatial threshold, the processor 110 determines that the alignment is unreliable and triggers a backtracking retrieval mechanism to expand the alignment range. The backtracking retrieval mechanism can retrieve multiple page snapshots or video frames within a preset time window (e.g., 10 seconds, configurable), determine the most likely corresponding target page snapshot based on page fingerprint similarity, and then re-execute element detection and matching on the target snapshot.

[0068] In one embodiment, to achieve the above-mentioned fallback retrieval, the processor 110 can calculate a page similarity score for each page snapshot. The snapshot with the highest score is selected as the target snapshot. Semantic similarity and structural similarity can be combined for calculation, for example: ,in , This is a semantic vector obtained based on the URL (Uniform Resource Locator) and the title text encoding (e.g., the CLS vector output by a BERT-like model). , This is a screenshot of the page after normalized resolution. SSIM is the structural similarity index; λ is the weight coefficient (e.g., 0.7 by default, configurable within the range of 0.5 to 0.9). Through this page similarity calculation, the system can reliably locate the page state matching the command time point even in scenarios where URL changes are not significant or in single-page application scenarios.

[0069] In one embodiment, the processor 110 may also record the "time deviation factor" (e.g., the difference between the voice timestamp and the final selected frame timestamp) obtained during the backtracking retrieval process, and use it as a training sample or statistic for subsequent alignment model or threshold adaptive update, so as to gradually optimize the alignment accuracy under different user habits and different device latency conditions.

[0070] In one embodiment, during the execution of compound execution instructions, the processor 110 utilizes cross-modal alignment results to trigger and confirm page switching, thereby achieving a closed loop of "completion of the previous operation segment—page switching—binding of the next operation segment." Specifically, when the processor 110 detects the completion of a key event in the first operation segment (e.g., a login button click event returning a successful state), the processor 110 compares the similarity between the current screenshot and the expected page fingerprint execution to confirm whether it has jumped to the target page of the next operation segment. If the comparison result meets a threshold condition, the page switching is determined to be successful, and the next operation segment is bound to the corresponding element under the new page (e.g., binding "Search iPhone" to the search box element of the "Homepage"). The above page switching confirmation can be combined with... Figure 10 The aforementioned URL change and DOM structure change monitoring mechanisms are executed in tandem to improve the real-time performance and accuracy of switch detection.

[0071] In another embodiment, when the alignment matching degree of block B430 is lower than a preset threshold, as shown by arrow A46, processor 110 triggers a rollback retrieval operation in block B440. As described in block B440, processor 110 retrieves candidate page snapshots within an extended time window and selects a target page snapshot based on page similarity to re-execute the target interface element positioning strategy. Specifically, processor 110 extends the preset time window step by step, for example, from 500 milliseconds before and after the initial current time to 1 second, 3 seconds, and 10 seconds before and after, and re-retrieves video frames or images matching the input timestamp within the extended time window. Processor 110 calculates the page fingerprint similarity between each candidate page snapshot and the current interface state, and preferentially selects the page snapshot with higher similarity and closer time distance as the target page snapshot. For example, when a user issues the voice command "Click this," the mouse cursor may have already moved quickly away from the target element, causing the match to fail at the moment the voice command is issued (e.g., the spatial consistency score between the mouse coordinates and the bounding boxes of all candidate elements in the current frame is less than a preset threshold X, or the mouse coordinates do not fall within the bounding box of any candidate element). In this case, the processor 110 extends the time window backward to 1 second, successfully retrieving the event in the video frame before the voice command was issued where the mouse hovered over the "Submit Order" button, thereby completing the spatiotemporal alignment and locking the target element.

[0072] In one embodiment, when the temporal consistency determination for cross-modal alignment fails, the processor 110 triggers a time window rollback retrieval mechanism to retrieve candidate page snapshots within an extended time window and re-execute the binding. Specifically, the processor 110 retrieves a cached set of page snapshots within a preset time window length, calculates the similarity score between the page fingerprint of each candidate snapshot and the target page fingerprint, selects the candidate snapshot with the highest score as the target page snapshot, and performs candidate element detection and semantic matching on the target page snapshot to determine the target element and complete the binding. By recording the time deviation parameter of this rollback retrieval, the processor 110 can also write this parameter into the interface context representation information for subsequent adaptive optimization of the spatiotemporal alignment threshold or time window strategy.

[0073] It is worth mentioning that, in one embodiment, to further enhance the accuracy of spatiotemporal consistency verification, the processor 110, while receiving image or video input, also directly reads the interface structure code of the target interactive interface through the input / output interface 130. This interface structure code includes the Hypertext Markup Language (HTML) source code of a webpage, Cascading Style Sheets (CSS), or the layout file (XML) of an application. The processor 110 parses this interface structure code to obtain the programmatic coordinates of each interface element. When performing spatiotemporal consistency verification in block B430, the processor 110 compares the visual coordinates identified by the visual feature extraction module B322 with this programmatic coordinate information. If the calculated Intersection over Union (IoU) is higher than a preset threshold, the target element's positioning is deemed reliable; if there is a significant deviation, the processor 110 uses the programmatic coordinate information to calibrate the visual coordinates, thereby correcting positioning errors caused by image rendering differences or occlusion.

[0074] In another embodiment, as shown by arrow A47, the feature information verified by spatiotemporal consistency is passed to block B450. As described in block B450, processor 110 performs a multimodal feature fusion algorithm operation, calculating the attention weights of each modality and performing weighted fusion. Specifically, processor 110 uses the operation intent extracted from the speech or text modality as the query vector and the candidate element features extracted from the image or video modality as the key vector, calculating the matching score between the query vector and each key vector. Based on the matching score, processor 110 calculates the attention weight of each modality through a normalization function, and uses the attention weight to perform a weighted summation of the feature vectors of each modality. The weighted summation of the feature vectors is then processed by a multilayer perceptron for feature alignment and nonlinear mapping to generate a fused intent feature vector.

[0075] In one embodiment, the multimodal feature fusion of block B450 can employ an attention-weighted fusion framework to uniformly output a fused feature vector F representing "operation intent - target element," and generate execution instructions B460 accordingly. Specifically, processor 110 constructs a corresponding feature vector F_i for each modality, where i can belong to speech, text, image, and video modalities. Speech modal features can be composed of semantic embeddings and acoustic confidence scores of ASR-transcribed text; text modal features can be composed of predicate-object structure embedding vectors, named entity embedding vectors, and instruction context embedding vectors obtained from dependency parsing; image modal features can be composed of candidate element image embedding vectors, element bounding box coordinate embedding vectors, and element text label embedding vectors; video modal features can be composed of temporal variation features, mouse trajectory / hover event features, and keyframe image embedding vectors. Processor 110 can perform dimensional alignment on the features of each modality to map them to a unified feature space for subsequent fusion.

[0076] In one embodiment, the processor 110 constructs a query vector Q from the core operational intent extracted from the speech / text modality, and constructs a key vector based on candidate page elements and their contextual features. (or construct for each candidate element) Processor 110 calculates the matching function. To characterize the degree of matching between intent and candidate elements, attention weights are obtained based on Softmax normalization. : In one implementation, Using scaled dot product ,in The dimension of the key vector (e.g., 64 by default, configurable); in another implementation... Semantic similarity, visual similarity, and spatiotemporal consistency verification scores can be further integrated to make... It also reflects multiple constraints such as "whether the semantics match", "whether the visuals are similar", and "whether the time and position are aligned".

[0077] In one embodiment, the processor 110 employs a multilayer perceptron (MLP) to analyze the features of each modality. Nonlinear mapping and feature alignment are performed to mitigate the impact of differences in feature distribution across different modalities on the fusion result. The MLP may contain at least two fully connected layers and a nonlinear activation function (e.g., ReLU or GELU), and layer normalization and dropout may be introduced to improve generalization ability. Subsequently, the processor 110 performs a weighted summation of the MLP-mapped modal features according to attention weights to obtain the fused feature vector F, for example: Through this fusion computing, the processor 110 can dynamically measure the contribution ratio of each modality in the current scene: when the voice / text instructions are clear, the weight of the language modality is increased; when the language instructions are ambiguous, the weight of the video mouse hover and the image candidate elements is increased; when the text of the page elements is missing, the weight of the visual embedding vector is increased.

[0078] In one embodiment, processor 110 generates execution instruction B460 based on the fused feature vector F. The execution instruction B460 may include: an action type field (e.g., click, input, scroll, drag, long press, hover), a target element field (containing at least one of the element's structural path information, visual coordinate information, and semantic description information), and optional parameter fields (e.g., input text content, scrolling direction and distance, drag start and end points). When the execution instruction involves multi-step operations, processor 110 may further output operation dependencies or operation sequences as input for subsequent cross-page path planning and control strategy generation.

[0079] In one embodiment, block B450 may also introduce a "confidence output" and "ambiguity resolution" mechanism. Specifically, processor 110 may base its output on the attention distribution entropy value or the highest weight. The difference Δα between the second-highest weight and the second-highest weight generates the confidence level of the instruction.

[0080] When attention weights are concentrated on a single candidate element, the entropy value is low and Δα is large, indicating high instruction confidence; when attention weights are distributed across multiple candidate elements, the entropy value is high and Δα is small, indicating low instruction confidence.

[0081] When the confidence level falls below a threshold, the processor 110 automatically triggers a supplementary strategy to improve the certainty of the target element. This supplementary strategy requires no user intervention. Examples of supplementary strategies include: extending the time window forward or backward to retrieve more video frames, thereby capturing more explicit mouse hover events or trajectory pointing information; introducing document object model structural constraints or control tree hierarchy constraints into the candidate element set to exclude candidate elements that do not conform to the contextual logic; inferring the most likely target element of the current instruction based on historical operation patterns recorded in the operation history graph; and increasing the analytical granularity of the visual detection model to identify finer element boundaries. Through these automated supplementary strategies, the processor 110 can complete ambiguity resolution without interrupting the execution flow and without additional user intervention.

[0082] In another embodiment, multimodal fusion can also employ alternative implementations to extend the scope of protection. For example, a gating network can be used to gate the output coefficients for each modality. and execute Alternatively, a cross-modal Transformer can be used to establish an explicit alignment relationship between language tokens and candidate element tokens through cross-attention; or late fusion can be used to generate candidate element ranking results for each modality separately, and then output the final target element through the learned ranking fusion model. All of the above alternative implementations can be combined with the spatiotemporal consistency verification and fallback retrieval mechanism disclosed herein to achieve robust instruction generation in complex interactive interface environments.

[0083] Taking e-commerce scenarios as an example, the application of the above alternative implementation methods is illustrated. When a user issues the voice command "Add this to your cart" and there are multiple similar products on the page, the gating fusion method can dynamically adjust the contribution weight of each modality according to the current scenario. When the mouse hover signal is clear, the gating coefficient of the video modality is increased, and when the voice description is detailed, the gating coefficient of the voice modality is increased. The cross-modal Transformer method can use a cross-attention mechanism to make the word "this" directly focus on the product image area corresponding to the mouse hover position, establishing an alignment between language reference and visual target. The post-fusion method can generate candidate product ranking lists based on voice semantics, mouse trajectory, and screenshot visual features respectively, and then combine the three lists through a ranking fusion model to output the final target product.

[0084] In one embodiment, as indicated by arrow A48, block B450 outputs execution instruction B460. As described in block B460, execution instruction B460 comprises two main components: an action type and a target interface element description. The action type indicates the type of automated operation to be performed, such as clicking, double-clicking, long-pressing, text input, scrolling, dragging, or hovering. The target interface element description provides multi-dimensional feature information of the interface element to be operated, including the element's text content, visual feature vector, semantic description, interface structure path, and visual coordinates. Execution instruction B460, as the output of the multimodal input parsing process, is passed to the control strategy generation module to determine the specific target interface element positioning strategy and generate control instructions. For example, the specific content of execution instruction B460 may include: {Action type: "Click", Target description: "A red button with the text 'Checkout', located in the lower right corner of the page"}, or {Action type: "Input", Target description: "Search input box", Input parameter: "Wireless headphones"}.

[0085] In one embodiment, the execution instruction B460 further includes a parameter field, which carries the input data required to perform the automated operation. The parameter field is organized in a structured format, and each parameter record includes at least three attributes: parameter name, parameter value, and parameter type. The parameter name identifies the purpose of the parameter in the operation process, such as "account," "password," "search keyword," or "shipping address." The parameter value is the specific content of the parameter, which is filled into the target interface element by the processor 110 during the automated operation. The parameter type indicates the data format and processing method of the parameter value; common parameter types include text, numeric, date, and enumeration types.

[0086] When the execution instruction involves operations requiring user credentials or sensitive information, the relevant parameters in the parameter field are pre-configured in the execution instruction. These pre-configured parameters can be provided by the user when generating the execution instruction, or they can be read from a pre-defined parameter configuration file or security credential storage. When executing automated steps involving input operations, the processor 110 extracts the corresponding parameter values ​​from the parameter field of the execution instruction and fills them into the located target interface elements. For example, when the execution instruction includes a login operation, the processor 110 extracts the parameter values ​​named "username" and "password" from the parameter field, fills them into the username and password input boxes respectively, and then triggers the login button click operation.

[0087] In another embodiment, the parameter field may also include a parameter sensitivity indicator to indicate whether the parameter involves sensitive information. When the parameter sensitivity indicator indicates that the parameter is a sensitive parameter, the processor 110 performs desensitization processing on the parameter value during log recording and error report generation to avoid accidental leakage of sensitive information. Desensitization processing methods include replacing part or all of the parameter value with mask characters, or completely omitting the record of the parameter value, etc.

[0088] In another embodiment, the parameter field supports a parameter referencing mechanism, allowing a parameter value in the execution instruction to reference the execution result of a previous operation step. For example, in an e-commerce order process, the product price displayed on the product details page can be extracted as a parameter and passed to the subsequent payment confirmation step. The processor 110 references this price parameter in the payment confirmation step to verify the amount. The parameter referencing mechanism enables cross-step data transfer to be defined at the execution instruction level, without the need for additional data storage and retrieval operations.

[0089] Based on the above, this embodiment achieves accurate page identification by combining visual and semantic features, and establishes a context-aware foundation for cross-page path prediction by utilizing graph neural network modeling operations.

[0090] In one embodiment, the parameters in the above multimodal alignment and fusion process can be configured to adapt to different device performance and different interface types. For example, the confidence threshold for element detection can be set to 0.5 to 0.9; the CLIP similarity threshold can be set to 0.5 to 0.8; and the time threshold... The time window for backtracking retrieval can be set from 200ms to 1500ms; the time window for backtracking retrieval can be set from 3 seconds to 30 seconds; the semantic weight λ in page similarity calculation can be set from 0.5 to 0.9; the key vector dimension... The parameters can be set to 32, 64, or 128; the number of MLP layers can be set to 2 to 4. The processor 110 can adaptively adjust the above parameters based on historical execution success rates and environmental statistics, thereby achieving a balance between execution efficiency and localization robustness.

[0091] For ease of understanding Figure 4 The multimodal input parsing process and its technical terminology illustrate how the processor 110 can be viewed as an intelligent assistant that observes user actions simultaneously through auditory channels such as voice or text, and visual channels such as images or video. First, the intelligent assistant identifies candidate element regions by analyzing screenshots; this is equivalent to circling all areas on the screen that resemble buttons or input boxes, listing them as potential targets the user might want to click. Simultaneously, it captures hover events by observing operation recordings, i.e., detecting when the mouse cursor pauses briefly at a certain location. This usually indicates that the user's attention is focused on that area.

[0092] Next, the intelligent assistant (processor 110) performs spatiotemporal consistency verification, which is similar to checking the voice and actions. It checks the timing of the user's speech, such as whether the mouse is pointing at a candidate element area at the exact moment the user says "click this." If the time and position match, it means that what the user said is what the mouse is pointing to. If the intelligent assistant doesn't see where the mouse is at the moment the user speaks, it will initiate a rollback retrieval mechanism, rewinding the recording for a few seconds to expand the time window and see if the mouse has pointed to anything in the short period before and after the speech. This is a fault-tolerance mechanism.

[0093] The core multimodal feature fusion algorithm can be considered one of the brains of the entire process. It analyzes all clues—heard intent, seen visual features, and mouse cursor trajectory—and determines which clue is more important. For example, sometimes the user doesn't speak but clicks the mouse rapidly, so it listens to the mouse; sometimes the mouse moves erratically but the speech is clear, so it listens to the voice. Finally, the intelligent assistant generates execution instructions after processing, translating the ambiguous user intent into precise machine language, such as clicking the red button marked "OK" at a specific coordinate, and then sends it to the automation execution module to perform the operation.

[0094] In one embodiment, the processor 110 can determine a unique target element through multimodal feature alignment and fusion in scenarios where there are multiple candidate target elements and the voice / text commands are ambiguous. For example, a user is on a product list page of an e-commerce website, where two similar-looking red sneakers, both labeled "new," are displayed. The user issues a voice command, "Add this red one to my cart," and briefly hovers the mouse cursor over the second red sneaker card before and after issuing the voice command. Simultaneously, the user uploads a screenshot showing partial details of the first shoe and adds "I want the new one" to the text box. In this scenario, relying solely on voice / text cannot distinguish between the two candidate elements, relying solely on the screenshot might lead to mistakenly selecting the first shoe, and relying solely on the hover trajectory lacks a clear type of operation intent. Therefore, the processor 110 simultaneously utilizes multimodal inputs such as voice, text, image, and video / trajectory to form a unified execution command.

[0095] In one embodiment, processor 110 converts the multimodal inputs into feature vectors and defines a query vector Q and a key vector set {Ki} to calculate attention weights. Specifically, processor 110 performs ASR and NLP parsing on the speech to obtain the operation intent (e.g., addToCart) and entity features (e.g., color:red, tag:new), and encodes them into a query vector Q; processor 110 detects hover events on video or screen recording data and extracts the screen coordinates and duration of the hovering area; processor 110 extracts visual embedding vectors from user-uploaded screenshots and locates their candidate matching areas on the current page; processor 110 extracts entity information (e.g., the "new" tag) from supplementary text input. Processor 110 uses each candidate product card / button element on the page as a key vector Ki, where Ki at least includes the image embedding, text tag (including OCR / accessibility text), and its coordinate information on the page for that candidate element. Subsequently, processor 110 calculates the matching function score(Q,Ki), where score(Q,Ki) integrates at least three pieces of evidence: semantic similarity, visual similarity, and spatiotemporal consistency. Semantic similarity reflects the degree of matching between the candidate element and descriptions such as "red / new / add to cart"; visual similarity reflects the degree of similarity between the candidate element region and the features of the user's screenshot; and spatiotemporal consistency reflects the degree of consistency between the candidate element coordinates and the mouse hover area within a time window. Processor 110 can obtain the attention weight αi of each candidate element based on softmax normalization and determine the final target element accordingly.

[0096] In one embodiment, the processor 110 can perform weighted fusion of the features of each modality according to the attention weights to generate a fused feature representation F, and output the final execution instruction. For example, for the first candidate element of red sneakers, its visual similarity to the screenshot is high, but its spatiotemporal consistency with the mouse hover area is low; for the second candidate element of red sneakers, its visual similarity to the screenshot is low, but its spatiotemporal consistency with the mouse hover area is high, and it also satisfies the semantic constraint of "red / new". Therefore, after softmax calculation, the weight α corresponding to the second candidate element is higher, making the fused feature F more strongly point to the second candidate element and carrying the operation intent of addToCart. The processor 110 finally generates the execution instruction, specifying to click the "Add to Cart" button corresponding to the second red sneaker, and after successful execution, writes back the selection basis (such as hover evidence strength, semantic matching score, and target element path / coordinates) to the interface context representation information, so that it can be directly reused or accelerated in subsequent similar ambiguous scenarios.

[0097] Reference Figure 5 In one embodiment, the processor 110 executes a process for constructing interface context representation information, which generates interface context representation information including page fingerprint information B510 and operation history graph information B520. As described in block B510, the page fingerprint information is used to uniquely identify the state of each page of the target interactive interface; as described in block B520, the operation history graph information is used to model the operation flow dependencies between interfaces. These two together constitute the system's "memory" and "map," enabling the processor 110 to understand the current environment and predict the execution path of cross-page operation chains.

[0098] Specifically, processor 110 generates page fingerprint information B510 by fusing visual and semantic features. As shown by arrow A51, processor 110 performs structural feature extraction on the interface screenshot of block B511, such as calculating a perceptual hash value (pHash) after size normalization and low-pass filtering, thereby generating a visual hash value with local invariance. Simultaneously, as shown by arrow A52, processor 110 performs semantic encoding on the interface identification information (such as URL and title) of block B512, such as extracting semantic vectors from the text sequence using a pre-trained language model (such as BERT). As shown by arrow A53, processor 110 combines the visual hash value and the semantic vector to generate page fingerprint B515. This dual feature combination mechanism improves the robustness of page recognition: when the visual layout of the interface is similar but the business logic is different (such as the details pages of different products), the semantic vector can be used as a distinguishing factor; when the interface changes visually due to minor style adjustments, the consistency of the semantic vector ensures that the page is correctly identified as the same node.

[0099] In one embodiment, the processor 110 calculates the page fingerprint similarity between two pages using the following mathematical formula. The page fingerprint similarity comprehensively considers the cosine similarity of semantic vectors and the structural similarity of interface screenshots, and the formula is expressed as: In the above formula, V1 and V2 are the semantic vectors of the two pages to be compared. These semantic vectors are generated by a pre-trained language model encoding the page's Uniform Resource Locator (URL) and title. The formula is expressed as follows: ,in The bidirectional encoder represents the basic scale of the model, where the URL is the page's Uniform Resource Locator (URL) and the Title is the page's title text. The processor 110 concatenates the URL and title text and inputs them into the pre-trained language model, extracting the model's output classification label vector as the page's semantic vector. cosine(V1, V2) represents the cosine similarity between two semantic vectors, calculated as the dot product of the two vectors divided by the product of their magnitudes. Its value ranges from -1 to 1, with values ​​closer to 1 indicating greater semantic similarity.

[0100] I1 and I2 are screenshots of the two pages to be compared. These screenshots are normalized in size and then used for structural similarity calculation. SSIM(I1, I2) represents the structural similarity index, which evaluates the visual similarity between two images from three dimensions: brightness, contrast, and structure. The value ranges from 0 to 1, with a value closer to 1 indicating greater visual structural similarity. In one implementation, the processor 110 adjusts the screenshots to a uniform resolution (e.g., 224×224 pixels) and calculates the structural similarity index using a preset window size (e.g., 11×11 pixels).

[0101] λ is the weighting coefficient between semantic similarity and structural similarity, used to adjust the contribution ratio of the two similarity measures in the overall score. The default value of the weighting coefficient λ can be set to 0.7, indicating that semantic similarity has a higher weight in the overall score. The value range of the weighting coefficient λ can be configured from 0.5 to 0.9, and the processor 110 can dynamically adjust this weighting coefficient according to the characteristics of the target interactive interface. For web pages with stable Uniform Resource Locator (URL) structures but frequent changes in the visual layout of the interface, the processor 110 can increase the weighting coefficient λ to enhance the role of semantic vectors in page recognition; for interfaces such as single-page applications where the URL structure does not change significantly but the visual layout varies greatly, the processor 110 can decrease the weighting coefficient λ to enhance the role of structural similarity in page recognition.

[0102] Based on this, processor 110 constructs and maintains operation history graph information B520. This information adopts a graph data structure, where graph nodes correspond to page fingerprint information and directed edges correspond to user operations. When processor 110 detects a page jump, it creates or updates a directed connection between the source page node and the target page node. In particular, processor 110 associates and stores the visual coordinates or region information corresponding to the user operation as an attribute feature of the directed connection. This design is crucial because it records "where the operation caused the jump," providing historical visual coordinates as a basis for subsequent multi-level fault-tolerant localization strategies.

[0103] In one embodiment, to achieve dynamic binding of "operation-page-element", the processor 110 further maintains a page context map in the interface context representation information. The map uses page fingerprint information as a key-value index, and each record includes at least: a set of candidate interactive elements corresponding to the page, and a set of page jump conditions related to the page.

[0104] In detail, each element record in the candidate interactive element set includes at least: element semantic tags (e.g., "username input box", "search box", "shopping cart icon"), element structural positioning information (e.g., id, CSS selector, XPath, or control tree path), element visual area information (e.g., bounding box coordinates and size), and element feature vectors for semantic matching (e.g., image embedding vectors and text tag vectors). The page jump condition set is used to characterize the rules or statistical relationships of "what operation performed on this page will trigger which page", such as "clicking the login button → jump to the homepage", "clicking the product card → jump to the details page". Through the above mapping table, the processor 110 can bind each operation to its corresponding page and corresponding element in cross-page, multi-step instruction scenarios, thereby reducing the ambiguity and misbinding risk during cross-page execution.

[0105] In one embodiment, to achieve dynamic page recognition, the processor 110 introduces a real-time environment awareness module, which includes at least a visual positioning engine, a URL listener, and an operation history graph inference unit. Before each operation, the visual positioning engine captures a screenshot of the current interface and performs visual element detection on the screenshot to extract a set of feature elements for the current page. The visual element detection model includes, but is not limited to, YOLO series detection models or ViT-based detection / segmentation models. Feature elements include interactive controls and page-specific identifying areas (e.g., the banner area unique to the homepage). The URL listener monitors browser URL changes and performs page type recognition based on page text features (e.g., title, DOM text summary). The page type recognition model includes, but is not limited to, a classification model based on a text convolutional network. The operation history graph inference unit predicts the next possible page node based on the node-edge relationships in the operation history graph information using a graph neural network, thus forming a priori constraints on page switching. The processor 110 performs consistency fusion on the above multi-source results, outputs the current page recognition result and confidence level, and drives subsequent positioning strategy selection and context updates accordingly.

[0106] Furthermore, as described in block B521, processor 110 utilizes a graph neural network (GNN) to model the operation history graph information B520. Processor 110 uses page fingerprints as node features and user operation types and visual coordinates as edge features, learning the contextual representation of nodes in the graph structure through a message passing mechanism. After GNN modeling, the operation history graph not only records static jump relationships but also captures dynamic patterns of interface transitions, thereby supporting processor 110 in predicting potential cross-page execution paths in complex business processes.

[0107] In one embodiment, the processor 110 also introduces a weighted fingerprint similarity calculation mechanism. The processor 110 dynamically adjusts the weight coefficients of the visual hash value and the semantic vector based on the stability characteristics of the target interactive interface. For example, for web pages with stable URL structures but frequently updated UIs, the processor 110 automatically increases the weight of the semantic vector; for single-page applications (SPAs) and other scenarios where URL changes are insignificant, the weight of the visual hash value is increased. Furthermore, the processor 110 can periodically optimize the operation history graph information B520, merging highly similar redundant nodes and pruning long-unvisited isolated paths to maintain the efficiency of the inference model.

[0108] Based on the above, this embodiment achieves accurate page identification by combining visual and semantic features, and establishes a context-aware foundation for cross-page path prediction by using graph neural networks to model the operation flow.

[0109] Reference Figure 6In one embodiment, the processor 110 executes a session recovery process. This process determines a session recovery point after an automated execution session is interrupted and restarted, enabling the processor 110 to continue executing subsequent operation instructions from the interruption point without re-executing completed operation steps. The session recovery point is a node position in the operation history graph information that matches the current interface state. This node position indicates the execution progress of the automated execution process at the time of interruption.

[0110] Specifically, as shown in step S610, the processor 110 detects an interruption in the automated execution session and restarts. Session interruption can be triggered by various reasons, such as an unexpected browser closure, application crash, or network connection interruption. After the processor 110 restarts and loads the automated execution program, it reads the previously persistently stored interface context representation information from the storage device 120, including page fingerprint information and operation history graph information.

[0111] Next, as shown in step S620, the processor 110 calculates the page fingerprint information of the current interface. The processor 110 obtains a screenshot of the current interface and interface identification information (such as URL and title) through the input / output interface 130, and generates a visual hash value and a semantic vector respectively, and combines them to generate the page fingerprint information of the current interface. This fingerprint information represents the current state of the target interactive interface after the session is restarted.

[0112] Subsequently, as shown in step S630, the processor 110 compares the page fingerprint information of the current interface with each node in the operation history graph information to locate the current execution progress. The processor 110 traverses each page fingerprint node in the operation history graph information and calculates the matching degree between the page fingerprint information of the current interface and each node. This matching degree is calculated based on the page fingerprint similarity, taking into account both the structural similarity of the visual hash value and the cosine similarity of the semantic vector.

[0113] In step S640, the processor 110 determines whether the matching degree is higher than a preset threshold. When the matching degree is higher than the preset threshold, the processor 110 proceeds to step S650, determining the node with the highest matching degree as the node position corresponding to the current interface, i.e., the session recovery point. At this time, the processor 110 loads the unexecuted subsequent operation instructions based on the operation history graph information and continues to execute the automated operation from the session recovery point, without having to re-execute the operation steps completed before the session recovery point.

[0114] In one embodiment, to achieve cross-session state persistence and task continuation, the processor 110 simultaneously writes a session checkpoint record when writing the interface context representation information to the storage device 120. This session checkpoint record includes at least: a current page fingerprint, a current operation segment identifier, a set of completed operation segments, a pointer to the operation segment to be executed, and a verification signal digest for determining segment completion. The verification signal digest may include, for example, URL redirection events, DOM key node changes, pop-up prompt text, changes in the number of shopping cart icons, or the clickable state of a specific button. Through this checkpoint record, the processor 110 can determine not only "which page" it is on after a session interruption and restart, but also "which step it has reached," thereby supporting skipping completed steps and directly proceeding to the operation to be executed.

[0115] In one embodiment, when a session interruption is detected and restarted, the processor 110 first presses... Figure 6 The processor 110 calculates the page fingerprint information of the current page and determines the session recovery point in the operation history graph information. Subsequently, the processor 110 reads the session checkpoint record associated with the session recovery point from the storage device 120, obtains the pointer of the operation segment to be executed, and generates a consistency constraint of "operation segment to be executed - expected page state". If the current page fingerprint is consistent with the expected page in the session checkpoint record, the processor 110 directly loads the target element set of the operation segment to be executed and issues control instructions; if they are inconsistent, the processor 110 calls the candidate page set output by the operation history graph inference unit to perform a second confirmation, and selects the page with the highest similarity to the current page fingerprint from the candidate set as the current page before resuming execution, so as to reduce the risk of misoperation caused by falling into a non-target page after restarting.

[0116] In one embodiment, to avoid repeatedly triggering completed business actions during resumption of execution, the processor 110 performs idempotency control based on the set of completed operation segments in the session checkpoint record and the verification signal summary. Specifically, if the checkpoint record indicates that the "add-to-cart segment is completed" and the corresponding verification signal meets the preset confidence conditions (e.g., a "successfully added" pop-up or a change in the shopping cart quantity logo has been detected and recorded in the historical graph edge attribute), the processor 110 will not re-execute the add-to-cart segment during resumption of execution, but will instead advance the pointer of the operation segment to be executed to the checkout segment. After entering the shopping cart page, the processor 110 can detect the appearance of the "checkout" button based on at least one of structural positioning, visual region monitoring, or semantic matching, and automatically trigger a click when the clickable condition is met, thereby completing cross-session continuation of execution. Through the above mechanism, the system can continue execution from the last interruption point after the browser is abnormally closed or the system is restarted without repeating the login, search, or add-to-cart steps, thereby ensuring the consistency of the business process and improving execution efficiency and robustness.

[0117] Conversely, when the matching degree is lower than a preset threshold, the processor 110 proceeds to step S660 to perform optical character recognition (OCR) assisted matching. The processor 110 performs OCR on the current interface to extract the interface title text, and matches the extracted text with the title information associated with each node in the operation history graph. If the node position is successfully determined based on the text matching, the processor 110 uses it as the session recovery point and proceeds to step S650; if it still cannot be determined, an error report is generated to notify the user to intervene.

[0118] For example, suppose a user executes the process of "login → search → add to cart → checkout," and the browser crashes when redirected to the "shopping cart page." After restarting, the processor 110 calculates the fingerprint of the current page and finds that it highly matches the "shopping cart page" node in the history graph (S640 determines this). Therefore, the processor 110 directly sets the "shopping cart page" as the recovery point (S650) and loads the subsequent "checkout" instruction to continue execution, skipping the previous login and search steps.

[0119] Based on the above, this embodiment utilizes page fingerprinting and OCR-assisted matching to achieve session interruption recovery, avoids repeated execution, and significantly improves the efficiency and anti-interference capability of long-process automated execution.

[0120] Reference Figure 7 In one embodiment, the processor 110 executes a multi-level fault-tolerant localization process, which is used to locate the target interface element specified by the execution instruction on the target interactive interface. The multi-level fault-tolerant localization process adopts a hierarchical strategy mechanism, trying different localization strategies in sequence according to preset priorities. When a higher priority strategy fails, it automatically falls back to the second-highest priority strategy, so that the localization of the target interface element can still be completed when the interface element changes.

[0121] Specifically, as shown in step S700, when the processor 110 needs to locate a target interface element, a multi-level fault-tolerant localization process is initiated. The processor 110 extracts the target interface element description from the executed instructions. This description includes feature information such as the element's text content, interface structure path, visual coordinates, and semantic description.

[0122] First, as shown in step S710, the processor 110 executes a first-level strategy, such as path location based on interface structure information. The processor 110 obtains the interface structure path information (such as XPath, CSS selector, or control tree path) of the target interface element from the target interface element description or interface context representation information, and sends a path query request to the target interactive interface through the input / output interface 130.

[0123] In step S720, processor 110 determines whether the first-level strategy has been successfully located. When the target interactive interface returns a matching interface element, the location is determined to be successful and the process jumps directly to step S790 to execute the automated operation. If a matching element is not returned (for example, due to changes in the DOM structure caused by front-end code refactoring), the location is determined to have failed, and the process proceeds to step S730.

[0124] As shown in step S730, when the first-level strategy fails, the processor 110 executes the second-level strategy, such as performing region positioning based on the historical visual coordinates or region information recorded in the operation history graph. The processor 110 retrieves the directed connection relationship related to the current instruction, obtains the historical visual coordinates of the target element, and searches for interactive elements in the corresponding area of ​​the current interface screenshot.

[0125] In step S740, processor 110 determines whether the second-level strategy has been successfully located. If a matching element is found within the historical area, the location is considered successful, and the process proceeds to step S760 to perform adaptive updates. If no matching element is found within the area (e.g., the element's position has shifted significantly), the location is considered to have failed, and the process proceeds to step S750.

[0126] Next, processor 110 executes a third-level strategy, such as localization based on semantic feature matching. Processor 110 uses a visual element detection model to identify all candidate elements in the current interface screenshot and uses a contrastive language-image pre-trained (CLIP) model to calculate the semantic similarity between the text description in the execution instruction and the image regions of each candidate element. During this process, the semantic features of the candidate elements can be generated based on interface control text, accessibility attribute text (such as aria-label), or OCR-recognized text. Processor 110 selects the candidate element with the highest semantic similarity that is above a threshold as the target.

[0127] In step S770, processor 110 determines whether the third-level strategy has been successfully located. If the target element is successfully identified, proceed to step S760; if the similarity of all candidate elements is lower than the threshold, the location is determined to have failed, and step S780 is initiated to trigger exception handling and generate an error report containing an interface snapshot.

[0128] It is worth noting that, as shown in step S760, after the processor 110 successfully locates the target interface element using the second-level or third-level strategy, it will perform an adaptive update operation. The processor 110 obtains the latest interface structure path information of the target interface element in the current interface and updates it to the interface context representation information. This self-healing mechanism ensures that when the same operation is performed subsequently, the system can reuse the efficient first-level strategy to complete the location without triggering the fault-tolerant process again.

[0129] For example, during a website redesign, an engineer changed the attribute ID of the "Submit Order" button from "submit_btn" to "btn_confirm," causing the XPath path recorded in the first-level strategy to become invalid. After the first-level strategy failed, the processor 110 initiated a third-level strategy and successfully located the button's new position through semantic matching. Subsequently, during adaptive updates, the processor 110 automatically retrieved the button's latest XPath path (containing the new ID "btn_confirm") and updated it in the interface context representation information. The next time the automated script executes this step, the processor 110 will directly use the updated XPath path to successfully locate the button, thus achieving self-healing of the script without human intervention.

[0130] As shown in step S780, when the third-level strategy also fails to locate the target interface element, the processor 110 triggers an exception handling process. This exception handling process aims to provide the user with diagnostic information about the location failure and to correct and supplement the interface context representation information based on user feedback.

[0131] Specifically, processor 110 first generates an error report. This error report includes a snapshot of the current interface, allowing the user to visually view the interface state when positioning fails. The error report also includes a missing positioning information prompt, explaining the specific reasons for the failure encountered by the system when executing positioning strategies at various levels, such as the interface structure path not existing, no matching elements within the historical visual coordinate area, or semantic similarity not reaching the threshold. Furthermore, the error report includes the original content of the executed instructions and a description of the target interface elements, enabling the user to confirm whether the system correctly understands the operation intent. When processor 110 detects that the positioning failure may stem from missing information in a specific input field, the error report may also include a prompt indicating the missing field, such as a notification to the user that there is an input box on the current page that requires input but has not yet received the input parameters.

[0132] The processor 110 presents the generated error report to the user through the input / output interface 130, waiting for the user to provide feedback instructions on the error report. The types of feedback instructions may include the following forms.

[0133] In one embodiment, the feedback instruction is a confirmation result of the target interface element. After viewing the interface snapshot in the error report, the user can directly indicate the actual position of the target interface element in the current interface by clicking or selecting. After receiving the confirmation result, the processor 110 obtains the interface element information corresponding to the position indicated by the user and uses it as the target interface element to perform subsequent automated operations.

[0134] In another embodiment, the feedback instruction is candidate element selection information. The processor 110 may list several candidate elements identified during the execution of the third-level strategy and their semantic similarity scores in the error report. The user selects the correct target element from the candidate element list, and the processor 110 performs subsequent operations based on the selection result.

[0135] In another embodiment, the feedback instruction is supplementary description information of the target interface element. When the user determines that the system's understanding of the operation intention is flawed, a more detailed or accurate description of the target interface element can be provided, such as supplementary text content, color characteristics, relative position description, or functional description. The processor 110 re-executes the target interface element positioning strategy based on the supplementary description.

[0136] In another embodiment, the feedback instruction is retry strategy selection information. The user can instruct the processor 110 to retry using a specific location strategy, such as specifying to use only a semantic matching strategy or specifying to search within an extended interface area. The processor 110 re-executes the location process according to the user-specified strategy parameters.

[0137] After receiving a feedback instruction, the processor 110 updates the interface context representation information based on the instruction. When the user successfully selects a target interface element through confirmation or candidate element selection, the processor 110 acquires the element's interface structure path information, visual coordinate information, and semantic feature information, and stores them in the corresponding directed connection attribute in the operation history graph. This user feedback-based information supplementation mechanism allows the system to learn from failed positioning cases. When the same or similar operation instructions are executed subsequently, the interface context representation information already contains the user-confirmed positioning data, thereby improving the positioning success rate.

[0138] In another embodiment, the processor 110 can identify the pattern characteristics of location failures based on accumulated user feedback data. When a specific type of interface element or a specific page repeatedly fails to locate, the processor 110 can automatically adjust the location strategy priority configuration for that type of element or page, or prompt the system administrator to maintain and update the feature descriptions of the relevant interface elements.

[0139] Finally, as shown in step S790, once any level-one strategy is successfully located, the processor 110 performs automated operations (such as clicking or inputting) based on the located element and updates the operation history graph information based on the operation results.

[0140] Based on the above, this embodiment effectively addresses the positioning failure caused by interface redesign through a multi-level fault-tolerant positioning and adaptive update mechanism, achieving script self-healing and significantly reducing maintenance costs.

[0141] Reference Figure 8AIn one embodiment, processor 110 executes a practical application scenario of cross-page execution path generation. Suppose a user needs to complete a full shopping process test on an e-commerce website, involving sequential operations across multiple pages. User terminal 200 displays the current web page interface (WBS), which includes elements such as a shopping cart icon, a product display area, and an account login entry. The user issues an execution command via voice command (VC): "Log in, search for headphones, proceed to checkout." Simultaneously, the user's mouse cursor hovers over the account login input area on the WBS; this hovering action is collected as part of the multimodal input and considered related to account login.

[0142] As shown by arrow A81, user terminal 200 sends the collected multimodal input data to server 100. Server 100 parses the data and generates an execution instruction containing multiple operation steps. Since this instruction involves an operation sequence spanning multiple pages, server 100 needs to determine the cross-page execution path to plan the page flow order of each operation step. The operation history graph information 100 maintained by server 100 records the historical page structure and jump relationships of the target interactive interface. Figure 8A As shown, the diagram includes login page node ND1, homepage node ND2, product list page node ND3, shopping cart page node ND4, and checkout page node ND5. The nodes are connected by directed links, representing redirects triggered by user actions. As indicated by arrow A82, server 100 generates a cross-page execution path B80 based on the operation history diagram information, indicating the page flow sequence corresponding to each operation step.

[0143] Reference Figure 8B In one embodiment, processor 110 executes a detailed process for generating cross-page execution paths. As shown in block B810, processor 110 first parses the execution instructions to extract operation dependencies. In this example, the instructions are "Log in to account, search for headphones, add to cart, proceed to checkout". Processor 110 identifies four operation steps: login, search, add to cart, and checkout.

[0144] In one embodiment, the processor 110 performs segmented parsing and binding for complex execution instructions containing multiple steps. The processor 110 performs natural language processing on the voice / text instructions, extracting multiple operation segments and establishing temporal dependencies between them. Segmentation is based on, but is not limited to, temporal keywords (e.g., "then," "following," "after completion," etc.) and dependency syntax relationships. The processor 110 maps each operation segment to a triple of "action type + target element description + target page constraint," where the target page constraint is determined jointly by a page operation context mapping table and operation history graph information. The processor 110 determines the expected landing page for the operation segment based on the historical jump paths recorded in the operation history graph information; and retrieves a set of candidate elements under the expected page whose semantics are consistent with the target element description from the page operation context mapping table, using this as the binding result for the operation segment. Through this binding result, the subsequent control strategy generation stage can directly generate the target element location strategy and operation parameters for the corresponding page for the operation segment.

[0145] As shown by arrow A811, processor 110 analyzes the timing dependencies and determines the operation dependencies as "login → search → add to cart → checkout", meaning that subsequent operations depend on the completion of the preceding operations.

[0146] Next, as shown by arrow A820, processor 110 passes the extracted operation dependencies to block B820 for processing. As shown by block B820, processor 110 maps the operation dependencies to the corresponding nodes in the operation history graph information. Processor 110 analyzes the semantic features of each operation step and retrieves matching page fingerprint nodes (also called page nodes or page fingerprints) in the history graph. As shown by arrow A821, based on the semantic features of each operation step, processor 110 maps various operations to the corresponding page fingerprint nodes. For example, processor 110 maps "Login" to login page node ND1, "Search" to homepage node ND2 and product list page node ND3 (identifying the flow between two pages involved in the search), "Add to Cart" to shopping cart page node ND4, and "Checkout" to checkout page node ND5. As shown by arrows A822 to A825, processor 110 determines the connection relationship between the corresponding nodes. For example, arrow A822 indicates that after successful login, the user jumps from ND1 to ND2.

[0147] Subsequently, as indicated by arrow A830, processor 110 passes the mapping result to block B830 for processing. As shown in block B830, processor 110 uses a graph neural network (GNN) to analyze the node connections in the operation history graph information. Processor 110 uses the page fingerprint features of each corresponding node as node input and the user operation type and visual coordinates or region information of each directed connection as edge attribute features. Processor 110 predicts the optimal page flow order from the source node to the target node through the message passing mechanism of the GNN. In particular, processor 110 analyzes the edge attribute features (such as visual coordinates) to confirm whether the historical operation is consistent with the intent of the current instruction. For example, if there are multiple paths from ND1 to ND5 in the history graph, processor 110 selects the optimal path based on the historical success rate of the path and the edge attribute features.

[0148] Finally, as indicated by arrow A840, processor 110 generates a cross-page execution path B840 based on the GNN analysis results. As shown in block B840, this path includes the page flow sequence, indicating the execution page and jump order for each operation step. In this example, the generated path is: Login Page → Homepage → Product List Page → Shopping Cart Page → Checkout Page. As indicated by arrows A841 to A844, this path sequentially passes through five page nodes, clearly defining the complete chain of automated execution. Processor 110 then executes automated operations based on this path and continuously monitors the page jump status. If an anomaly is detected (such as failure to jump as expected), it can replan the path based on the operation history graph or trigger a fault-tolerance mechanism.

[0149] In one embodiment, when the content of the execution instruction is incomplete or ambiguous, the processor 110 infers and completes the intermediate page transition steps required to execute the instruction based on the historical operation patterns recorded in the operation history graph information.

[0150] Specifically, processor 110 first parses the execution instruction to identify the explicitly specified starting and target states. Taking the execution instruction "Pay Order" as an example, processor 110 identifies that the target state of the instruction is to complete the order payment, but the instruction does not explicitly specify the current starting page or the intermediate steps required to reach the payment page. Processor 110 then retrieves page fingerprint nodes related to the target state from the operation history graph information and determines the payment page node as the target node.

[0151] Processor 110 utilizes a graph neural network to perform reverse path deduction on the operation history graph information. Starting from the target node, it traverses in the reverse direction of the directed connections to identify all feasible paths to the target node and their corresponding preceding node sequences. In the example above, processor 110 might identify the path "shopping cart page → checkout page → payment page" as a typical preceding path to the payment page node. Processor 110 further combines the current interface state to calculate the page fingerprint information of the current interface and locate the current node position in the operation history graph information.

[0152] When a recorded connection path exists between the current node and the target node, the processor 110 automatically completes the intermediate operation steps on that path to generate a complete cross-page execution path. For example, if the current interface is a product details page, the processor 110 completes the complete path "product details page → shopping cart page → checkout page → payment page" based on the operation history graph information, and determines the corresponding trigger operation for each jump step on the path, such as clicking the "Add to Cart" button, clicking the "Proceed to Checkout" button, and clicking the "Confirm Payment" button.

[0153] When no direct connection path from the current node to the target node exists in the operation history graph information, the processor 110 calculates the semantic similarity between nodes based on the node embedding vectors of the graph neural network, inferring potential paths that may exist but have not yet been recorded. The processor 110 selects a sequence of nodes with high semantic similarity and that conforms to the business logic as the inferred path, and verifies the validity of the path during execution. If the inferred path is verified to be valid during execution, the processor 110 adds the new path to the operation history graph information for subsequent reuse.

[0154] Through the aforementioned intent-driven page prediction mechanism, the processor 110 can automatically plan and execute a complete cross-page operation process even when the user only provides a high-level business objective without specifying the operation steps in detail, thereby reducing the complexity of the user writing execution instructions.

[0155] In one embodiment, the processor 110 uses the dynamic page recognition result to drive the binding and switching control of operation segments in the cross-page execution chain. Specifically, when the processor 110 executes a composite execution instruction consisting of multiple operation steps, it divides the composite execution instruction into at least two operation segments and maintains the corresponding expected page state for each operation segment, so as to establish a synchronous constraint relationship between "operation segment advancement" and "page state evolution".

[0156] In one embodiment, when the processor 110 detects that the recognition result of the current page has changed and matches the expected page state of the next operation segment, the processor 110 binds the target interface element of the next operation segment to the new page, and generates a control instruction corresponding to the operation segment based on the binding result to continue execution, thereby achieving a smooth transition of the operation segment after page switching.

[0157] In one embodiment, when the processor 110 detects that the page recognition result does not match the expected page state, the processor 110 triggers a secondary confirmation process: based on the next page candidate set predicted by the operation history graph inference unit, the processor 110 calculates the similarity between each candidate page in the candidate set and the fingerprint information of the current page, and selects the candidate page with the highest similarity as the current page; subsequently, the processor 110 binds the target interface element of the next operation segment to the determined current page and continues execution. Through the above linkage mechanism, the processor 110 can stably advance the execution of each operation segment in the cross-page execution chain, reducing the probability of abnormal situations such as "operation segment has been switched but page has not been switched" or "page has been switched but operation segment has not been switched".

[0158] Based on the above, this embodiment realizes intelligent path planning for complex cross-page business processes by parsing operation dependencies and using graph neural networks to analyze historical graph connections, thereby improving the success rate of automated execution of long links.

[0159] The following example illustrates an order process that spans multiple pages.

[0160] In a practical application scenario, the execution instruction might be something like, "Log in to your account, search for headphones on the product page, select the first result and add it to your cart, then proceed to checkout." The processor 110 divides this complex instruction into multiple operation segments and maintains the expected page state for each segment to achieve consistent control between the operation segment progression and the page state evolution. Before each operation, the processor 110 obtains the current interface state and confirms the current page type using at least one of the following methods: URL change monitoring, DOM structure change monitoring, and page fingerprint similarity calculation. When the page identification result matches the expected page, it proceeds to the next operation segment. When the page identification result does not match the expected page, the processor 110 performs a secondary confirmation based on the candidate page set derived from the operation history graph, and selects the page with the highest fingerprint similarity to the current page as the current page before proceeding with execution.

[0161] In one embodiment, when performing a login operation on page 1 (login page), processor 110 prioritizes locating the account input box and password input box based on the interface structure path information. When the structure path fails or the input box lacks a stable identifier, processor 110 can further use OCR recognition or accessibility attribute text extraction to obtain the prompt text near the input box, and combine it with visual coordinates to complete element location in the corresponding area. Subsequently, processor 110 writes the account parameters obtained from voice / text parsing into the corresponding input box and triggers the login button click. After the login operation, processor 110 listens for URL redirection or page fingerprint changes to confirm that it has switched to page 2 (homepage), and uses this switching result as a verification signal for the completion of the operation segment.

[0162] Next, when performing a search operation on page 2 (the homepage), processor 110 extracts the target element description of "search for headphones" from the execution instruction, locates the search box on the homepage, enters the keyword, and then triggers the search submission event. Processor 110 detects that the page has switched to page 3 (the product list page) by methods such as the proportion of DOM structure reconstruction, changes in the URL path, or a decrease and reconvergence of page fingerprint similarity. When it is detected that the URL remains unchanged due to a single-page application, processor 110 can combine changes in the key DOM regions with changes in the visual fingerprint to determine that the view jump is complete.

[0163] Subsequently, when executing "Select the first one to add to cart" on page 3 (product list page), processor 110 can use a visual element detection model to identify the product card area and the candidate set of "Add to cart" buttons, and select the candidate card that is first in the list as the target based on the order constraint of "first one"; then, processor 110 clicks the "Add to cart" button in the target card. To verify the success of the operation, processor 110 can detect at least one of the following as operation verification signals: "Add successful" pop-up, change in the number of cart icons, or change in DOM nodes related to the cart in the page state; when the verification is successful, processor 110 confirms entry into page 4 (shopping cart page) based on jump detection, and records the element path, visual coordinates and semantic features of this operation to update the edge attribute features in the operation history graph.

[0164] Finally, when performing the checkout operation on page 4 (shopping cart page), processor 110 can automatically advance based on context rules without explicit instructions: for example, automatically clicking when the checkout button is detected and meets the clickable condition. Processor 110 can detect the appearance of the checkout button using at least one of the following methods: structural positioning (XPath / CSS selectors), visual coordinate region monitoring, or semantic matching (such as extracting the "checkout" text based on OCR and matching it with candidate elements). After clicking the checkout button, processor 110 continues to monitor page fingerprint changes and navigation status to advance to the next page or complete the process loop, and feeds back the execution results to update page fingerprint information and operation history graph information, thereby improving the execution stability and self-healing capability of subsequent similar tasks.

[0165] To further illustrate the consistency control mechanism between operation segments and page states in the cross-page execution chain, another embodiment is provided below, which explains it from the perspective of operation segment decomposition, expected page state maintenance, and secondary confirmation logic.

[0166] In another embodiment, the processor 110 can decompose execution instructions containing multi-step operations into multiple operation segments arranged in a time-dependent manner, and maintain the corresponding expected page state for each operation segment to achieve linkage control between operation segment advancement and page state evolution during cross-page execution. Specifically, the processor 110 performs natural language parsing on the execution instructions. Taking the user instruction "Log in to account, search for headphones on the product page, select the first one to add to cart, and finally proceed to checkout" as an example, the processor 110 extracts multiple operation segments such as login segment, search segment, add-to-cart segment, and checkout segment, and determines the page fingerprint node corresponding to each operation segment based on the operation history graph information. The login segment corresponds to the login page node, the search segment corresponds to the homepage node and the product list page node, the add-to-cart segment corresponds to the product list page node or the product details page node, and the checkout segment corresponds to the shopping cart page node and the checkout page node. The processor 110 records the target element set within each operation segment, the segment completion judgment condition, and the segment switching trigger condition for each operation segment. The segment switching trigger condition includes situations such as changes in page fingerprint information, changes in Uniform Resource Locators (URLs), or document object model (DOM) structure reconstruction ratio exceeding a preset threshold.

[0167] During the execution of the current operation segment, processor 110 continuously performs dynamic page recognition to confirm whether the current interface has switched to the expected page state of the operation segment, and accordingly binds the target interface element of the next operation segment to the corresponding page. Processor 110 obtains the Uniform Resource Locator (URL), Document Object Model (DOM) structure summary, and interface screenshot of the current interface through input / output interface 130, and calculates the similarity between the current interface and the expected page based on page fingerprint information. When processor 110 detects a change in page fingerprint information that matches the expected page, it binds the target interface element of the next operation segment to that new page. For example, when processor 110 detects that the URL jumps to the homepage after a login operation and the page fingerprint similarity is higher than a preset threshold, it binds the target interface element of the search segment to the search input box element on the homepage and generates a control command for inputting keywords. When the page fingerprint information does not match the expected page, processor 110 uses a graph neural network to deduce the operation history graph information, outputs a candidate node set for the next page, and performs a secondary confirmation. The processor 110 selects the node with the highest similarity to the page fingerprint information of the current interface from the candidate node set as the node position corresponding to the current interface, so as to reduce the abnormal probability of the operation segment being switched but the page not being switched or the page being switched but the operation segment not being switched.

[0168] Processor 110 can use observable events as verification signals for the completion of an operation segment, and update the operation history graph information and page fingerprint information accordingly. During the execution of the login segment, processor 110 identifies the account input box through optical character recognition or path positioning based on interface structure information, and triggers the login button click operation after filling in the account parameters. When processor 110 detects a change in the Uniform Resource Locator and the page fingerprint information matches the homepage node, it determines that the login segment is complete and switches to the search segment. During the execution of the search segment, processor 110 locates the search input box on the homepage and enters search keywords, and then triggers the search submission operation. When processor 110 detects that the document object model structure changes from the homepage layout to the product list layout and the page fingerprint information matches the product list page node, it determines that the search segment is complete and switches to the add-to-cart segment. During the execution of the add-to-cart segment, processor 110 identifies the set of product card areas through visual detection, and determines that the target card is the first item in the list by combining the ordinal description in the execution instruction, and then performs a click operation on the add-to-cart button corresponding to the target card. When processor 110 detects the appearance of a successful add-to-cart notification pop-up or a change in the shopping cart quantity icon, it treats this event as a verification signal for a successful operation and writes the verification result into the attribute features of the corresponding directed connection relationship in the operation history graph information to improve the reliability of subsequent path deduction. During the checkout process, after entering the shopping cart page, processor 110 continuously monitors the appearance or clickable status of the checkout button. When preset conditions are met, it automatically triggers a click operation to complete the cross-page order process.

[0169] In the aforementioned cross-page execution process, different target interface elements exhibit varying stability in terms of structure, position, and semantics. This difference influences the priority of strategies at each level within the target interface element positioning strategy. Based on this, the following references... Figure 9 This paper explains the calculation method of the positioning stability score and its dynamic adjustment mechanism for the strategy execution priority in the multi-level fault-tolerant positioning mechanism.

[0170] Reference Figure 9 In one embodiment, the processor 110 performs a positioning stability score calculation and a dynamic strategy adjustment process based on the score. This process enables the processor 110 to intelligently adjust the execution mode of each level of the multi-level fault-tolerant positioning mechanism according to the historical positioning performance of the target interface element, thereby selecting the most suitable positioning strategy under different interface change scenarios.

[0171] Specifically, as shown in block B910, processor 110 obtains the historical records of the target interface elements from the operation history graph information as the data basis for score calculation. This historical record contains data such as the interface structure path, visual coordinates, and semantic features of the target element in previous positioning operations.

[0172] Next, as shown in block B920, processor 110 calculates a localization stability score based on historical records. This score integrates stability across three dimensions (A921-A923). As shown in annotation A921, semantic stability ( The similarity is calculated based on the semantic feature consistency of target elements in historical records, such as the similarity of text content or CLIP feature vectors. As shown in annotation A922, positional stability ( The score is obtained based on the statistical analysis of the visual coordinate offset of the target element in historical records; the smaller the offset, the higher the score. As shown in annotation A923, structural stability ( This is based on statistics of the consistency ratio of interface structure paths, that is, the frequency with which the DOM path of the element remains unchanged in historical versions.

[0173] Regarding semantic stability, processor 110 uses a contrastive language-image pre-trained model to calculate the semantic similarity between the image regions of target interface elements and their text descriptions, expressed by the formula: ,in Element_desc represents the image region of the target interface element in the screenshot, and Element_desc represents the text description information of that element. Processor 110 calculates the stability of this semantic similarity in the historical records based on multiple location results recorded in the operation history graph. A higher semantic stability score is achieved when the semantic similarity remains consistent across multiple location attempts. Regarding positional stability, processor 110 performs statistical calculations based on the visual coordinate offset of the target interface element in the historical records, expressed by the formula: = 1 - ||Δx, Δy|| / max(W, H), where Δx and Δy are the horizontal and vertical coordinate offsets of the target interface element in two adjacent positioning records, respectively, and W and H are the width and height of the target interactive interface, respectively. When the position of the target interface element remains stable in multiple positioning, the coordinate offset is small, and the position stability score approaches 1; when the position of the target interface element changes frequently, the coordinate offset is large, and the position stability score decreases. Regarding structural stability, the processor 110 calculates based on the consistency ratio of the interface structure path of the target interface element in the historical records. In one implementation, the structural stability score is related to the path depth of the target element in the document object model tree, expressed by the formula: = 1 / ( + 1), where This represents the hierarchical depth of the target UI element within the Document Object Model (DOM) tree. Elements with shallower path depths typically exhibit higher structural stability because they are less likely to be affected by front-end code refactoring. In another implementation, processor 110 calculates the proportion of times the UI structure path of the target UI element remains unchanged throughout the historical records, and uses this proportion as the structural stability score.

[0174] Subsequently, as shown in block B930, processor 110 weights and combines the scores from the above three dimensions to generate the final positioning stability score. For example, the calculation formula can be: = α* + β* + γ* , where α, β, and γ are weighting coefficients.

[0175] In step B940, the processor 110 determines whether the score meets the preset threshold condition and decides the strategy selection path accordingly.

[0176] If the score meets the high threshold condition (e.g.) (≥ 0.8), as shown in block B950, processor 110 directly selects the preferred strategy. This means that the target element is very stable, and processor 110 prioritizes the most efficient "XPath positioning" as the first choice, only falling back to "visual coordinates" in case of failure. This mechanism avoids unnecessary complex calculations on stable pages.

[0177] If the score is in the middle range (e.g., 0.5 ≤ If the value is less than 0.8, the processor 110 may determine that the DOM structure of the element changes frequently but its position is relatively fixed. Therefore, it will prioritize "visual coordinate positioning" as the preferred strategy and use "semantic matching" as an alternative.

[0178] If the score is below the low threshold (e.g.) <0.5), as shown in block B960, the processor 110 performs dynamic adjustments, prioritizing the most interference-resistant "semantic matching (CLIP+OCR)" as the preferred strategy, and even skipping XPath and coordinate positioning to cope with high-risk scenarios such as major interface redesigns or element reconstruction. This scoring-based dynamic adjustment mechanism ensures that the system can always find the most cost-effective positioning solution when facing interface elements with different levels of stability.

[0179] In one embodiment, the processor 110 determines the combination configuration of the preferred positioning strategy and the alternative positioning strategy according to a preset strategy selection rule based on the specific numerical range of the positioning stability score.

[0180] When the positioning stability score is greater than or equal to a first threshold (e.g., 0.8), the processor 110 determines that the target interface element has high stability in the three dimensions of structure, position, and semantics, and the probability that the positioning of the element will fail due to a redesign of the target interactive interface is low. In this case, the processor 110 sets path positioning based on interface structure information as the preferred strategy and region positioning based on historical visual coordinates as the alternative strategy. This configuration prioritizes the use of the more efficient structural path query method to complete the positioning, and only enables visual coordinate matching when the structural path fails.

[0181] When the localization stability score is greater than or equal to the second threshold (e.g., 0.5) and less than the first threshold, the processor 110 determines that the document object model structure of the target interface element may be at risk of change, but its visual position in the interface is relatively stable. In this case, the processor 110 sets region localization based on historical visual coordinates as the preferred strategy and localization based on semantic feature matching as the alternative strategy. This configuration skips potentially invalid structural path queries and directly uses visual coordinate matching to improve the localization success rate.

[0182] When the positioning stability score is less than the second threshold, the processor 110 determines that the target interface element has a high risk of change in both structure and position, and the target interactive interface may have undergone significant redesign or element reconstruction. In this case, the processor 110 sets semantic feature matching as the preferred strategy. Semantic feature matching uses a contrastive language-image pre-trained model to calculate the semantic similarity between the execution instruction and the candidate element, which has a strong tolerance for changes in the interface layout. The processor 110 sets layout similarity-based matching as an alternative strategy. When semantic feature matching fails to locate the target element, the processor 110 analyzes the overall layout similarity between the current interface screenshot and the historical interface screenshots recorded in the operation history graph information, retrieves the relative position information of the target element in the historical interfaces with high layout similarity, and maps the relative position to the current interface to complete the positioning.

[0183] The threshold parameters mentioned above can be configured and adjusted according to the characteristics of the target interactive interface and actual application requirements. For stable interfaces with low update frequency, the first threshold can be appropriately lowered to expand the applicable scope of structural path localization; for dynamic interfaces with high update frequency, the second threshold can be appropriately increased to prioritize the use of semantic matching strategies with stronger anti-interference capabilities.

[0184] Based on the above, this embodiment achieves maximum execution efficiency while ensuring positioning accuracy by quantitatively evaluating the stability of elements in semantic, positional and structural dimensions and dynamically optimizing the priority of positioning strategies.

[0185] Reference Figure 10 In one embodiment, the processor 110 executes a real-time environment monitoring process, which, as part of building and maintaining interface context representation information, continuously monitors the state changes of the target interactive interface and updates the interface context representation information when a page jump is detected. Through real-time environment monitoring, the processor 110 can capture the actual interaction process between the user and the target interactive interface, enabling the operation history graph information to reflect the user's operation trajectory and page flow history.

[0186] Specifically, as shown in step S1010, the processor 110 determines the target interactive interface that needs to be monitored. This interface can be a web page running in a browser or an application interface running on a mobile device.

[0187] Next, as shown in step S1020, the processor 110 continuously monitors changes to the target interactive interface. The processor 110 establishes a monitoring connection through the input / output interface 130 to acquire state update events. For web page interfaces, the processor 110 monitors two types of indicators: changes in the Uniform Resource Locator (URL) and changes in the Document Object Model (DOM) structure. For example, the processor 110 can use the browser's MutationObserver interface to monitor changes in DOM tree nodes, or capture URL change notifications through automated testing protocols. Furthermore, the processor 110 can also combine video stream analysis technology to detect changes in visual features of the interface frame by frame (such as the disappearance of the navigation bar or the appearance of pop-ups) as an auxiliary basis for judgment.

[0188] In one embodiment, when the page recognition result is inconsistent with the expected page or the target element location fails, the processor 110 triggers a multimodal fallback mechanism and executes a fallback location strategy sequence according to a preset priority. This fallback location strategy sequence includes: first, attempting location based on a structural path (e.g., XPath, CSS selectors, or control tree paths); if structural path location fails, attempting region location based on historical visual coordinates; if region location still fails, attempting location based on text semantic / image semantic matching, where image semantic matching uses a contrastive language-image pre-trained model to calculate the similarity between the target element description and the candidate element image region, selecting the candidate element with the highest similarity and meeting the threshold condition as the target element. After a successful fallback, the processor 110 writes the latest structural path and visual region information of the target element on the current page back to the interface context representation information, enabling the restoration of high-priority strategies during subsequent execution, thus forming an adaptive repair closed loop.

[0189] In step S1030, when a change is detected, processor 110 determines whether a page jump has occurred on the current interface. Processor 110 uses a multi-dimensional comprehensive judgment logic: when the URL path changes, it is determined to be a jump; when the proportion of DOM structure reconstruction exceeds a preset threshold but the URL remains unchanged, it is determined to be a view jump within the single-page application; when only local element attributes change, it is determined to be an in-page update. If it is determined that no jump has occurred, it returns to step S1020 to continue monitoring.

[0190] In one embodiment, when the similarity score between the current page fingerprint calculated by the processor 110 and the target page fingerprint is lower than a preset threshold, the processor 110 initiates a fuzzy matching strategy to assist in page localization. The fuzzy matching strategy includes: performing OCR recognition on the current page to extract the page title text or key area text, and comparing the extracted text with the title text, URL semantic vector, or page type tag recorded in the interface context representation information of the target page; when the text comparison meets the threshold condition, the processor 110 determines the page as the target page and continues to perform subsequent localization and operations. By cross-validating the OCR results with the page type results output by the URL listener, the processor 110 can maintain the stability of page recognition even when UI redesigns cause changes in visual layout.

[0191] As shown in step S1040, when a page jump is determined to occur, the processor 110 calculates the page fingerprint information of the interface after the jump in real time. The processor 110 obtains a screenshot and identification information of the new interface, extracts the visual hash value and semantic vector respectively, and combines them.

[0192] Subsequently, as shown in step S1050, the processor 110 updates the operation history graph information based on the newly calculated fingerprint information. The processor 110 compares the similarity of the new fingerprint with existing nodes in the history graph. If a match is successful, the current state is located at an existing node; if a match fails, a new page node is created. The processor 110 creates or updates a directed connection between the source node and the target node, and stores the user operation type that triggered this jump and the visual coordinates at the time of the operation as attribute features of the connection.

[0193] Through the aforementioned loop monitoring mechanism, processor 110 constructs an operation history graph. For example, when a user browses the login page, homepage, and product pages sequentially on an e-commerce website, processor 110 captures each jump, adds these three pages as nodes to the graph, and records operations such as clicking login and clicking products as connecting edges. This records the page structure and business flow logic, providing a data foundation for subsequent path prediction and breakpoint recovery.

[0194] Based on the above, this embodiment achieves dynamic construction and updating of the operation history graph by real-time monitoring of URL and DOM changes and combining fingerprint comparison, ensuring that the system can perceive and accurately record the interface flow status in real time.

[0195] Reference Figure 11 In one embodiment, processor 110 executes an internal data interaction flow for an automated execution method of an interactive interface based on interface context representation information. This flow demonstrates how processor 110 coordinates with input / output interface 130 and storage device 120 to complete a full closed loop from instruction reception, context construction, policy generation to execution feedback.

[0196] Specifically, in the instruction receiving stage, as shown in step S1101, the processor 110 receives the execution instruction for the target interactive interface through the input / output interface 130. Next, in step S1102, the processor 110 parses the instruction, extracting the operation intent and target element description. Then, it enters the stage of constructing and maintaining interface context representation information. As shown in steps S1103 and S1104, the processor 110 sends a read request to the storage device 120 and obtains the stored page fingerprint information and operation history graph information. In step S1105, the processor 110 updates the page fingerprint information based on the current interface state collected through the input / output interface 130. In step S1106, the processor 110 uses a graph neural network to model or update the operation history graph information. After construction is completed, as shown in step S1107, the processor 110 writes the updated interface context representation information back to the storage device 120 to ensure the timeliness of the persistent data.

[0197] In the control strategy determination phase, as shown in steps S1108 and S1109, processor 110 re-confirms or reads the interface context representation information. Based on this information, processor 110 determines the target interface element location strategy (such as a multi-level fault-tolerant strategy) in step S1110 and determines the cross-page execution path in step S1111. Subsequently, in step S1112, processor 110 integrates the above decisions to generate a specific control strategy.

[0198] Entering the automated operation execution phase, as shown in step S1113, the processor 110 converts the control strategy into control commands and sends them to the target interactive interface through the input / output interface 130. As shown in step S1114, the input / output interface 130 drives the target interactive interface to execute automated operations, and in step S1115, the operation results (including execution status and a screenshot of the new interface) are returned to the processor 110. Finally, in the characterization information update phase, the processor 110 calculates new page fingerprint information based on the operation results in step S1116, and updates the operation history graph information and page fingerprint information in steps S1117 and S1118 respectively, completing one full automated execution cycle.

[0199] Based on the above, this embodiment realizes real-time reading and writing of context information and dynamic generation of automated execution strategies through data interaction between the processor, storage device and interface, ensuring data consistency, accuracy and system response speed in the automated execution process.

[0200] Reference Figure 12In one embodiment, this disclosure provides an end-to-cloud interaction process for an automated execution system of an interactive interface based on interface context representation information. This process demonstrates a collaboration mechanism between a user terminal 200 and a server 100, where the user terminal 200 is responsible for lightweight data acquisition and execution, while the server 100 is responsible for computationally intensive fusion parsing and strategy generation.

[0201] Specifically, in the multimodal data acquisition and uploading stage, as shown in step S1201, the user terminal 200 acquires multimodal input data for the target interactive interface, including voice, text, images, and video. Simultaneously, as shown in step S1202, the user terminal 200 detects and obtains the current interface state of the target interactive interface, such as a screenshot and document object model structure. Subsequently, as shown in steps S1203 and S1204, the user terminal 200 sends the acquired multimodal input data and the current interface state to the server 100.

[0202] After receiving the data, server 100 enters the multimodal fusion parsing stage. As shown in step S1205, server 100 uses a multimodal fusion algorithm to parse the input data and extract the operation intent and entity information. In step S1206, server 100 generates a unified intent command. Next, it enters the context-aware model stage. As shown in steps S1207 and S1208, server 100 constructs and maintains page fingerprint information and operation history graph information. In step S1209, server 100 models the interface context representation information based on a graph neural network to understand the flow dependencies between interfaces.

[0203] In the control policy generation phase, as shown in step S1210, server 100 determines the control policy based on interface context representation information and the received current interface state. This includes determining the target interface element positioning policy in step S1211, such as selecting a suitable multi-level fault-tolerant policy. Subsequently, in step S1212, server 100 translates the policy into specific control instructions.

[0204] Entering the automated operation execution phase, as shown in step S1213, the server 100 sends control commands back to the user terminal 200. As shown in steps S1214 and S1215, the automated execution module of the user terminal 200 receives the commands and executes the corresponding automated operations on the target interactive interface. Finally, in the information update phase, as shown in step S1216, the user terminal 200 feeds back the operation results to the server 100. In steps S1217 and S1218, the server 100 updates the operation history graph information and page fingerprint information based on the operation results, and in step S1219 performs necessary adaptive or self-healing processing, such as updating the failed element location paths.

[0205] Based on the above, this embodiment uses an edge-cloud collaborative architecture to offload complex context modeling and strategy calculation to the server side, while ensuring the real-time execution of the terminal, thus realizing closed-loop control and dynamic evolution of the automated testing process.

[0206] This disclosure also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various steps of the above-described automated execution method for interactive interfaces based on interface context representation information. The computer-readable storage medium can be a non-transitory computer-readable storage medium, such as a read-only memory, random access memory, optical disc read-only memory, magnetic tape, flash memory, hard disk, or solid-state drive. The computer program includes computer-readable instructions. When the processor 110 executes the computer-readable instructions, it causes the automated execution device 10 to execute the automated execution method for interactive interfaces based on interface context representation information described in the above embodiments, including steps such as receiving execution instructions, constructing interface context representation information, determining control strategies, performing automated operations, and updating representation information.

[0207] More specifically, the main functions and processing procedures in the interactive interface automation execution method based on interface context representation information disclosed in this disclosure can be implemented through software program modules, dedicated computing circuits, or dedicated logic circuits. In the software implementation, each function is defined by computer program code stored in memory, and the processor executes this program code to implement the corresponding function. In the hardware implementation, each function is implemented by a specially designed digital signal processing circuit, a field-programmable gate array, or an application-specific integrated circuit to achieve higher execution efficiency or lower power consumption. In the hybrid implementation, some functions are implemented by software program modules, while others are implemented by dedicated circuits; the two work together to balance flexibility and execution efficiency.

[0208] Regarding the multimodal input parsing function, the speech recognition function can be implemented by a speech recognition module, which is responsible for converting speech signals into text sequences; the natural language processing function can be implemented by a natural language processing module, which is responsible for performing dependency parsing and named entity recognition on the text to extract the operation intent and target element description; the image element recognition function can be implemented by a vision detection module, which is responsible for detecting interactive elements in the image and extracting their visual feature vectors and region coordinates; and the video temporal analysis function can be implemented by a temporal analysis module, which is responsible for detecting interface changes in the video frame sequence and parsing mouse movement trajectories and hover events.

[0209] Regarding the multimodal information fusion function, the spatiotemporal consistency verification function can be implemented by the spatiotemporal alignment module. This module is responsible for retrieving video frames or images that match the input timestamp and spatially mapping the mouse trajectory or hover event with the target element description. The multimodal feature fusion function can be implemented by the feature fusion module. This module is responsible for calculating the attention weights of each modality and performing weighted fusion operations to generate a unified intent feature vector. The fallback retrieval function can be implemented by the fallback retrieval module. This module is responsible for expanding the time window to retrieve candidate page snapshots when the spatiotemporal alignment matching degree is insufficient.

[0210] Regarding the interface context representation information construction function, the visual hash calculation function can be implemented by the visual hash module, which is responsible for extracting structural features from the interface screenshot and generating perceptual hash values; the semantic encoding function can be implemented by the semantic encoding module, which is responsible for semantic vector encoding of the Uniform Resource Locator and title; the page fingerprint generation function can be implemented by the fingerprint generation module, which is responsible for combining the visual hash value and semantic vector to generate page fingerprint information; the operation history graph maintenance function can be implemented by the graph data management module, which is responsible for creating graph nodes, establishing directed connections, and associating and storing edge attribute features; and the graph neural network modeling function can be implemented by the graph neural network inference module, which is responsible for performing message passing computation on the operation history graph to learn the context representation of nodes.

[0211] Regarding the target interface element location function, the structural path location function can be implemented by the structural location module, which is responsible for performing element path queries based on the document object model structure or control tree structure; the visual region location function can be implemented by the region location module, which is responsible for searching for matching elements in the current interface based on historical visual coordinates or region information; the semantic feature matching function can be implemented by the semantic matching module, which is responsible for calculating the semantic similarity between the execution instruction and the candidate element using a contrastive language-image pre-trained model; the location stability evaluation function can be implemented by the stability evaluation module, which is responsible for calculating the stability score of the target element in the semantic, positional, and structural dimensions based on historical records; and the location strategy scheduling function can be implemented by the strategy scheduling module, which is responsible for dynamically adjusting the execution priority of each level of location strategy according to the stability score.

[0212] Regarding the real-time environment monitoring function, the Uniform Resource Locator (URL) monitoring function can be implemented by the URL monitoring module, which is responsible for monitoring the address changes of the target interactive interface and triggering corresponding events; the Document Object Model (DOM) monitoring function can be implemented by the structure change monitoring module, which is responsible for monitoring the node changes of the DOM tree and determining whether a page jump has occurred; and the interface status acquisition function can be implemented by the status acquisition module, which is responsible for acquiring screenshots and structural information of the current interface for subsequent processing.

[0213] Regarding session management functions, the session state persistence function can be implemented by the state persistence module, which is responsible for storing the interface context representation information and execution progress to non-volatile storage media; the session recovery function can be implemented by the session recovery module, which is responsible for calculating the page fingerprint of the current interface and comparing it with historical graph nodes after detecting a session interruption and restarting to determine the recovery point; the optical character recognition auxiliary function can be implemented by the optical character recognition module, which is responsible for extracting the interface title text to assist node positioning when the fingerprint matching degree is insufficient.

[0214] Regarding cross-page execution control functionality, the operation dependency analysis function can be implemented by the dependency analysis module, which is responsible for parsing execution instructions to extract the dependencies between each operation step; the execution path planning function can be implemented by the path planning module, which is responsible for predicting and generating cross-page execution paths based on the node connection relationships in the operation history graph; the control strategy generation function can be implemented by the strategy generation module, which is responsible for determining the control strategy required for execution by comprehensively considering the interface context representation information and the current interface state; and the control instruction conversion function can be implemented by the instruction conversion module, which is responsible for converting abstract control strategies into underlying control instructions that can be recognized by the target interactive interface.

[0215] Regarding information update and adaptive functions, the operation result processing function can be implemented by the result processing module, which is responsible for receiving and parsing the execution results of automated operations and the new interface state; the representation information update function can be implemented by the information update module, which is responsible for updating the operation history graph information and page fingerprint information based on the operation results; the adaptive repair function can be implemented by the adaptive repair module, which is responsible for obtaining the latest structural path information of the target element and updating it to the interface context representation information after successfully locating it using the fault tolerance strategy; and the exception handling function can be implemented by the exception handling module, which is responsible for generating an error report containing an interface snapshot and missing information prompts and processing user feedback when locating fails.

[0216] The aforementioned functional modules are logically independent, but their implementation can be flexibly combined according to specific deployment requirements. In resource-constrained embedded environments, computationally intensive modules (such as the neural network inference module and semantic matching module) can be implemented using dedicated acceleration circuits, while logic control modules can be implemented using software programs. In cloud deployment environments, each module can be implemented as an independent microservice component, communicating and collaborating through network interfaces, thereby supporting elastic scaling and distributed processing. This modular architecture enables the technical solution disclosed herein to adapt to diverse hardware platforms and application scenarios.

[0217] In summary, the interactive interface automated execution method, apparatus, and system provided in this disclosure, based on interface context representation information, enable the automated execution system to possess a deep perception capability of interface state and operation flow logic by constructing and maintaining interface context representation information including page fingerprint information and operation history graph information. Compared with traditional automation solutions that rely solely on a fixed document object model structure, this disclosure utilizes a multi-level fault-tolerant positioning strategy and semantic feature matching technology to dynamically adjust the positioning method and achieve adaptive repair of the control strategy when the target interactive interface is redesigned or element attributes are changed. Furthermore, combined with the path prediction and session recovery mechanism of the operation history graph, this disclosure effectively solves the problem of state continuity after cross-page business process interruption, improving the execution stability and maintenance efficiency of automated scripts in complex dynamic environments without frequent manual intervention.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automating the execution of an interactive interface based on interface context representation information, characterized in that, Includes the following steps: Receive execution instructions for the target interactive interface; Construct and maintain the interface context representation information of the target interactive interface, which includes page fingerprint information for representing the interface state and operation history graph information for modeling the operation flow dependency relationship between interfaces. Based on the interface context representation information and the current interface state, a control strategy required to execute the execution instruction is determined, and the control strategy includes at least a target interface element positioning strategy. as well as The control strategy is used to perform automated operations on the target interactive interface, and the operation history graph information and the page fingerprint information are updated based on the operation results.

2. The method according to claim 1, characterized in that, The step of receiving the execution instruction for the target interactive interface includes: Receive input data in at least two modalities, the input data including at least two of voice, text, image and video; The input data is parsed using a multimodal feature fusion algorithm to generate the execution instructions, wherein the execution instructions include action type and target interface element description.

3. The method according to claim 2, characterized in that, The step of parsing the input data using the multimodal feature fusion algorithm includes one or more of the following: Obtain the input timestamp of the voice or text, and retrieve the video frame or image that matches the input timestamp within a preset time window; The mouse trajectory coordinates or hover events are obtained from the video. Align the mouse trajectory coordinates or hover events in the video with the target interface element descriptions in the input data.

4. The method according to claim 1, characterized in that, The steps for constructing and maintaining the interface context representation information include: A visual hash value is generated based on the structural features of the screenshot, and a semantic vector is generated based on the interface identification information of the interface. The visual hash value and the semantic vector are combined to generate the page fingerprint information, wherein the interface identification information includes at least one of a Uniform Resource Locator, a title, or an application page identifier. Graph nodes are constructed based on the page fingerprint information, and directed connections between different graph nodes are built based on user operations to generate the operation history graph information; and The visual coordinates or region information corresponding to the user operation are associated and stored as attribute features of the directed connection relationship; wherein the operation history graph information is modeled using a graph neural network.

5. The method according to claim 4, characterized in that, The control strategy also includes determining cross-page execution paths, including: The graph neural network is used to analyze the node connection relationships in the operation history graph information; The execution instructions are parsed to extract operation dependencies, and the operation dependencies are mapped to the corresponding nodes in the operation history graph information; Based on the connection relationship of the corresponding nodes, the cross-page execution path containing the page flow order is predicted and generated.

6. The method according to claim 4, characterized in that, The step of determining the target interface element positioning strategy includes generating a multi-level fault-tolerant positioning mechanism, wherein the multi-level fault-tolerant positioning mechanism executes the following strategies in order of priority: The first-level strategy includes: path positioning based on interface structure information, wherein the interface structure information includes the document object model structure of the web page interface or the control tree structure of the application interface; the second-level strategy includes: when the higher-priority first-level strategy fails, region positioning is performed based on the historical visual coordinates or region information recorded in the operation history graph information. The third-level strategy includes: when the higher-priority first-level strategy fails, positioning is performed based on semantic feature matching, and the target interface element is locked by calculating the semantic similarity between the execution instruction and the candidate elements in the current interface screenshot.

7. The method according to claim 6, characterized in that, The semantic feature matching in the third-level strategy is achieved using a contrastive language-image pre-trained model, and the method further includes: After successfully executing the operation using the second-level strategy or the third-level strategy, the latest interface structure path information of the target interface element is obtained, and the interface context representation information is updated based on the latest interface structure path information to achieve adaptive repair of the target interface element positioning strategy.

8. The method according to claim 6, characterized in that, The method further includes: Calculate the positioning stability score of the target interface element. The positioning stability score is calculated based on at least one of semantic stability, positional stability and structural stability. The semantic stability, positional stability or structural stability is calculated based on the historical records of the operation history graph information. The execution priority of each strategy in the multi-level fault-tolerant positioning mechanism is dynamically adjusted based on the positioning stability score, or the strategy that meets the score threshold is directly selected as the preferred strategy.

9. The method according to claim 4, characterized in that, The step of constructing and maintaining the interface context representation information also includes real-time environment monitoring: Monitor changes in the Uniform Resource Locator (URL) or Document Object Model (DOM) structure of the target interactive interface; When a change is detected, determine whether a page redirection occurs on the current screen; If a page redirect occurs, the page fingerprint information of the redirected interface is calculated in real time, and the operation history graph information is updated.

10. The method according to claim 1, characterized in that, The control strategy also includes determining the session recovery point, including: After detecting an interruption and restart of the automated execution session, the page fingerprint information of the current interface is calculated in real time; The page fingerprint information of the current interface is compared with the nodes in the operation history graph information to locate the current execution progress; The location of the successfully matched node is determined as the session recovery point, and the unexecuted subsequent operation instructions are loaded based on the operation history graph information, wherein the unexecuted subsequent operation instructions are the operation instructions corresponding to the operation sequence to be executed recorded in the operation history graph information.

11. The method according to claim 10, characterized in that, The step of comparing the page fingerprint information of the current interface with the nodes in the operation history graph information includes: Calculate the degree of matching between the page fingerprint information of the current interface and each node in the operation history graph information; When the matching degree is higher than a preset threshold, the node with the highest matching degree is determined as the node position corresponding to the current interface; When the matching degree is lower than the preset threshold, optical character recognition is performed on the current interface to extract the interface title text. The interface title text is then matched with the title information associated with each node in the operation history graph information, and the node position corresponding to the current interface is determined based on the matching result.

12. The method according to claim 1, characterized in that, The method further includes: When the control strategy fails to successfully perform automated operations on the target interactive interface, an error report is generated that includes a snapshot of the current interface and a prompt indicating missing location information. Receive feedback instructions for the error report, and update the interface context representation information based on the feedback instructions.

13. The method according to claim 3, characterized in that, When the alignment matching degree is lower than the threshold, candidate page snapshots are retrieved within the extended time window, and the target page snapshot is selected based on page similarity to re-execute the target interface element positioning strategy.

14. The method according to claim 6, characterized in that, The semantic features of the candidate elements are generated based on at least one of the following: interface control text, accessibility attribute text, and text obtained by optical character recognition.

15. The method according to claim 8, characterized in that, The positional stability is obtained based on the statistical analysis of the visual coordinate offset of the target interface elements in the historical records, and the structural stability is obtained based on the statistical analysis of the consistency ratio of the interface structure path.

16. The method according to claim 12, characterized in that, The feedback instructions include one of the following: target interface element confirmation result, candidate element selection information, supplementary target interface element description information, and retry strategy selection information.

17. An automated execution device for an interactive interface based on interface context representation information, characterized in that, include: Memory, used to store computer programs; A processor, coupled to the memory, is configured to execute the computer program to perform the following steps: Receive execution instructions for the target interactive interface; Construct and maintain the interface context representation information of the target interactive interface, which includes page fingerprint information for representing the interface state and operation history graph information for modeling the operation flow dependency relationship between interfaces. Based on the interface context representation information and the current interface state, a control strategy required to execute the execution instruction is determined, and the control strategy includes at least a target interface element positioning strategy. as well as The control strategy is used to perform automated operations on the target interactive interface, and the operation history graph information and the page fingerprint information are updated based on the operation results.

18. An automated execution system for an interactive interface based on interface context representation information, characterized in that, include: User terminal; as well as The server is communicatively connected to the user terminal. The user terminal is used for: Collect multimodal input data for the target interactive interface and send it to the server; as well as Receive control commands from the server and perform automated operations on the target interactive interface; The server mentioned above is used for: The system receives the multimodal input data from the user terminal and uses a multimodal fusion algorithm to parse and generate intent commands. Construct and maintain the interface context representation information of the target interactive interface, which includes page fingerprint information for representing the interface state and operation history graph information for modeling the operation flow dependency relationship between interfaces. Based on the interface context representation information and the current interface state uploaded by the user terminal, a control strategy required to execute the intent instruction is determined, and the control strategy includes at least a target interface element positioning strategy. as well as The control command is generated based on the control strategy and sent to the user terminal. The operation history graph information and the page fingerprint information are updated according to the operation results fed back by the user terminal.

Images