Interface operation method, device, apparatus and computer readable medium

Abstract

The application discloses an interface operation method and device, equipment and a computer readable medium, and relates to the technical field of computers. A specific embodiment of the method comprises: in response to a task request in the form of a natural language task description sent by a terminal, sending a screenshot request to the terminal; receiving a first screenshot returned by the terminal, using a graphical user interface intelligent agent to perform multi-modal semantic understanding on the first screenshot, identifying an interface element and a function of the interface element; combining a preset knowledge base to generate an operation planning sequence based on the interface element and the function of the interface element, and sending an atomic operation instruction converted from the operation planning sequence to implement interface operation on the terminal by executing the atomic operation instruction. The embodiment can cope with complex and variable terminal interface environments.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an interface operation method, apparatus, device, and computer-readable medium. Background Technology

[0002] A Graphical User Interface Agent (GUI Agent) is an artificial intelligence technology that uses a multimodal large model to understand screen content and automatically execute sequences of operations. This technology is primarily applied in automated service scenarios on PCs and mobile devices, enabling users to perform repetitive tasks such as service applications, approval forms, and information entry.

[0003] In recent years, with the rapid development of Multimodal Large Language Model (MLLM), GUI Agent technology has made significant progress.

[0004] In the process of realizing this invention, the inventors discovered that the prior art has at least the following problems: because the terminals involve different platforms, the GUI Agent has difficulty coping with the complex and ever-changing terminal interface environment. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an interface operation method, apparatus, device, and computer-readable medium that can cope with complex and ever-changing terminal interface environments.

[0006] To achieve the above objectives, according to one aspect of the present invention, a user interface operation method is provided, applied to a server, comprising: In response to a task request sent by a terminal in the form of a natural language task description, a screenshot request is sent to the terminal. The system receives a first screenshot transmitted back from the terminal, and uses a graphical user interface agent to perform multimodal semantic understanding on the first screenshot to identify interface elements and their functions. Based on the interface elements and their functions, and using a preset knowledge base, an operation planning sequence is generated, and atomic operation instructions converted from the operation planning sequence are sent to the terminal to execute the atomic operation instructions and realize the interface operation.

[0007] The method further includes: A second screenshot of the terminal after executing the atomic operation instruction is obtained. The second screenshot is compared with the first screenshot to evaluate the instruction execution effect. Based on the instruction execution effect, it is determined whether to regenerate the operation planning sequence.

[0008] The step of determining whether to regenerate the operation planning sequence based on the execution effect of the instruction includes: If the execution effect of the instruction is that the atomic operation instruction does not achieve the expected effect, then the operation planning sequence is regenerated based on the second screenshot; If the execution effect of the instruction is that the atomic operation instruction achieves the expected effect, and the task request is not completed, then the next operation planning sequence is generated based on the second screenshot; If the execution of the instruction results in the task request being completed, a task success message is sent to the terminal.

[0009] The step of using a graphical user interface (GUI) agent to perform multimodal semantic understanding on the first screenshot, and identifying interface elements and their functions, includes: The first screenshot is input into a graphical user interface agent, which identifies the buttons, input boxes, list items, and text content in the first screenshot, and identifies the semantic functions of the buttons, input boxes, list items, and text content in the business scenario corresponding to the task request.

[0010] Sending a screenshot request to the terminal includes: sending a screenshot request to the terminal through a WebSocket full-duplex communication channel established with the terminal; Sending the atomic operation instruction transformed from the operation planning sequence includes: sending the atomic operation instruction transformed from the operation planning sequence through the WebSocket full-duplex communication channel; And / or, The method further includes: In response to a user takeover request sent by the terminal, the sending of the atomic operation instruction is stopped, so as to return control of the interface operation to the user corresponding to the terminal; And / or, The task request includes service application, enterprise approval form filling or information entry; The first screenshot and the second screenshot are screenshots of the mobile application interface.

[0011] According to a second aspect of the present invention, a user interface operation method is provided, applied to a terminal, comprising: Send a task request described in natural language; Receive and respond to a screenshot request to obtain a first screenshot of the terminal, and send the first screenshot; The system receives atomic operation instructions and executes them to perform interface operations. The atomic operation instructions are derived from the operation planning sequence generated from the first screenshot. The operation planning sequence is determined by combining a preset knowledge base with a graphical user interface agent, based on the interface elements of the first screenshot and the functions of the interface elements.

[0012] The execution of the atomic operation instructions to implement interface operations includes: The runtime environment is determined based on the instruction type and target parameters in the atomic operation instructions; If the runtime environment is WebView or H5, the operation corresponding to the atomic operation instruction is executed on the page DOM element through the extended CDP protocol. If the runtime environment is a native application, the operation corresponding to the atomic operation instruction is executed through the accessibility service or automation framework of the operating system.

[0013] According to a third aspect of the present invention, a user interface operation device is provided, applied to a server, comprising: The task module is used to send a screenshot request to the terminal in response to a task request described in natural language. The intelligent agent module is used to receive the first screenshot transmitted back by the terminal, and use the graphical user interface intelligent agent to perform multimodal semantic understanding on the first screenshot to identify interface elements and the functions of the interface elements. The knowledge module is used to combine a preset knowledge base with the interface elements and their functions to generate an operation planning sequence, and send atomic operation instructions converted from the operation planning sequence to execute the atomic operation instructions on the terminal to realize interface operations.

[0014] According to a fourth aspect of the present invention, an interface operation device is provided, applied to a terminal, comprising: The communication module is used to send task requests described in natural language; The screenshot acquisition module is used to receive and respond to a screenshot request to acquire a first screenshot of the terminal, and send the first screenshot. An operation execution module is used to receive atomic operation instructions and execute the atomic operation instructions to realize interface operation. The atomic operation instructions are converted from the operation planning sequence generated by the first screenshot. The operation planning sequence is determined by combining a preset knowledge base with a graphical user interface agent based on the interface elements of the first screenshot and the functions of the interface elements.

[0015] According to a fifth aspect of the present invention, an interface-operated electronic device is provided, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0016] According to a sixth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described above.

[0017] One embodiment of the above invention has the following advantages or beneficial effects: In response to a task request described in natural language sent by a terminal, a screenshot request is sent to the terminal; a first screenshot returned by the terminal is received, and a graphical user interface agent is used to perform multimodal semantic understanding on the first screenshot to identify interface elements and their functions; based on the interface elements and their functions, an operation planning sequence is generated using a preset knowledge base, and atomic operation instructions converted from the operation planning sequence are sent to the terminal to execute the atomic operation instructions to realize interface operation. By using screenshot recognition to realize interface operation, it can cope with complex and ever-changing terminal interface environments.

[0018] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0019] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the main flow of the interface operation method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating whether or not to regenerate the operation planning sequence according to an embodiment of the present invention; Figure 3 This is a timing diagram of the WebSocket communication process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the main flow of another interface operation method according to an embodiment of the present invention; Figure 5 This is a flowchart illustrating the implementation of interface operations according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the main structure of the interface operation device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the main structure of another interface operation device according to an embodiment of the present invention; Figure 8 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 9This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0020] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] Currently, GUI Agent technology solutions are mainly divided into three categories: ADB solutions, simulated user operation solutions, and DOM operation solutions.

[0022] The ADB (Android Debug Bridge) solution obtains user screen information and operates the entire phone through the Android developer mode's debug protocol. It captures screenshots and simulates user input events via a debug interface, but this requires enabling USB debugging mode in the device's developer options and obtaining high-level system privileges, leading to serious security and privacy risks. Furthermore, some phone manufacturers have strictly limited these permissions. In addition, the ADB solution requires running on a computer or deep integration with the phone manufacturer, and cannot run independently on ordinary user mobile devices.

[0023] The simulated user operation solution is a downgraded version of the ADB solution. Its core idea is to only read screen information and perform phone operations within the application itself, without involving system-level debugging interfaces. This solution reads the layout information of application interface elements, such as Android's View Hierarchy or iOS's UI Automation tree, and combines this with coordinate simulation technology to implement click, swipe, and input operations. Specific implementation methods include obtaining interface element information through the AccessibilityService and simulating touch and button events using the Instrumentation framework. Its main drawbacks are high power consumption due to the need for continuous background service monitoring of interface changes; and low accuracy, frequently resulting in accidental touches and inaccurate swipe distances. This solution also exhibits significant instability and reliability issues when dealing with complex interface structures, such as nested scroll views and overlapping pop-ups.

[0024] DOM manipulation solutions, exemplified by Browser Use, rely on the core idea of ​​communicating with the browser via the Chrome DevTools Protocol (CDP) to read and manipulate web page content. This approach has been validated in desktop browser environments, accurately locating UI elements by parsing the DOM tree and simulating mouse clicks and keyboard input. Compared to user-simulated operation solutions, DOM manipulation consumes less power and offers higher precision because it directly manipulates DOM nodes rather than relying on coordinate simulation. However, the feasibility of this approach in mobile WebView mode has not been fully verified. WebView is a browser component embedded within native applications on mobile devices, and its supported CDP protocol range differs from that of desktop Chrome.

[0025] Based on the above analysis, the following main drawbacks can be identified: Excessive reliance on permissions The ADB solution requires advanced system debugging permissions on the device, while the user simulation solution requires accessibility service permissions. This over-reliance on high permissions not only poses serious security and privacy risks but also renders the technology inoperable on some devices. Some mobile phone manufacturers strictly limit high-permission requests from third-party applications, requiring users to manually enable USB debugging or grant special permissions in developer options. On iOS devices, due to even stricter system security restrictions, the viability of the above solutions is further reduced. This dependency on permissions severely restricts the cross-device and cross-platform deployment capabilities of GUI Agent technology, preventing the technology from being used out of the box.

[0026] It is difficult to balance power consumption and accuracy.

[0027] There is an irreconcilable conflict between power consumption control and operational accuracy. Simulating user operations involves continuously monitoring interface changes to achieve simulation. While the accuracy generally meets requirements, continuous background operation leads to significant power consumption, drastically reducing the user's phone's battery life. The DOM manipulation solution, although power-efficient, is limited by the WebView's CDP protocol support, and its feasibility in mobile environments has not been fully validated. The MobileAgent solution balances a certain level of accuracy and functional completeness, but its large number of model parameters places high demands on device computing resources, resulting in significant performance bottlenecks on low- to mid-range mobile devices.

[0028] Poor cross-platform consistency

[0029] The implementation of the solution varies significantly across different mobile operating system platforms. The implementation mechanisms of the ADB solution on Android and iOS are completely different. This lack of cross-platform consistency forces development teams to maintain multiple technical implementations for different platforms, increasing development and maintenance costs. Furthermore, the performance of the GUIAgent may differ when users switch between different operating systems.

[0030] Lack of semantic understanding ability

[0031] Most technical solutions remain at the level of identifying and locating interface elements, lacking a deep understanding of the semantic content of the GUI. This makes it easy to make mistakes when dealing with complex interface structures, such as dynamically loaded content, conditional rendering, and multi-step forms. The lack of semantic understanding also limits the performance of the GUI agent in open-ended tasks.

[0032] Because the terminals involve different platforms, the GUI Agent has difficulty coping with the complex and ever-changing terminal interface environment.

[0033] To cope with the complex and ever-changing terminal interface environment, the following technical solutions in the embodiments of the present invention can be adopted.

[0034] See Figure 1 , Figure 1 This is a schematic diagram of the main flow of the interface operation method according to an embodiment of the present invention. Specifically, it includes the following steps: S101. In response to the task request sent by the terminal in the form of a task description in natural language, send a screenshot request to the terminal.

[0035] The technical solutions of this invention can be applied to automated service scenarios, such as service applications, enterprise approval forms, and information entry, which involve repetitive tasks performed on a terminal application interface. Service applications include social security inquiries, housing provident fund withdrawals, or license processing. Enterprise approval forms include annual business reports, tax declarations, or qualification reviews. Information entry includes personal information registration, questionnaires, or form entry. Figure 1 The server is the main entity executing each step in the process, and the corresponding terminal can be a mobile device or a personal computer.

[0036] The server receives task requests sent by the terminal, which are described in natural language. For example, task requests may include service applications, enterprise approval forms, or information entry.

[0037] Task requests are described in natural language, specifying the task the user wants to complete. Examples include: "Help me complete my housing provident fund withdrawal application" or "Fill out this company's annual business report." Using natural language to describe tasks eliminates the need for users to manually record scripts or preset rules; they can initiate interface operations simply by expressing their intent, much like conversing with a human, thus lowering the barrier to entry.

[0038] After receiving the task request, the server does not immediately begin execution. Instead, it first sends a screenshot request to the terminal. The purpose is that the server needs to obtain the current state of the terminal's interface. This is because the terminal's interface may be in different states, such as the application's home screen, in the middle of a data entry process, or with pop-up windows overlaying it.

[0039] S102. Receive the first screenshot returned by the terminal, and use the graphical user interface intelligent agent to perform multimodal semantic understanding on the first screenshot to identify interface elements and their functions.

[0040] After receiving the first screenshot from the terminal, the server inputs it into the graphical user interface (GUI) agent for processing. As an example, the GUI agent can include an intelligent engine, a government affairs agent, or an AI agent. The GUI agent integrates a multimodal big model to perform multimodal semantic understanding of the first screenshot. It can not only identify interface elements such as buttons, input boxes, lists, and text in the screenshot, but also understand the semantic functions of these elements. The multimodal big model includes a government affairs big model.

[0041] Multimodal semantic understanding enables the understanding of the function of UI elements in specific business scenarios. For example, it can identify whether a button is a submit button or a cancel button, or whether an input field is a name input field or an ID number input field.

[0042] S103. Based on the preset knowledge base and the functions of the interface elements, generate an operation planning sequence and send atomic operation instructions transformed from the operation planning sequence to execute the atomic operation instructions on the terminal to realize the interface operation.

[0043] The server generates an operation planning sequence based on a pre-set knowledge base. The pre-set knowledge base includes operation manuals, component definition documents, and related documents pre-entered by operations personnel, providing domain knowledge support for planning decisions. For example, the pre-set knowledge base may include a government affairs knowledge base.

[0044] Specifically, the graphical user interface (GUI) agent retrieves structured knowledge related to the current task from a pre-defined knowledge base based on the business type corresponding to the task request. For example, when the task request is to process housing provident fund withdrawal, the GUI agent retrieves the pre-stored housing provident fund withdrawal operation manual from the pre-defined knowledge base to obtain the standard operation process, such as first step, clicking the housing provident fund withdrawal entry; second step, selecting the withdrawal type; and third step, entering the withdrawal amount. Simultaneously, the GUI agent matches the component definition documents retrieved from the knowledge base with the interface elements identified in the first screenshot to determine which button on the current interface corresponds to the housing provident fund withdrawal entry and which input box corresponds to the withdrawal amount field, thereby generating an operation planning sequence that conforms to the current business specifications.

[0045] After the operation plan sequence is generated, the graphical user interface (GUI) agent is responsible for converting the operation plan sequence into specific atomic operation instructions. An atomic operation instruction is an indivisible basic operation unit. After conversion, the GUI agent sequentially sends the atomic operation instructions to the terminal. The server's GUI agent maintains the operation queue and sends the atomic operation instructions sequentially according to the order of the operation plan sequence.

[0046] After receiving an atomic operation instruction, the terminal parses the instruction type and target parameters, and executes the corresponding operation on the application interface. Through this process, atomic operation instructions are executed on the terminal to complete interface operations, thus replacing manual operations by the user on the application interface.

[0047] exist Figure 1 In this embodiment, in response to a task request described in natural language sent by the terminal, a screenshot request is sent to the terminal; a first screenshot returned by the terminal is received; a graphical user interface agent is used to perform multimodal semantic understanding on the first screenshot to identify interface elements and their functions; based on the interface elements and their functions, an operation planning sequence is generated using a preset knowledge base; and atomic operation instructions converted from the operation planning sequence are sent to the terminal to execute the atomic operation instructions to achieve interface operation. By using screenshot recognition to achieve interface operation, it can cope with complex and ever-changing terminal interface environments.

[0048] In one embodiment of the present invention, a second screenshot of the terminal after executing an atomic operation instruction is obtained, the second screenshot is compared with the first screenshot to evaluate the instruction execution effect, and the operation planning sequence is regenerated based on the instruction execution effect.

[0049] Specifically, after the terminal executes the atomic operation command issued by the server, the server sends a new screenshot request to the terminal and receives a second screenshot from the terminal after the execution of the atomic operation command. The second screenshot reflects the latest state of the terminal interface after the execution of the atomic operation command and serves as the basis for evaluating whether the operation has achieved the expected results.

[0050] After the server obtains the second screenshot, it analyzes and compares it with the previous first screenshot. By comparing the changes in the screenshots before and after the operation, it determines whether the operation achieved the expected effect. The evaluation criteria include the operation result reported by the terminal, which includes success or failure, and the second screenshot. For example, if the previous atomic operation instruction was clicking the housing provident fund withdrawal button, comparing the first and second screenshots determines whether the interface has jumped from the current page to the next level page for housing provident fund withdrawal. If the previous atomic operation instruction was entering text in the withdrawal amount input box, comparing the changes in the content of the input box in the two screenshots determines whether the text has been correctly entered.

[0051] By using a before-and-after comparison evaluation method, the effectiveness of command execution is automatically verified. Screenshot comparison evaluation enables the server to determine the outcome of command execution.

[0052] See Figure 2 , Figure 2 This is a flowchart illustrating whether or not to regenerate the operation planning sequence according to an embodiment of the present invention. Specifically, it includes the following steps: S201. If the execution effect of the instruction is that the atomic operation instruction does not achieve the expected effect, then the operation planning sequence is regenerated based on the second screenshot.

[0053] If, after comparing the second screenshot with the first screenshot, the interface state does not change as expected (e.g., the page does not redirect after clicking a button, the list does not scroll after swiping, or the input box is empty after text input), then the atomic operation instruction is determined to have failed to achieve the expected effect.

[0054] Possible reasons why atomic operation instructions may not achieve the expected results include: deviation between the actual position of the interface element and the position identified during planning, pop-up windows obscuring the target element during the operation, page loading delays causing elements to become uninteractive, or dynamic changes occurring in the interface.

[0055] In this scenario, the information indicating the failure of the atomic operation instruction is sent back to the server, triggering the regeneration of the operation planning sequence. Using the received second screenshot as new input, multimodal semantic understanding is re-performed to identify the interface elements and their functions on the current screen, and a new operation planning sequence is generated based on a pre-defined knowledge base.

[0056] The above-mentioned operation planning sequence is triggered to regenerate based on the effect of instruction execution, which ensures that it has the ability to self-correct when facing complex interfaces. It can effectively overcome the shortcomings of existing solutions in terms of stability and reliability when facing complex interface structures such as nested scroll views and pop-up overlays.

[0057] Additionally, the number of times the operation plan sequence is regenerated is set, i.e., the maximum number of steps for task execution is limited by PHONE_AGENT_MAX_STEPS. If the instruction execution result is that the atomic operation instruction does not achieve the expected effect, and the current task execution step count has reached the maximum step limit, then the task is considered to have failed and a TASK_ERROR message is sent to the terminal. The maximum step limit acts as a circuit breaker mechanism for task execution, preventing infinite loops.

[0058] S202. If the execution effect of the instruction is that the atomic operation instruction achieves the expected effect, and the task request is not completed, then generate the next operation planning sequence based on the second screenshot.

[0059] By comparing the second screenshot with the first screenshot, it was confirmed that the previous atomic operation command had achieved the expected results, such as the page successfully redirecting to the target page after the button was clicked, the text in the input box being filled in correctly, and the swipe operation reaching the target area. However, according to the complete business process of the task request, there are still subsequent steps to be executed. For example, the housing provident fund withdrawal process usually includes multiple steps such as clicking the entry point, selecting the withdrawal type, filling in the amount, confirming information, and submitting.

[0060] If the task request is not completed, multimodal semantic understanding continues based on the second screenshot, and the subsequent steps of the business process in the preset knowledge base are combined to generate the next operation planning sequence. The new operation planning sequence is then converted into atomic operation instructions and sent to the terminal. Thus, the graphical user interface intelligent agent can handle complex business processes such as multi-step forms, gradually replacing manual operations on the user interface.

[0061] S203. If the result of the instruction execution is that the task request has been completed, then send a task success message to the terminal.

[0062] Based on the execution results of the instructions and the completion status of the task request, it is determined that all operational steps involved in the current task request have been successfully executed and the task objective has been achieved. For example, the form has been successfully submitted, the application has been submitted to the approval system, and the information has been filled in and saved. A task success message is sent to the terminal to notify that the task has been successfully completed.

[0063] exist Figure 2 In the embodiments described, the automatic execution of task requests is achieved by utilizing the effect of instruction execution, thereby improving the execution efficiency of task requests.

[0064] In one embodiment of the present invention, a first screenshot is input into a graphical user interface agent, which identifies buttons, input boxes, list items, and text content in the first screenshot and identifies the semantic functions of the buttons, input boxes, list items, and text content in the business scenario corresponding to the task request.

[0065] The graphical user interface (GUI) agent integrates a multimodal large model, enabling comprehensive visual understanding of screenshots. During the recognition process, the GUI agent first identifies various UI elements in the initial screenshot, including buttons, input fields, list items, and text content.

[0066] Building upon its ability to identify interface element types, the graphical user interface (GUI) agent can understand the actual purpose and semantics of these elements within specific business scenarios. For example, in an automated service application scenario, when the task request is to process housing provident fund withdrawal, the GUI agent not only identifies multiple buttons in the first screenshot but also identifies which button is the housing provident fund withdrawal entry point, which button returns to the previous level, and which button provides online consultation. Similarly, when recognizing input boxes, it not only identifies their existence but also identifies which input box is for withdrawing funds, which is for displaying an ID number, and which is for displaying a bank card number. Through the semantic understanding capabilities of the GUI agent, it can comprehend the semantic functions of buttons, input boxes, list items, and text content within the business scenario corresponding to the task request.

[0067] In the above embodiments, when faced with a complex interface structure, the graphical user interface agent can accurately determine the current state of the interface and the next operation to be performed based on its understanding of the semantic functions of the interface elements, effectively avoiding misoperation.

[0068] In one embodiment of the present invention, a full-duplex communication protocol is established between the server and the terminal, allowing the terminal and the server to conduct bidirectional real-time message passing through a WebSocket full-duplex communication channel on a single TCP connection.

[0069] The server sends a screenshot request to the terminal via a WebSocket full-duplex communication channel. Upon receiving the screenshot request, the terminal performs the screenshot operation. Screenshot modes support full-screen screenshots, in-application screenshots, and partial screenshots. Figure 3In this mode, after taking a screenshot, it is compressed into JPEG format and encoded using Base64, then transmitted back to the server via the same WebSocket full-duplex communication channel. The screenshot resolution can be adaptively adjusted according to network bandwidth and task requirements, reducing data transmission volume while ensuring the understanding accuracy of the graphical user interface agent. The use of a WebSocket full-duplex communication channel significantly reduces communication latency and resource consumption. Compared to traditional HTTP polling or one-way push solutions, it enables real-time interaction between screenshot requests and screenshot data, ensuring timely server-side awareness of the terminal interface status.

[0070] After the server's graphical user interface (GUI) agent transforms the operation planning sequence into atomic operation instructions, it sends these instructions to the terminal in real time via the established WebSocket full-duplex communication channel. The GUI agent maintains an operation queue, sending instructions sequentially according to the planning sequence. Each instruction is accompanied by a unique operation identifier for association and tracking. Sending instructions via the WebSocket full-duplex communication channel ensures that the operation instructions reach the terminal in real time, and the terminal can immediately send back the execution results through the same channel after execution, forming a highly efficient instruction response loop.

[0071] See Figure 3 , Figure 3 This is a timing diagram of the WebSocket communication process according to an embodiment of the present invention.

[0072] Figure 3 It includes the following core interactive entities: client, server, GUI intelligent engine, and knowledge base. The client is the terminal described in the above embodiments, and the GUI intelligent engine is the graphical user interface intelligent agent.

[0073] The client sends a task request (TASK_REQUEST) to the server. The task request is used to initiate a UI operation. After receiving the task request, the server returns a task confirmation message (TASK_CONFIRMED) to the client, informing the client that the task has been successfully created and is ready.

[0074] The server sends a `SCREENSHOT_REQUEST` message to the client, requesting a screenshot of the current screen. The client sends back the screenshot data via a `SCREENSHOT_RESPONSE` message. Based on the screenshot, the server performs GUI semantic analysis and operation planning, sending specific operation instructions to the client, such as click (CLICK), long press (LONG_PRESS), swipe (SWIPE), text input (TEXT_INPUT), and double-click (DOUBLE_CLICK).

[0075] After the client performs the operation, it reports the execution result through the ACTION_RESULT message. The above loop of screenshot request-operation issuance-result reporting continues until the task is completed or the maximum number of steps (PHONE_AGENT_MAX_STEPS) is reached.

[0076] After the task is completed, the server sends a TASK_SUCCESS message to notify the client. If an error occurs during execution, a TASK_ERROR message is sent.

[0077] In one embodiment of the present invention, the terminal can send a takeover request (TAKE_OVER) message to the server via a WebSocket full-duplex communication channel. Upon receiving the takeover request, the server's graphical user interface agent immediately suspends the issuance of atomic operation instructions and returns control to the user.

[0078] This feature allows users to intervene at any time during task execution, suitable for scenarios such as: users wanting to manually adjust certain input information during automated processes; graphical user interface agents encountering unmanageable anomalies requiring manual decision-making; or users having specific needs for the current operation step that they wish to complete independently. It solves the problem of inflexible response in abnormal scenarios, ensuring automation efficiency while maintaining user autonomy.

[0079] In one embodiment of the present invention, both the first screenshot and the second screenshot are screenshots of the mobile application interface, that is, image data actually collected from the mobile device during task execution that reflects the state of the mobile application interface. For example, in a service application scenario, the first screenshot is a screenshot of the service list page of the service app, and the second screenshot is a screenshot of the information filling page that is accessed after clicking on a service entry. In an enterprise approval and form filling scenario, the first screenshot is a screenshot of the initial state of the approval form, and the second screenshot is a screenshot of the form state after some fields have been filled in.

[0080] The technical solutions adopted in the embodiments of the present invention can effectively solve the technical problems of repetitive operations such as service application, enterprise approval and information filling, and realize a safe and reliable operation interface.

[0081] See Figure 4 , Figure 4 This is a schematic flowchart of another interface operation method according to an embodiment of the present invention, wherein the execution subject is a terminal. Specifically, it includes the following steps: S401. Send a task request described in natural language.

[0082] Users send task requests to the server by describing tasks in natural language through a unified JavaScript interface provided by the mobile SDK deployed on the terminal. For example, a user might input natural language commands such as: "Help me complete my housing provident fund withdrawal application" or "Fill out this company's annual business report." The mobile SDK provides a unified interface and capability encapsulation.

[0083] By using natural language to describe tasks, users can initiate automated workflows without manually defining pre-defined operation rules or recording demonstration videos. The mobile SDK encapsulates the user's natural language description into a task request and sends it to the server via a WebSocket full-duplex communication channel.

[0084] S402. Receive and respond to the screenshot request to obtain the first screenshot of the terminal, and send the first screenshot.

[0085] After sending a task request, the terminal receives a screenshot request from the server via a WebSocket full-duplex communication channel. In response to this screenshot request, the terminal performs a screenshot operation, capturing the first screenshot of the terminal's current interface. Screenshot modes support full-screen screenshots, in-application screenshots, and partial screenshots. Figure 3 There are several modes that can be flexibly selected based on task requirements and the current application status.

[0086] After processing is complete, the terminal sends the first screenshot back to the server via the same WebSocket communication channel. The first screenshot is a screenshot of the mobile application interface, reflecting the actual interface state of the terminal before task execution, providing an accurate basis for the server to perform semantic understanding and operation planning.

[0087] It should be noted that in this step, the terminal only performs screenshot capture and data transmission operations, without performing any interface semantic understanding or operational decisions. This reflects the transfer of reliance on high-level device permissions to the server. The terminal only needs basic screenshot permissions to participate in automated processes, without needing to obtain advanced system debugging permissions or accessibility service permissions from the device.

[0088] S403. Receive atomic operation instructions and execute atomic operation instructions to realize interface operation. The atomic operation instructions are transformed from the operation planning sequence generated by the first screen screenshot. The operation planning sequence is determined by combining the preset knowledge base and the graphical user interface intelligent agent based on the interface elements and functions of the interface elements in the first screen screenshot.

[0089] The terminal receives atomic operation instructions from the server via a WebSocket full-duplex communication channel. These atomic operation instructions are generated by the server's execution agent from a sequence of operation plans.

[0090] After receiving an atomic operation command, the terminal parses the command type and target parameters, and executes the corresponding operation on the mobile application interface. During execution, the terminal uses a unified mobile SDK to abstract cross-platform capabilities. The SDK interfaces with the native operation capabilities of various platforms: in the native Android environment, it obtains interface element information through accessibility services and executes operations through Instrumentation or TouchDriver. In the native iOS environment, it obtains interface information through the UI Automation framework and executes operations through XCEvent. In H5 and WebView environments, it communicates with the front-end page through an extended CDP protocol to achieve precise DOM-level operations. By executing atomic operation commands, the terminal simulates the user's manual operations on the interface, thereby replacing the user in completing automated mobile tasks.

[0091] exist Figure 4 In this embodiment, the terminal is responsible for capturing screenshots and executing instructions, enabling terminals on different operating system platforms to participate in interface operations through a consistent interface.

[0092] See Figure 5 , Figure 5 This is a flowchart illustrating the implementation of interface operations according to an embodiment of the present invention. Specifically, it includes the following steps: S501. Determine the operating environment based on the instruction type and target parameters in the atomic operation instruction.

[0093] After receiving the atomic operation command from the server, the terminal first parses the command type and target parameters. Command types include one or more of CLICK, LONG_PRESS, SWIPE, TEXT_INPUT, and DOUBLE_CLICK. The target parameters vary depending on the command type. For example, a click operation requires specifying the target coordinates (x, y), a long press operation requires specifying the target coordinates and press duration, a swipe operation requires specifying the start coordinates (startX, startY) and end coordinates (endX, endY), and a text input operation requires specifying the text content to be written.

[0094] After parsing the instruction type and target parameters, the terminal needs to determine the current runtime environment of the application. This includes determining whether the application is a native application, a page loaded via WebView, or an H5 page. The SDK can identify the current runtime environment type through the JavaScript interface.

[0095] S502 If the runtime environment is WebView or H5, then the atomic operation instructions are executed on the page DOM elements through the extended CDP protocol. If the runtime environment is a native application, then the atomic operation instructions are executed through the accessibility service or automation framework of the operating system.

[0096] When the terminal determines that the current running environment is WebView or H5, it executes atomic operation instructions using DOM manipulation. WebView is a browser component embedded in the native application on the mobile device, implemented based on the WebKit engine; H5 is a mobile web application. In the above environment, the terminal SDK communicates with the front-end page through the extended CDP (Chrome DevTools Protocol) protocol, directly manipulating the page's Document Object Model (DOM) elements.

[0097] Compared to simulating user actions, manipulating the DOM consumes less power and offers higher precision because it directly manipulates DOM nodes rather than relying on coordinate simulation. By parsing the DOM tree, the terminal can accurately locate UI elements and perform actions such as clicking and text input on target DOM elements based on the target parameters in the atomic operation instructions, without relying on coordinate calculations. Compatibility is enhanced through an extended CDP protocol.

[0098] When the terminal determines that the current running environment is a native application, it uses the operating system's accessibility services or automation framework to execute the operation corresponding to the atomic operation instruction.

[0099] In the native Android environment, the terminal SDK obtains UI element information through accessibility services and performs operations via Instrumentation or TouchDriver. In the native iOS environment, the terminal SDK obtains UI information through the UI Automation framework and performs operations via XCEvent.

[0100] The above solutions are restricted to use in native application environments, while low-power DOM manipulation solutions are preferred in WebView / H5 environments.

[0101] exist Figure 5 In the embodiments described, interface operations can be implemented in native Android, native iOS, H5, and WebView environments, thereby improving the effectiveness of interface operations.

[0102] In one embodiment of the present invention, the terminal sends a second screenshot of the screen after executing the atomic operation instruction to the server. This allows the server to compare the second screenshot with the first screenshot to evaluate the instruction execution effect and determine whether to regenerate the operation planning sequence based on the instruction execution effect.

[0103] Specifically, after executing the atomic operation command sent by the server, the terminal sends the execution result to the server. Simultaneously, to evaluate whether the atomic operation command achieved the expected effect, the server sends a screenshot request to the terminal again via a WebSocket full-duplex communication channel. In response to this screenshot request, the terminal performs the screenshot operation again, obtaining a second screenshot of the terminal after executing the atomic operation command.

[0104] The second screenshot has a clear temporal sequence and state difference from the first screenshot: the first screenshot reflects the terminal's interface state before the atomic operation instruction is executed, while the second screenshot reflects the terminal's interface state after the atomic operation instruction is executed. By sending the second screenshot to the server and comparing the changes in the screenshots before and after the operation, it is determined whether the operation has achieved the expected effect. Based on the execution result, it is then determined whether to regenerate the operation planning sequence, continue generating the next operation planning sequence, or determine that the task is completed.

[0105] See Figure 6 , Figure 6 This is a schematic diagram of the main structure of the interface operation device according to an embodiment of the present invention. The interface operation device can implement the interface operation method and is applied to a server, specifically including: Task module 601 is used to send a screenshot request to the terminal in response to a task request described in natural language. The intelligent agent module 602 is used to receive the first screenshot returned by the terminal, and use the graphical user interface intelligent agent to perform multimodal semantic understanding on the first screenshot to identify interface elements and the functions of the interface elements. The knowledge module 603 is used to combine a preset knowledge base with the interface elements and their functions to generate an operation planning sequence, and send atomic operation instructions converted from the operation planning sequence to execute the atomic operation instructions on the terminal to realize interface operation.

[0106] In one embodiment of the present invention, the intelligent agent module 602 is used to acquire a second screenshot after the terminal executes the atomic operation instruction, compare the second screenshot with the first screenshot to evaluate the instruction execution effect, and determine whether to regenerate the operation planning sequence based on the instruction execution effect.

[0107] In one embodiment of the present invention, the intelligent agent module 602 is used to regenerate the operation planning sequence based on the second screenshot if the execution effect of the instruction is that the atomic operation instruction does not achieve the expected effect. If the execution effect of the instruction is that the atomic operation instruction achieves the expected effect, and the task request is not completed, then the next operation planning sequence is generated based on the second screenshot; If the execution of the instruction results in the task request being completed, a task success message is sent to the terminal.

[0108] In one embodiment of the present invention, the intelligent agent module 602 is used to input the first screenshot into a graphical user interface intelligent agent, which identifies the buttons, input boxes, list items and text content in the first screenshot, and identifies the semantic functions of the buttons, input boxes, list items and text content in the business scenario corresponding to the task request.

[0109] In one embodiment of the present invention, the task module 601 is used to send a screenshot request to the terminal through a WebSocket full-duplex communication channel established with the terminal; Knowledge module 603 is used to send atomic operation instructions transformed from the operation planning sequence through the WebSocket full-duplex communication channel; And / or, Knowledge module 603 is used to stop sending the atomic operation instruction in response to a user takeover request sent by the terminal, so as to return the control of the interface operation to the user corresponding to the terminal. And / or, The task request includes service application, enterprise approval form filling or information entry; The first screenshot and the second screenshot are screenshots of the mobile application interface.

[0110] See Figure 7 , Figure 7 This is a schematic diagram of the main structure of another interface operation device according to an embodiment of the present invention. The interface operation device can implement an interface operation method and is applied to a terminal. Specifically, it includes: Communication module 701 is used to send task requests described in natural language; The screenshot acquisition module 702 is used to receive and respond to a screenshot request to acquire a first screenshot of the terminal, and send the first screenshot. The operation execution module 703 is used to receive atomic operation instructions and execute the atomic operation instructions to realize interface operation. The atomic operation instructions are converted from the operation planning sequence generated by the first screenshot. The operation planning sequence is determined by combining a preset knowledge base with a graphical user interface agent based on the interface elements of the first screenshot and the functions of the interface elements.

[0111] In one embodiment of the present invention, the operation execution module 703 is used to determine the operating environment based on the instruction type and target parameters in the atomic operation instruction; If the runtime environment is WebView or H5, the operation corresponding to the atomic operation instruction is executed on the page DOM element through the extended CDP protocol. If the runtime environment is a native application, the operation corresponding to the atomic operation instruction is executed through the accessibility service or automation framework of the operating system.

[0112] Figure 8 An exemplary system architecture 800 is shown that can be applied to the interface operation method or interface operation device of the present invention.

[0113] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, and 803, a network 804, and a server 805. Network 804 serves as the medium for providing communication links between terminal devices 801, 802, and 803 and server 805. Network 804 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0114] Users can use terminal devices 801, 802, and 803 to interact with server 805 via network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0115] Terminal devices 801, 802, and 803 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0116] Server 805 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 801, 802, and 803 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information and product information - for example only) to the terminal devices.

[0117] It should be noted that the interface operation method provided in the embodiments of the present invention is generally executed by the server 805, and correspondingly, the interface operation device is generally set in the server 805.

[0118] It should be understood that Figure 8 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0119] The following is for reference. Figure 9It shows a schematic diagram of the structure of a computer system 900 suitable for implementing a terminal device of the present invention. Figure 9 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0120] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the system 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0121] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0122] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this invention.

[0123] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a task module, an agent module, and a knowledge module. The names of these modules do not necessarily limit the module itself; for example, a task module can be described as "for sending a screenshot request to a terminal in response to a task request described in natural language."

[0126] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: In response to a task request sent by a terminal in the form of a natural language task description, a screenshot request is sent to the terminal. The system receives a first screenshot transmitted back from the terminal, and uses a graphical user interface agent to perform multimodal semantic understanding on the first screenshot to identify interface elements and their functions. Based on the interface elements and their functions, and using a preset knowledge base, an operation planning sequence is generated, and atomic operation instructions converted from the operation planning sequence are sent to the terminal to execute the atomic operation instructions and realize the interface operation.

[0127] According to the technical solution of this embodiment of the invention, in response to a task request described in natural language sent by a terminal, a screenshot request is sent to the terminal; a first screenshot returned by the terminal is received; a graphical user interface agent is used to perform multimodal semantic understanding on the first screenshot to identify interface elements and their functions; based on the interface elements and their functions, an operation planning sequence is generated using a preset knowledge base; and atomic operation instructions converted from the operation planning sequence are sent to the terminal to execute the atomic operation instructions to realize interface operation. By using screenshot recognition to realize interface operation, it can cope with complex and ever-changing terminal interface environments.

[0128] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention. It should be noted that the acquisition, storage, and application of user personal information involved in the technical solutions of this disclosure comply with relevant laws and regulations and do not violate public order and good morals.

Claims

1. A method for operating an interface, characterized in that, Applied to servers, including: In response to a task request sent by a terminal in the form of a natural language task description, a screenshot request is sent to the terminal. The system receives a first screenshot transmitted back from the terminal, and uses a graphical user interface agent to perform multimodal semantic understanding on the first screenshot to identify interface elements and their functions. Based on the interface elements and their functions, and using a preset knowledge base, an operation planning sequence is generated, and atomic operation instructions converted from the operation planning sequence are sent to the terminal to execute the atomic operation instructions and realize the interface operation.

2. The interface operation method according to claim 1, characterized in that, The method further includes: A second screenshot of the terminal after executing the atomic operation instruction is obtained. The second screenshot is compared with the first screenshot to evaluate the instruction execution effect. Based on the instruction execution effect, it is determined whether to regenerate the operation planning sequence.

3. The interface operation method according to claim 2, characterized in that, The step of determining whether to regenerate the operation planning sequence based on the execution effect of the instruction includes: If the execution effect of the instruction is that the atomic operation instruction does not achieve the expected effect, then the operation planning sequence is regenerated based on the second screenshot; If the execution effect of the instruction is that the atomic operation instruction achieves the expected effect, and the task request is not completed, then the next operation planning sequence is generated based on the second screenshot; If the execution of the instruction results in the task request being completed, a task success message is sent to the terminal.

4. The interface operation method according to claim 1, characterized in that, The step of using a graphical user interface (GUI) agent to perform multimodal semantic understanding on the first screenshot, and identifying interface elements and their functions, includes: The first screenshot is input into a graphical user interface agent, which identifies the buttons, input boxes, list items, and text content in the first screenshot, and identifies the semantic functions of the buttons, input boxes, list items, and text content in the business scenario corresponding to the task request.

5. The interface operation method according to claim 1, characterized in that, Sending a screenshot request to the terminal includes: sending a screenshot request to the terminal through a WebSocket full-duplex communication channel established with the terminal; Sending the atomic operation instruction transformed from the operation planning sequence includes: sending the atomic operation instruction transformed from the operation planning sequence through the WebSocket full-duplex communication channel; And / or, The method further includes: In response to a user takeover request sent by the terminal, the sending of the atomic operation instruction is stopped, so as to return control of the interface operation to the user corresponding to the terminal; And / or, The task request includes service application, enterprise approval form filling or information entry; The first screenshot and the second screenshot are screenshots of the mobile application interface.

6. A method for operating an interface, characterized in that, Applied to terminals, including: Send a task request described in natural language; Receive and respond to a screenshot request to obtain a first screenshot of the terminal, and send the first screenshot; The system receives atomic operation instructions and executes them to perform interface operations. The atomic operation instructions are derived from the operation planning sequence generated from the first screenshot. The operation planning sequence is determined by combining a preset knowledge base with a graphical user interface agent, based on the interface elements of the first screenshot and the functions of the interface elements.

7. The interface operation method according to claim 6, characterized in that, The execution of the atomic operation instructions to implement interface operations includes: The runtime environment is determined based on the instruction type and target parameters in the atomic operation instructions; If the runtime environment is WebView or H5, the operation corresponding to the atomic operation instruction is executed on the page DOM element through the extended CDP protocol. If the runtime environment is a native application, the operation corresponding to the atomic operation instruction is executed through the accessibility service or automation framework of the operating system.

8. A user interface operation device, characterized in that, Applied to servers, including: The task module is used to send a screenshot request to the terminal in response to a task request described in natural language. The intelligent agent module is used to receive the first screenshot transmitted back by the terminal, and use the graphical user interface intelligent agent to perform multimodal semantic understanding on the first screenshot to identify interface elements and the functions of the interface elements. The knowledge module is used to combine a preset knowledge base with the interface elements and their functions to generate an operation planning sequence, and send atomic operation instructions converted from the operation planning sequence to execute the atomic operation instructions on the terminal to realize interface operations.

9. A user interface operation device, characterized in that, Applied to terminals, including: The communication module is used to send task requests described in natural language; The screenshot acquisition module is used to receive and respond to a screenshot request to acquire a first screenshot of the terminal, and send the first screenshot. An operation execution module is used to receive atomic operation instructions and execute the atomic operation instructions to realize interface operation. The atomic operation instructions are converted from the operation planning sequence generated by the first screenshot. The operation planning sequence is determined by combining a preset knowledge base with a graphical user interface agent based on the interface elements of the first screenshot and the functions of the interface elements.

10. An electronic device with a user interface, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images